Getting Started With Scala

The other day I was discussing about how to have a gentle introduction to the Scala language. Here is what I prescribe ..

• [Paper] An Overview of the Scala Programming Language
• [Doc] Brief Scala Tutorial.
• [Doc] Scala by Example by Martin Odersky
• [Book – Beginner] Atomic Scala 
• [Book – Beginner] Scala for the Impatient
• [Code Online] Scala Tutorials
• [Code Online] Scala Exercises
• [Online Tutorial] https://learnxinyminutes.com/docs/scala/
• [Online Tutorial] Official Documentation
• [Online Tutorial] Twitter’s Scala School
• [Online Tutorial] https://www.tutorialspoint.com/scala/
• [Course] Functional Programming Principles in Scala by Martin Odersky
• [Book – Intermediate] Programming in Scala by Martin Odersky, Lex Spoon, and Bill Venners

Sufficient Scala:

• http://www.dhgarrette.com/nlpclass/scala/basics.html
• http://alvinalexander.com/downloads/scala/Scala-Cheat-Sheet-devdaily.pdf
• https://github.com/lampepfl/progfun-wiki/blob/gh-pages/CheatSheet.md

Additional Resources:

• [Tutorial] Official Guide and Overview
• [Book – Intermediate] Programming Scala – by Alex Payne and Dean Wampler
• [Book – Intermediate] Functional Programming in Scala –  by Paul Chiusano and Rúnar Bjarnason
• [Book – Intermediate] Scala in Depth – by Joshua D. Suereth
• [Book – Intermediate] Scala Puzzlers – Available Now by Andrew Phillips and Nermin Šerifović

References:

• http://blog.codacy.com/2015/07/03/how-to-learn-scala/

DataFrame . A Dataset of Rows

References:

• https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-dataframe.html

 

Java, Maven, Scala, SBT Concepts

Am toe dipping into maven. Trying to make sense of how maven fits in with  IDE, command line maven, POM files blah blah etc.

tips:

• intellij will have default support within the IDE for both Maven and SBT.  So as long as we are not using mvn and sbt  from the command line we should be good.

java fundamental concepts:

• java application launcher
  • http://docs.oracle.com/javase/1.5.0/docs/tooldocs/windows/java.html
• java language compiler
  • http://docs.oracle.com/javase/1.5.0/docs/tooldocs/windows/javac.html
• class path
  • http://docs.oracle.com/javase/1.5.0/docs/tooldocs/windows/classpath.html
• jar archive tool
  • http://docs.oracle.com/javase/1.5.0/docs/tooldocs/windows/jar.html
  • http://stackoverflow.com/questions/9689793/cant-execute-jar-file-no-main-manifest-attribute

maven:

• http://www.vogella.com/tutorials/ApacheMaven/article.html
• http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html
  • these 2 go hand in hand. the first one has more details
• https://maven.apache.org/run-maven/
• https://www.tutorialspoint.com/maven/index.htm
• http://www.javacreed.com/why-should-we-use-maven/

base scala in intellij:

• https://www.jetbrains.com/help/idea/2016.2/scala.html
• https://www.jetbrains.com/help/idea/2016.2/creating-and-running-your-scala-application.html
• http://www.scala-lang.org/documentation/getting-started.html

scala with SBT:

• https://www.jetbrains.com/help/idea/2016.2/getting-started-with-sbt.html
• https://code.tutsplus.com/articles/from-zero-to-hello-world-in-scala–net-31894
• https://github.com/shekhargulati/52-technologies-in-2016/blob/master/02-sbt/README.md
  • This is a very interesting article on SBT

Recursion: Tail v/s Linear

Merits of Tail Recursion v/s Linear Recursion.

[TBD]

First Order Logic. Relationship between ‘for all’ and ‘exists’

 

References:

• https://en.wikipedia.org/wiki/First-order_logic

Code:

 

Spark App Development (Scala)

The spark app development process is pretty interesting.

I am jotting down some notes as I make progress in the process.

Notes:

• The easiest way is to develop your app in intellijIDEA and run it either from (a) intellij IDEA itself or  (b) from the sbt console using ‘run’
  • for this, we must necessarily  have the ‘master’ URL set to ‘local’
  • else you will get an error : “org.apache.spark.SparkException: A master URL must be set in your configuration”
• sbt package
  • the official spark quick start guide actually has an example of this, whereby you don’ t have to set the ‘master’ URL in the app itself.
  • instead, you specify the master when doing spark submit
  • Note:
    • if you have master set in code, then –master in spark-submit doesn’t take effect.
  •  Example:
    • sbt assembly
    • spark-submit –master spark://spark-host:7077 target/scala-2.11/HelloRedmondApp-assembly-1.0.jar
• sbt assembly
  • this is similar to the workflow for ‘sbt package’
  • in the build.sbt file,  there is a keyword  “Provided”  which has ramifications when one uses ‘sbt assembly’.
    • //libraryDependencies += “org.apache.spark” %% “spark-sql” % “2.0.0” % Provided
      //libraryDependencies += “org.apache.spark” %% “spark-mllib” % “2.0.0” % Provided
  • I need to follow up more on this Provided keyword…

Exercise:

To get additional insights this exercise was very useful.

• Step 1: Run Spark Standalone.
  • From tools/spark and run ./sbin/start-master.sh
  • Run ./sbin/start-slave.sh spark://spark-host:7077
  • At this point you should have Spark Standalone up and running
  • Open Standalone’s Web UI available at http://localhost:8080. Confirm you have got a node connected in Workers section at the top

• Step 2: Start spark-shell  (which is also a scala app itself) and attach it to the master.
  • spark-shell –master spark://spark-host:7077

• Step 3: Actually submit the application to the same cluster
  • sbt assembly
  • spark-submit –master spark://spark-host:7077 target/scala-2.11/HelloRedmondApp-assembly-1.0.jar
  • Note how the app is now in WAITING state. Its waiting because the cores have actually been allocated to the spark-shell from Step 2.

• Step 4: Kill the spark-shell that was started in Step 2. You will notice that the WAITING app now starts RUNNING.
  • this is because it now has the resources to run.

 

 

Code:

scalatrialapps

Running Tests in Scala IntelliJ IDEA

So I set up my first test suite in IntelliJ IDEA to test out some Scala code.

To run unit tests, I used the FunSuite package in ScalaTest.

import org.scalatest.FunSuite

• http://www.scalatest.org/user_guide/running_your_tests
• http://doc.scalatest.org/1.9.1/index.html#org.scalatest.FunSuite

 

Tip:

[1]  I modified the build.sbt to include the line:

libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"

I reloaded the project. However, for the changes to take effect I had to terminate the running sbt prompt and open a new one.

 

[2]  I did not use junit

 

References:

• http://stackoverflow.com/questions/33582350/scala-intellij-running-test-suite

 

 

Getting Started with Spark on Windows 10 (Part 1)

The references below helped me get started with spark on windows. I am listing down a few additional tips based on my experience:

Tips:

• Added the following as system environment variables for Sbt, Spark, Scala, Hadoop and Java.
  • 
  • 
• Added the following to the system PATH environment variable:
  • 
  • Note the java path is automatically picked up. I believe that’s because there is a java class path already present inside PATH [C:\ProgramData\Oracle\Java\javapath]
• spark-shell on Cygwin given an error on lauch itself.
  • Error looks something [: too many arguments
  • I think Spark on Cygwin has not been fully tried out yet. Read this.
• After getting spark-shell to launch on the Cmd window, there is another weird stack trace that I hit
  • Try doing this :  scala> spark.read.csv(“people.csv”).show
  • The error says something about ‘spark-warehouse’.
  • These two links helped to fix the issue:
    • http://stackoverflow.com/questions/38940312/why-does-spark-report-java-net-urisyntaxexception-relative-path-in-absolute-ur
    • http://stackoverflow.com/questions/38669206/spark-2-0-relative-path-in-absolute-uri-spark-warehouse
  • Doing something like this helped solve the issue:
    • spark-shell –conf spark.sql.warehouse.dir=file:///C:/tmp
• There is a weird stack dump that happens when exiting the spark-shell. (i.e on hitting :q within he spark shell)
  • It seems it is non-deterministic. i am not aware of the root cause of this issue..

References:

• https://hernandezpaul.wordpress.com/2016/01/24/apache-spark-installation-on-windows-10/
• https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-tips-and-tricks-running-spark-windows.html
• http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-5-2-error-on-quitting-spark-in-windows-7-td25659.html
  • This link gives a good overview of the SYSTEM/PATH variables that need to be set