Getting Started with Spark on Windows 10 (Part 1)

The references below helped me get started with spark on windows. I am listing down a few additional tips based on my experience:

Tips:

  • Added the following as system environment variables for Sbt, Spark, Scala, Hadoop and Java.
    • sbt_spark_scala
    • hadoop_java
  • Added the following to the system PATH environment variable:
    • systempath
    • Note the java path is automatically picked up. I believe that’s because there is a java class path already present inside PATH [C:\ProgramData\Oracle\Java\javapath]
  • spark-shell on Cygwin given an error on lauch itself.
    • Error looks something [: too many arguments
    • I think Spark on Cygwin has not been fully tried out yet. Read this.
  • After getting spark-shell to launch on the Cmd window, there is another weird stack trace that I hit
  • There is a weird stack dump that happens when exiting the spark-shell. (i.e on hitting :q within he spark shell)
    • It seems it is non-deterministic. i am not aware of the root cause of this issue..

References:

 

Advertisements

One thought on “Getting Started with Spark on Windows 10 (Part 1)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s