Spark App Development (Python)

In a previous post, I wrote about the spark app development process for Scala.

In this post/example, I have provided examples of how to develop a spark app using the pyspark library.

For Python 3.5

  • For interactive use I have to do ‘export PYSPARK_PYTHON=python3’ ¬†before doing pyspark
  • For standalone programs in local mode I first have to add the following in the script:
    • #os.environ[“PYSPARK_PYTHON”]=”/usr/bin/python3″