Spark App Development (Python)

In a previous post, I wrote about the spark app development process for Scala.

In this post/example, I have provided examples of how to develop a spark app using the pyspark library.

For Python 3.5

  • For interactive use I have to do ‘export PYSPARK_PYTHON=python3’  before doing pyspark
  • For standalone programs in local mode I first have to add the following in the script:
    • #os.environ[“PYSPARK_PYTHON”]=”/usr/bin/python3″



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s