In a previous post, I wrote about the spark app development process for Scala.
In this post/example, I have provided examples of how to develop a spark app using the pyspark library.
For Python 3.5
- For interactive use I have to do ‘export PYSPARK_PYTHON=python3’ before doing pyspark
- For standalone programs in local mode I first have to add the following in the script: