Tuning Spark Jobs

I recently got into a discussion of how to tune spark jobs.

This led me to learnings related to dynamic allocation and stuff.

Some interesting links:

• http://jerryshao.me/architecture/2015/08/22/spark-dynamic-allocation-investigation/
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/

 

Spark Streaming

• https://newcircle.com/s/post/1763/intro-to-spark-streaming-video-tutorial

Good read :

• https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

 

 

Contributing to Open Source Big Data Tech

There are 6 projects that I have on my list of Open Source Technologies in the Big Data space that I want to contribute to :

(in no particular order)

• Spark
• Kafka
• Beam
• Flink
• Mahout
• Streams

 

kafka-spark integration

Am trying to get to a point whereby i will have  kafka + spark streaming running locally on my machine.

there are several things to figure out along the way:

• kafka
• scala
• spark
• spark-streaming

References:

• http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/
• http://aseigneurin.github.io/2016/03/02/kafka-spark-avro-kafka-101.html
• https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
• https://www.tutorialspoint.com/apache_kafka/apache_kafka_simple_producer_example.htm
• https://www.mapr.com/blog/real-time-streaming-data-pipelines-apache-apis-kafka-spark-streaming-and-hbase

 

DataFrame . A Dataset of Rows

References:

• https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-dataframe.html

 

Web Server Log Analysis with Apache Spark

References:

• http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html
• https://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format

 

 

EdX course ‘Introduction to Apache Spark’ resources

I am spending some time learning spark. As I make progress I think it would be a good idea to keep track of some resources I have found useful.

• Lecture 1 slides (PDF)
• Lecture 2 slides (PDF)
  • has very nice references on getting started / research papers etc.
• Lecture 3 slides (PDF)
• Deep Dive into Spark SQL’s Catalyst Optimizer
• Two notebooks from DataBricks:
  • A Gentle Introduction to Apache Spark on Databricks
  • Apache Spark on Databricks for Data Engineers
  • Apache Spark on Databricks for Data Scientists
    • the interesting thing to note about these 2 notebooks above is how databricks has informally tried to distinguish between data engineers and data scientists.
• Lab1 tutorial
• Lab2 tutorial
  • the labs were super useful

 

Code Repo: 

• edx_intro2spark