Toe Dipping Into Apache Mahout

I went to this talk titled ‘Apache Mahout – What’s next?’ by Trevor Grant

Few things struck me after attending the talk:

• apache mahout seems to be a pretty interesting framework for distributed matrix operations.
  • the ppt can be found here
• trevor’s blog post has great pointers for getting started with a lot of the technologies on the fringes like  flink, mahot etc
  • https://trevorgrant.org/

kafka-spark integration

Am trying to get to a point whereby i will have  kafka + spark streaming running locally on my machine.

there are several things to figure out along the way:

• kafka
• scala
• spark
• spark-streaming

References:

• http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/
• http://aseigneurin.github.io/2016/03/02/kafka-spark-avro-kafka-101.html
• https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
• https://www.tutorialspoint.com/apache_kafka/apache_kafka_simple_producer_example.htm
• https://www.mapr.com/blog/real-time-streaming-data-pipelines-apache-apis-kafka-spark-streaming-and-hbase

 

Polyglot Persistence

I came across this term polyglot persistence  while reading a  post about ACID in the context of NoSQL databases.

The post  gave me insights  into the world of NoSQL  databases, and  how to think about ACID in the context  of NoSQL systems.

It has some great links and pointers. in particular some interesting pointers to Eric Brewers posts on highscalability.com

Check it out!

Distributed Locks

One reason why Redis has custom locking, instead of using operating system–level locks, language-level locks, and so forth, is a matter of scope. Clients want to have exclusive access to data stored on Redis, so clients need to have access to a lock defined in a scope that all clients can see—Redis.

Redis does have a basic sort of lock already available as part of the command set (SETNX), which we use, but it’s not full-featured and doesn’t offer advanced functionality that users would expect of a distributed lock.

In fact there are two patterns which have emerged for locking in Redis.

1. Locking with SETNX
   • it’s not full-featured and doesn’t offer advanced functionality
2. Redlock
   • the distributed locking algorithm.

 

References:

• http://redis.io/topics/distlock
• http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
• http://antirez.com/news/101
• http://redis.io/commands/setnx

Redis Internals

References:

• https://pauladamsmith.com/articles/redis-under-the-hood.html
• https://pauladamsmith.com/blog/2011/03/redis_get_set.html
  • These two are nice references worth reading.
• https://github.com/antirez/redis
  • The README file in the github repo gives a nice overview of Redis down to its implementation details.
• http://redis.io/topics/faq
• http://redis.io/commands/INFO

 

Data Types:

• http://redis.io/topics/data-types-intro

On Persistence:

• http://redis.io/topics/persistence
• http://oldblog.antirez.com/post/redis-persistence-demystified.html

Pub / Sub:

• http://redis.io/topics/pubsub

Memory:

• http://redis.io/topics/lru-cache
• http://redis.io/topics/virtual-memory
• http://redis.io/topics/memory-optimization

Redis Cluster Design and Specification

• http://redis.io/topics/cluster-spec#scaling-reads-using-slave-nodes
• http://bencane.com/2013/11/12/installing-redis-and-setting-up-master-slave-replication/
• http://stackoverflow.com/questions/37498293/should-i-read-from-a-redis-cluster-slave

Pipelining/Transactions

• http://stackoverflow.com/questions/29327544/pipelining-vs-transaction-in-redis
• http://www.terminalstate.net/2011/05/redis-pipelines-and-transactions.html

 

Scaling real time processing jobs in Azure

I recently was faced with an issue about how to scale real time processing jobs in Azure.

I finally managed to do it by using the concept of partitions.  Using partitions in EventHub along with Azure Stream Analytics got the job done for me.

References:

• https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-scale-jobs/
• https://azure.microsoft.com/en-us/documentation/articles/event-hubs-programming-guide/#partition-key
• https://msdn.microsoft.com/en-us/library/azure/mt652148.aspx
• https://azure.microsoft.com/en-us/documentation/articles/event-hubs-overview/
• https://azure.microsoft.com/en-us/documentation/articles/event-hubs-api-overview/

Multithreading in Python

References:

• https://pymotw.com/2/threading/
• https://pymotw.com/2/logging/

Code:

• threads1
• threads2
• logging
• Exp_21_RRS_Train_Threads