NPM v/s Bower – Intro

This is a nice intro to NPM.

• https://futurestud.io/tutorials/npm-quick-tips-1-install-packages

 

And here’s a nice intro to NPM.

• http://blog.teamtreehouse.com/getting-started-bower

 

This post captures some difference between npm and bower:

• http://stackoverflow.com/questions/18641899/what-is-the-difference-between-bower-and-npm

 

 

 

Visualization Using D3 (and dependent libraries)

This link gives a nice summary of data visualization libraries using D3:

• http://mikemcdearmon.com/portfolio/techposts/charting-libraries-using-d3.

Interestingly, it mentions mermaid and rickshaw! Two cool libraries I recently came across

Real time:

• Rickshaw with PubNub
  • https://www.pubnub.com/blog/2015-01-29-stream-data-to-create-realtime-charts-w-d3js-and-rickshaw/
• https://www.pubnub.com/blog/tag/visualizations/
  • This link has some cool examples. In particular, pubnub has examples of how to integrate the pubnub network with 3 popular JS charting libraries – Rickshaw, C3 and NVD3
  • http://blog.bronto.com/engineering/exploring-js-charting-part-1/
• Building Realtime, Live-Updating Animated Graphs with Eon C3.js
  • https://www.pubnub.com/blog/2015-02-26-building-realtime-live-updating-animated-graphs-c3-js/
• Stream Data to Create Realtime Charts w/ D3.js and Rickshaw
  • https://www.pubnub.com/blog/2015-01-29-stream-data-to-create-realtime-charts-w-d3js-and-rickshaw/

 

D3 – Getting Started

References:

• http://alignedleft.com/tutorials/d3/
• http://www.jeromecukier.net/blog/2012/09/04/getting-to-hello-world-with-d3/
• http://www.recursion.org/d3-for-mere-mortals/
• http://javascript.tutorialhorizon.com/2014/11/20/a-visual-explanation-of-the-enter-update-and-exit-selections-in-d3js/
• https://www.dashingd3js.com/table-of-contents
• http://square.github.io/intro-to-d3/examples/

 

Debugging Standard Deviation

In one of my previous posts, I had noted my thoughts around statistical measures like standard deviation and confidence intervals.

The fun part is of course when one has to debug these measures.

To that end I developed some insights by trying to visualize the data and plotting different kinds of charts using matplotlib

• The code below also acts as a reference to one of the pet peeves I have when trying to plot data from a python dataframe.
• Use the code below as reference going forward.

Also, sometimes you have to debug plots when they make no sense at all. Like this one below:

• The first plot didnt make sense to me initially. But once I started debugging it made total sense.
• Check the 2nd plot below which is what I get when I ‘sort’ the data

Code:

• std_var_trials
• std_var_trials_2

 

Plotting Errorbars in Python

As part of my previous post on confidence intervals, I wanted to now plot the error bars  that I got from my dataset.

I had to play around a bit to get this going. The referenced links helped me in this effort.

Tip:

• one interesting point was around the intervals. its +/- the y-value.

References:

• http://matplotlib.org/1.2.1/examples/pylab_examples/errorbar_demo.html
• http://matplotlib.org/examples/api/barchart_demo.html

(additional references which might be useful)

• http://matplotlib.org/examples/statistics/errorbar_demo_features.html
• http://matplotlib.org/examples/statistics/errorbar_limits.html
• http://matplotlib.org/examples/pylab_examples/errorbar_limits.html

 

Code:

• errorbars_yahoo

 

 

Comparing ML algos : Multi Armed bandit, Contextual Bandit, Logistic Regression, Online Learning

We have a system running Multi-Armed Bandit.

So when it came to select the next generation of ML algo to try out, we had a few choices:

1. Multi-Armed Bandit  (we had this running)
   • This entails ranking the items based on their respective conversion rates till that point of time.
2. Contextual Bandit
   • We use Vowpal Wabbit for this.
   • Internally Vowpal Wabbit treats contextual bandit in 2 distinct ways:
     • Without Action Dependent Features (non-ADF)
     • With Action Dependent Features (ADF)
   • Interestingly there is a difference between non-ADF and ADF modes.
     • In non-ADF mode, the VW creates multiple models (i.e. creates a model for each class).
     • In ADF mode, VW creates a single model.
3. Logistic Regression.
   • This entails reducing the problem to a binary classification problem.
   • Then using the model to score the items. Finally ranking the items based on the model score.
4. Online ML
   • Again treating this as a binary classification model, except this time we are updating the model in an online fashion.

 

Interestingly, on the dataset I was using I didn’t see much of a difference in algorithmic performance across the 4 different algorithms above.

 

Code:

_trials_compare3

 

Learning Curves. What to try next in ML ?

A very interesting problem in ML is : What to try next ?  Andrew Ng has some very interesting insights on this topic. (See the reference section below)

• Nowadays most ML platforms, e.g. AzureML give the ability to do parameter sweeps.
  • Most of the time they also do cross validation when doing sweeps.
  • This simplifies model selection, the platforms will automatically select the parameters during cross validation which give the best accuracy/AUC on the cross validation dataset.
  • This is usually the 1st thing to do for pretty much all ML problems.

 

• However, an interesting question still remains esp from a practical standpoint –
  • Should I focus more on feature engineering i.e. add more features.  OR Should I focus more on getting more data
  • For these cases I would generally use learning curves.
  • There are some nuances. So let me explain what I usually do.

 

• Plot of Training Error v/s Cross Validation Error.
  • This usually indicates whether I am currently suffering from a high bias (underfit) problem or a high variance (overfit) problem.
  • High Bias (underfit):
    • high training error. high generalization (CV) error
  • High Variance (overfit):
    • low training error. high generalization (CV) error

• High Variance (Overfitting) : Plot how the Error / Accuracy varies with increasing data.
  • A good idea here is to use log-base2 scale on the x-axis.
  • Using a log-base-2 scheme gives a good sense of how much the Error/Accuracy with decrease/increase with more data

• Based on the intuition above the following steps can be taken. 

 

What to Try next ?	Underfit (high bias)	Overfit (high variance)
Getting More Training Examples	No	Yes
Try smaller set of features	No	Yes. But first see if you can get more training examples.
Additional features	Yes	Maybe. If we get a feature that gives a strong signal then yes add it. But also invest in more data collection in parallel.

 

Code:

• usernameModeling

 

References:

1. https://class.coursera.org/ml-005/lecture