Using SSH Keys on Cloud Platforms

Azure:

• openssl.exe req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myPrivateKey.key -out myCert.pem
  • We will mostly use the .key file
  • The .pem file is only needed for Classic deployments. Typically we wont use this.

—-

• Look up use of req : https://linux.die.net/man/1/req
  • The req command primarily creates and processes certificate requests . Thats why the output of req is a cerificate (myCert.pem)
  • But we are interested in the private key (myPrivateKey.key). Hence we are using the -keyout flag

---

• openssl.exe rsa -pubout -in myPrivateKey.key -out myPublicKey.key
  • This did not help. Because Azure complained about SSH2
• ssh-keygen -e -f myPrivateKey.key
  • This generated the SSH2 key (public) which I copy pasted into azure
  • http://stackoverflow.com/questions/41167219/how-do-i-convert-my-open-ssh-public-key-to-ssh2
•  ssh-add myPrivateKey.key
  • eval `ssh-agent -s`
  • ssh–add
    
  • in windows I had to start ssh-agent before running the ssh-add command
  • http://stackoverflow.com/questions/17846529/could-not-open-a-connection-to-your-authentication-agent

References:

• https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-ssh-from-windows
• https://www.jeff.wilcox.name/2013/06/secure-linux-vms-with-ssh-certificates/

 

AWS:

• In AWS,  the private key is saved in a .pem file . you just use the .pem file to connect to the instances.
  • Ideally the .pem extension is for certificates, not for keys.
  • This was one of my confusions – because AWS saves the key in the .pem file 

 

Tip:

• Use ssh-agent to store private keys. Makes life much simpler!

 

NPM v/s Bower – Intro

This is a nice intro to NPM.

• https://futurestud.io/tutorials/npm-quick-tips-1-install-packages

 

And here’s a nice intro to NPM.

• http://blog.teamtreehouse.com/getting-started-bower

 

This post captures some difference between npm and bower:

• http://stackoverflow.com/questions/18641899/what-is-the-difference-between-bower-and-npm

 

 

 

Visualization Using D3 (and dependent libraries)

This link gives a nice summary of data visualization libraries using D3:

• http://mikemcdearmon.com/portfolio/techposts/charting-libraries-using-d3.

Interestingly, it mentions mermaid and rickshaw! Two cool libraries I recently came across

Real time:

• Rickshaw with PubNub
  • https://www.pubnub.com/blog/2015-01-29-stream-data-to-create-realtime-charts-w-d3js-and-rickshaw/
• https://www.pubnub.com/blog/tag/visualizations/
  • This link has some cool examples. In particular, pubnub has examples of how to integrate the pubnub network with 3 popular JS charting libraries – Rickshaw, C3 and NVD3
  • http://blog.bronto.com/engineering/exploring-js-charting-part-1/
• Building Realtime, Live-Updating Animated Graphs with Eon C3.js
  • https://www.pubnub.com/blog/2015-02-26-building-realtime-live-updating-animated-graphs-c3-js/
• Stream Data to Create Realtime Charts w/ D3.js and Rickshaw
  • https://www.pubnub.com/blog/2015-01-29-stream-data-to-create-realtime-charts-w-d3js-and-rickshaw/

 

Spark Streaming

• https://newcircle.com/s/post/1763/intro-to-spark-streaming-video-tutorial

Good read :

• https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

 

 

Details running R Scripts

• trace(functionName, edit=TRUE).  Then write browser() where you want it to break
• source(‘~/scripts/trial3_criteo_ensembles_ag.R’, echo=TRUE)
• Rscript -e “.libPaths()”

Other functions in R I didn’t know:

• class(score)
• names(score)
• head(criteoTest)
• class(criteoTest)
• rxGetVarInfo(criteoTest)
• warnings()
• sapply(score, class)

Alias method

“You are given an n-sided die where side i has probability p_i of being rolled. What is the most efficient data structure for simulating rolls of the die?”

A very similar question was posted to me recently  :

• http://stackoverflow.com/questions/4511331/randomly-selecting-an-element-from-a-weighted-list
• http://stackoverflow.com/questions/3120035/indexing-count-of-buckets/3120179#3120179

The approaches above are very cool, and illustrate the use of augmented search trees.

However, it seems there is a better method for this problem – and it has been out there for a while now. This was a fascinating read :

• Darts, Dice, and Coins: Sampling from a Discrete Distribution

Additional pointers for the alias method:

• https://prxq.wordpress.com/2006/04/17/the-alias-method/
• https://en.wikipedia.org/wiki/Alias_method

 

 

Docker. Getting Started.

For installing docker on Ubuntu 18.04 [bionic] and to ensure the docker files are stored in /mnt   (insted of root filesystem)

• https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04
• https://stackoverflow.com/questions/52488300/how-to-change-root-dir-of-docker-on-ubuntu-18-04-lts-docker-change-location-of

 

 

References:

Installation:

• https://docs.docker.com/engine/installation/linux/ubuntu/
• https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04
• https://www.howtoforge.com/tutorial/docker-installation-and-usage-on-ubuntu-16.04/

Concepts

• https://docs.docker.com/engine/getstarted/step_two/
  • multiple steps here which would be quite useful

Commands:

docker cp <containerId>:/file/path/within/container /host/path/target

Feature Scaling in SGD

SGD is the perfect algorithm for use in online learning. Except it has one major drawback – is sensitive to feature scaling.

In some of my trials with the SGD learner in scikit-learn, I have seen terrible performance if I don’t do feature scaling.

Which begs the question – How does VW do feature scaling ? After all VW does online learning.

Tip:

It seems VW uses a kind of SGD that is scale variant:

• https://arxiv.org/abs/1408.2065
• https://arxiv.org/abs/1011.1576

References:

• http://scikit-learn.org/stable/modules/sgd.html

Code:

• winesgd
• winescikit

 

Software Packages for Graphical Models

This is a nice link which lists the packages out there for Grpahical Models.

• http://www.cs.ubc.ca/~murphyk/Software/bnsoft.html

 

Projects in Deep Learning

I have been thinking about a project involving Deep Learning.

Here are some links which list some interesting projects developed using deep learning.

• http://machinelearningmastery.com/inspirational-applications-deep-learning/
• http://deeplearninggallery.com/
• https://github.com/ChristosChristofidis/awesome-deep-learning
• https://github.com/ujjwalkarn/Machine-Learning-Tutorials