du command (Linux)

June 10, 2017June 10, 2017 abgoswam linux

If i need to look up directory sizes only till 1 layer, then the following command is useful:

Tip:

the ‘-s’ flag is useful to summarize the contents on directory

$ du -csh *
52K Desktop
4.0K Documents
4.0K Downloads
0 media
4.0K Music
353M notebooks
4.0K Pictures
4.0K Public
4.0K Templates
947M transferLearning
4.0K Videos
1.3G total

References:

https://www.cyberciti.biz/faq/checking-directory-size-in-linux-command/

C# OutOfProcess Python

June 6, 2017June 6, 2017 abgoswam Uncategorized

WaitForExit Method (Int32):
- true if the associated process has exited; otherwise, false.
- https://msdn.microsoft.com/en-us/library/ty0d8k56(v=vs.110).aspx
UseShellExecute:
- true if the shell should be used when starting the process; false if the process should be created directly from the executable file. The default is true.
- https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.useshellexecute(v=vs.110).aspx
ProcessStartInfo.RedirectStandardOutput:
- true if output should be written to Process.StandardOutput; otherwise, false. The default is false.
- https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardoutput(v=vs.110).aspx
ProcessStartInfo.RedirectStandardError:
- true if error output should be written to Process.StandardError; otherwise, false. The default is false.
- https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandarderror(v=vs.110).aspx

CreateNoWindow:

true if the process should be started without creating a new window to contain it; otherwise, false. The default is false.
https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.createnowindow(v=vs.110).aspx

CreateNoWindow works in conjunction with UseShellExecute as follows:

To run the process without any window:

ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
info.CreateNoWindow = true; 
info.UseShellExecute = false;
Process processChild = Process.Start(info);

To run the child process in it’s own window (new console)

ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
info.UseShellExecute = true; // which is the default value.
Process processChild = Process.Start(info); // separate window

To run the child process in the parent’s console window –-> This is what we are using

ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
info.UseShellExecute = false; // causes consoles to share window 
Process processChild = Process.Start(info);

WindowStyle :
- is a recommendation passed to the process
- One of the enumeration values that indicates whether the process is started in a window that is maximized, minimized, normal (neither maximized nor minimized), or not visible. The default is Normal.

Git branch cleanups

April 7, 2017April 28, 2017 abgoswam git, Uncategorized

git remote prune origin –dry-run
(Local) git branch -d <localbranch>

References:

https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches#Tracking-Branches

Tuning Spark Jobs

April 5, 2017 abgoswam r, spark

I recently got into a discussion of how to tune spark jobs.

This led me to learnings related to dynamic allocation and stuff.

Some interesting links:

ML Algos in R

March 31, 2017March 31, 2017 abgoswam machinelearning, r

I have been trying to understand how ML algos in R fit together / compare with each other.

#	R	RevoScaleR	MML	Comment
1	lm	rxLinMod	—	Linear models
2	glm	rxGlm	—	Linear models
3	Glm w/ Binomial family and the logit link function	rxLogit	rxLogisticRegression	Logistic regression
4	rpart	rxDtree	—	Decision Trees implementations
5	gbm	rxBTrees	rxFastTrees	Boosted Decision Tree implementations
6	—-	rxDForest	rxFastForest

References:

#	Title	Links
1	Fitting Logistic Regression Models	https://msdn.microsoft.com/en-us/microsoft-r/scaler-user-guide-logistic-regression
2	Generalized Linear Models	https://msdn.microsoft.com/en-us/microsoft-r/scaler-user-guide-generalized-linear-mode
3	rxDTree(): a new type of tree algorithm for big data	http://blog.revolutionanalytics.com/2013/07/rxdtree-a-new-type-of-tree-algorithm.html
4	A first look at rxBTrees	http://blog.revolutionanalytics.com/2015/03/a-first-look-at-rxbtrees.html
5	A First Look at rxDForest()	http://blog.revolutionanalytics.com/2014/01/a-first-look-at-rxdforest.html

Using SSH Keys on Cloud Platforms

March 28, 2017April 6, 2017 abgoswam Uncategorized

Azure:

openssl.exe req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myPrivateKey.key -out myCert.pem
- We will mostly use the .key file
- The .pem file is only needed for Classic deployments. Typically we wont use this.

—-

Look up use of req : https://linux.die.net/man/1/req
- The req command primarily creates and processes certificate requests . Thats why the output of req is a cerificate (myCert.pem)
- But we are interested in the private key (myPrivateKey.key). Hence we are using the -keyout flag

openssl.exe rsa -pubout -in myPrivateKey.key -out myPublicKey.key
- This did not help. Because Azure complained about SSH2
ssh-keygen -e -f myPrivateKey.key
- This generated the SSH2 key (public) which I copy pasted into azure
- http://stackoverflow.com/questions/41167219/how-do-i-convert-my-open-ssh-public-key-to-ssh2
ssh-add myPrivateKey.key
- eval `ssh-agent -s`
- ssh–add
- in windows I had to start ssh-agent before running the ssh-add command
- http://stackoverflow.com/questions/17846529/could-not-open-a-connection-to-your-authentication-agent

References:

AWS:

In AWS, the private key is saved in a .pem file . you just use the .pem file to connect to the instances.
- Ideally the .pem extension is for certificates, not for keys.
- This was one of my confusions – because AWS saves the key in the .pem file

Tip: