Named Pipes. C# Python. .NET Core

I recently ran into issues  using the NamedPipeServerStream API  with .NET Core.

• In Windows,  NamedPipeServerStream creates a pipe with a defined name in a specific location on the Windows filesystem (\\.pipe\\)
  • In a Python client application,  we were able to open this pipe for communication  using  the   code  snippet  :  open(r’\\.\pipe\\’ + pipe_name, ‘r+b’, 0)

 

• However, on Linux,  the behavior for the NamedPipeServerStream API is different.
  • Looking at the source code for .NET Core,  I saw that NamedPipeServer/ClientStream in .NET Core are built on top of Unix domain sockets.
  • So, if we want to communicate with a Python client, we have to use Python’s socket module

Code:

• C# (Server) : https://gist.github.com/abgoswam/025bbf7b845259e6cbfb0671c12b51ac
• Python (Client) : https://gist.github.com/abgoswam/7305409388e7192cf7ad2cba72296aab

 

References:

• https://github.com/dotnet/corefx/commit/fcc5cc9d78b84958394395611ebad9e7be222658
  • This is the PR that makes the NamedPipe*Stream API  on Unix use Unix domain sockets.
  • The description in the commit helped me understand the issue.
• http://jonathonreinhart.blogspot.com/2012/12/named-pipes-between-c-and-python.html
  • This is a nice post  on C# <-> Python bridge using Named Pipes.
  • However,  this technique doesn’t work for Linux (when using .NET Core). Hence I am posting this article  on how to use Unix Domain sockets instead. 🙂

C# OutOfProcess Python

• WaitForExit Method (Int32):
  • true if the associated process has exited; otherwise, false.
  • https://msdn.microsoft.com/en-us/library/ty0d8k56(v=vs.110).aspx
• UseShellExecute:
  • true if the shell should be used when starting the process; false if the process should be created directly from the executable file. The default is true.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.useshellexecute(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardOutput:
  • true if output should be written to Process.StandardOutput; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardoutput(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardError:
  • true if error output should be written to Process.StandardError; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandarderror(v=vs.110).aspx
• CreateNoWindow:
  • true if the process should be started without creating a new window to contain it; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.createnowindow(v=vs.110).aspx
  • CreateNoWindow works in conjunction with UseShellExecute as follows:

    To run the process without any window:

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.CreateNoWindow = true; 
    info.UseShellExecute = false;
    Process processChild = Process.Start(info); 

    To run the child process in it’s own window (new console)

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = true; // which is the default value.
    Process processChild = Process.Start(info); // separate window

    To run the child process in the parent’s console window –-> This is what we are using

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = false; // causes consoles to share window 
    Process processChild = Process.Start(info); 

• WindowStyle :
  • is a recommendation passed to the process
  • One of the enumeration values that indicates whether the process is started in a window that is maximized, minimized, normal (neither maximized nor minimized), or not visible. The default is Normal.

Git branch cleanups

• git remote prune origin –dry-run
• (Local) git branch -d <localbranch>

 

References:

• https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches#Tracking-Branches

Using SSH Keys on Cloud Platforms

Azure:

• openssl.exe req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myPrivateKey.key -out myCert.pem
  • We will mostly use the .key file
  • The .pem file is only needed for Classic deployments. Typically we wont use this.

—-

• Look up use of req : https://linux.die.net/man/1/req
  • The req command primarily creates and processes certificate requests . Thats why the output of req is a cerificate (myCert.pem)
  • But we are interested in the private key (myPrivateKey.key). Hence we are using the -keyout flag

---

• openssl.exe rsa -pubout -in myPrivateKey.key -out myPublicKey.key
  • This did not help. Because Azure complained about SSH2
• ssh-keygen -e -f myPrivateKey.key
  • This generated the SSH2 key (public) which I copy pasted into azure
  • http://stackoverflow.com/questions/41167219/how-do-i-convert-my-open-ssh-public-key-to-ssh2
•  ssh-add myPrivateKey.key
  • eval `ssh-agent -s`
  • ssh–add
    
  • in windows I had to start ssh-agent before running the ssh-add command
  • http://stackoverflow.com/questions/17846529/could-not-open-a-connection-to-your-authentication-agent

References:

• https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-ssh-from-windows
• https://www.jeff.wilcox.name/2013/06/secure-linux-vms-with-ssh-certificates/

 

AWS:

• In AWS,  the private key is saved in a .pem file . you just use the .pem file to connect to the instances.
  • Ideally the .pem extension is for certificates, not for keys.
  • This was one of my confusions – because AWS saves the key in the .pem file 

 

Tip:

• Use ssh-agent to store private keys. Makes life much simpler!

 

Terminal Choices and Tips

It seems there is a deluge of terminal options on the windows platform.

• Cygwin
• Mintty
  • Whats the relationship between cygwin and mintty ? 
  • The cygwin shortcut on my machine looks as follows:
    • C:\cygwin64\bin\mintty.exe -i /Cygwin-Terminal.ico –
• GitBash
• MYSYS
• CMD
• PS

Tips:

• In cygwin, I had an issue whereby the backspace key did not work in the ‘sbt console’ REPL.
  • Fix was to go to Options... -> Keys, and make sure Send Backspace as ^H is checked. This will allow the REPL to correctly interpret backspaces
  • http://stackoverflow.com/questions/17124689/i-really-would-like-sbt-and-its-console-to-work-under-cygwin-any-way-you-think-i

 

References:

 

Multi-dimensional (axial) data handling in Python

Recently I was playing around with multi-dimensional data structures in Python.

Some interesting observations:

1. Multi-dimensional lists and multi-dimensional arrays are fundamentally handled differently.
2. Slicing of multi-dimensional arrays (numpy) need to be carefully considered in regards to shallow copy etc

 

Some references below for further examination:

References:

1. http://ilan.schnell-web.net/prog/slicing/
2. https://docs.python.org/2/library/copy.html
3. http://stackoverflow.com/questions/509211/explain-pythons-slice-notation
4. http://cs231n.github.io/python-numpy-tutorial/
5. http://www.physics.nyu.edu/pine/pymanual/html/chap3/chap3_arrays.html
6. http://www.astro.ufl.edu/~warner/prog/python.html

 

Pending Posts

• Authentication and Authorization in ASP.NET Web API.
  • http://www.asp.net/web-api/overview/security/authentication-and-authorization-in-aspnet-web-api
• Reservoir Sampling.
• Shuffle command in Linux.
• Regex in Python.
• Http Headers.
• Await-Async in C#.
• Singleton in Python.
• Singleton in C#.
• Redis on Azure.
  • Horizontal Scalability
  • Write Throughput
• Vowpal Wabbit.
• Vowpal Wabbit Format.
• Multi-Threading in C#
• Multi-Threading in Python
• Wikipedia – Linux Solution
• Wikipedia – Spark
• Data Science From Scratch – Clustering.
• Stock Analysis in Python
  • https://simplypython.wordpress.com/2014/12/24/basic-stock-technical-analysis-with-python/
• jQuery
  • Video
• Decision Trees
• Confidence Intervals – Discussion
• Question about computing Std Dev.
• Looping over pandas DataFrames – When is it necessary ? When can it be avoided ?
• L1 / L2 regularization error – What does it mean ?
• Learning With Counts
• Difference between HttpClient and HttpRequestMessage
  • http://d-fens.ch/2014/04/12/httpclient-and-how-to-use-headers-content-type-and-postasync/
  • http://stackoverflow.com/questions/10679214/how-do-you-set-the-content-type-header-for-an-httpclient-request
• Count Min Sketch

 

References:

1. http://www.codeproject.com/Tips/805923/Asynchronous-programming-in-Web-API-ASP-NET-MVC
2. http://stackoverflow.com/questions/8463809/customize-the-authorization-http-header