Factory Pattern in Python

I recently used the factory Pattern in Python.

• Gist

It was a little surprising to me comparing typical usage of this pattern in C# v/s Python.

In particular, in C# you’s typically have an Interface defining the methods and then provide an implementation of these methods in the concrete classes.

In Python, there is no interface – In the example above I embed a static  method in the base  to select the appropriate derived class.

Named Pipes. C# Python. .NET Core

I recently ran into issues  using the NamedPipeServerStream API  with .NET Core.

• In Windows,  NamedPipeServerStream creates a pipe with a defined name in a specific location on the Windows filesystem (\\.pipe\\)
  • In a Python client application,  we were able to open this pipe for communication  using  the   code  snippet  :  open(r’\\.\pipe\\’ + pipe_name, ‘r+b’, 0)

 

• However, on Linux,  the behavior for the NamedPipeServerStream API is different.
  • Looking at the source code for .NET Core,  I saw that NamedPipeServer/ClientStream in .NET Core are built on top of Unix domain sockets.
  • So, if we want to communicate with a Python client, we have to use Python’s socket module

Code:

• C# (Server) : https://gist.github.com/abgoswam/025bbf7b845259e6cbfb0671c12b51ac
• Python (Client) : https://gist.github.com/abgoswam/7305409388e7192cf7ad2cba72296aab

 

References:

• https://github.com/dotnet/corefx/commit/fcc5cc9d78b84958394395611ebad9e7be222658
  • This is the PR that makes the NamedPipe*Stream API  on Unix use Unix domain sockets.
  • The description in the commit helped me understand the issue.
• http://jonathonreinhart.blogspot.com/2012/12/named-pipes-between-c-and-python.html
  • This is a nice post  on C# <-> Python bridge using Named Pipes.
  • However,  this technique doesn’t work for Linux (when using .NET Core). Hence I am posting this article  on how to use Unix Domain sockets instead. 🙂

Temp files in Python.

Recently, I hit a nasty bug while working with temp files in Python.

The issue was manifesting itself in Windows.  Interestingly I wasnt hitting it on Linux.

• As it so turns out I would have eventually hit it on Linux too.

 

The issue was because of a leaked file descriptor.  The following link explains the issue

• https://www.logilab.org/blogentry/17873

du command (Linux)

Note:

• du command hides hidden files.  A more convenient command is ‘ncdu‘
  • sudo apt-get install ncdu
• Please refer to his link :
  • https://askubuntu.com/questions/356902/why-doesnt-this-show-the-hidden-files-folders

 

If i need to look up directory sizes  only till 1 layer, then the following command is useful:

Tip:

• the ‘-s’ flag is useful to summarize the contents on directory

 

$ du -csh *
52K Desktop
4.0K Documents
4.0K Downloads
0 media
4.0K Music
353M notebooks
4.0K Pictures
4.0K Public
4.0K Templates
947M transferLearning
4.0K Videos
1.3G total

References:

https://www.cyberciti.biz/faq/checking-directory-size-in-linux-command/

 

C# OutOfProcess Python

• WaitForExit Method (Int32):
  • true if the associated process has exited; otherwise, false.
  • https://msdn.microsoft.com/en-us/library/ty0d8k56(v=vs.110).aspx
• UseShellExecute:
  • true if the shell should be used when starting the process; false if the process should be created directly from the executable file. The default is true.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.useshellexecute(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardOutput:
  • true if output should be written to Process.StandardOutput; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardoutput(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardError:
  • true if error output should be written to Process.StandardError; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandarderror(v=vs.110).aspx
• CreateNoWindow:
  • true if the process should be started without creating a new window to contain it; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.createnowindow(v=vs.110).aspx
  • CreateNoWindow works in conjunction with UseShellExecute as follows:

    To run the process without any window:

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.CreateNoWindow = true; 
    info.UseShellExecute = false;
    Process processChild = Process.Start(info); 

    To run the child process in it’s own window (new console)

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = true; // which is the default value.
    Process processChild = Process.Start(info); // separate window

    To run the child process in the parent’s console window –-> This is what we are using

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = false; // causes consoles to share window 
    Process processChild = Process.Start(info); 

• WindowStyle :
  • is a recommendation passed to the process
  • One of the enumeration values that indicates whether the process is started in a window that is maximized, minimized, normal (neither maximized nor minimized), or not visible. The default is Normal.

Git branch cleanups

• git remote prune origin –dry-run
• (Local) git branch -d <localbranch>

 

References:

• https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches#Tracking-Branches

Tuning Spark Jobs

I recently got into a discussion of how to tune spark jobs.

This led me to learnings related to dynamic allocation and stuff.

Some interesting links:

• http://jerryshao.me/architecture/2015/08/22/spark-dynamic-allocation-investigation/
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/