PyTest live logging in PyCharm

PyTest does allow output to be ‘live printed’ 

Also, it possible to see logging output in PyTest

Checkout these two links:

• https://intellij-support.jetbrains.com/hc/en-us/community/posts/360007644040-Show-logging-output-when-using-pytest
• https://docs.pytest.org/en/latest/capture.html#setting-capturing-methods-or-disabling-capturing

Updating the transformers package

Few steps I follow each time i update the transformers package

git pull
pip install --upgrade .
pip install -r ./examples/requirements.txt

that’s

Tensorflow GPU windows

conda install tensorflow-gpu

 

• https://towardsdatascience.com/tensorflow-gpu-installation-made-easy-use-conda-instead-of-pip-52e5249374bc

Latex quirks

its interesting how many quirks Latex has.

• I had to use a particular .sty  file for the paper.
  • to my dismay, I found its not sufficient to simple put it in the same folder as the .tex file
  • I had to do the following :
    • create directory where my other package files were present
    • place the .sty file there
    • do texhash
    • https://crmda.dept.ku.edu/guides/32.latex_config/32.latex_config.pdf
    • for me it was here : C:\Users\agoswami\AppData\Local\Programs\MiKTeX 2.9\tex\latex
• Bibliography
  • currently i am doing the following steps. i wonder if this can just be automated without using the command line each time i have to add a citation.
    • https://tex.stackexchange.com/questions/43325/citations-not-showing-up-in-text-and-bibiography
  • pdflatex main_trials
  • bibtex main_trials
  • pdflatex main_trials
  • pdflatex main_trials

System Design Interview

The system design interview can be of two kinds:

Object Oriented / Data Model :

• [Course] Programming-foundations-object-oriented-design
• http://www.andiamogo.com/S-OOD.pdf
• https://www.udemy.com/design-patterns-csharp-dotnet/

 

System Design :

• System Design Interview Questions

• Grokking the System Design Interview
• System Design

 

References:

• https://www.quora.com/How-do-I-answer-design-related-questions-like-design-a-parking-lot-in-an-Amazon-interview-Do-we-need-to-write-code-for-design-questions
• https://www.quora.com/How-do-I-prepare-to-answer-design-questions-in-a-technical-interview

Things you do as a well rounded ML Engineer / Data Scientist

I came across this list somewhere on the web.

I felt the list nicely captures the different kinds of skills a well rounded ML Engineer/Data Scientist develops over his/her career:

Architect data analytics solutions Access and Design Data Warehouse / DSS infrastructures Build and maintain data-driven applications Conduct AI / ML related research Conduct statistical analysis of the data Create BI reports and dashboards Create data visualizations Create / update predictive models using machine learning Develop AI / ML tools and libraries for internal or external customers Develop new data pipelines (ETL / ELT) Integrate new data sources for my organization Maintain and monitor data pipelines Maintain and monitor data warehouse Make decisions based on data-driven insights Operationalize machine learning models Support the AI / ML education

Python Class Attributes

I recently came across this interesting article on class attributes v/s instance attributes in Python.

References:

• https://www.toptal.com/python/python-class-attributes-an-overly-thorough-guide
  • This actually let me to 2 other guides which are very interesting:
  • https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/
  • https://timothyawiseman.wordpress.com/2012/10/06/class-and-instance-variables-in-python-2-7/
• http://www2.lib.uchicago.edu/keith/courses/python/class/5/##classinst
  • This is a very thorough guide. Highly recommended!

Code:

• _trail2.py

 

Named Pipes. C# Python. .NET Core

I recently ran into issues  using the NamedPipeServerStream API  with .NET Core.

• In Windows,  NamedPipeServerStream creates a pipe with a defined name in a specific location on the Windows filesystem (\\.pipe\\)
  • In a Python client application,  we were able to open this pipe for communication  using  the   code  snippet  :  open(r’\\.\pipe\\’ + pipe_name, ‘r+b’, 0)

 

• However, on Linux,  the behavior for the NamedPipeServerStream API is different.
  • Looking at the source code for .NET Core,  I saw that NamedPipeServer/ClientStream in .NET Core are built on top of Unix domain sockets.
  • So, if we want to communicate with a Python client, we have to use Python’s socket module

Code:

• C# (Server) : https://gist.github.com/abgoswam/025bbf7b845259e6cbfb0671c12b51ac
• Python (Client) : https://gist.github.com/abgoswam/7305409388e7192cf7ad2cba72296aab

 

References:

• https://github.com/dotnet/corefx/commit/fcc5cc9d78b84958394395611ebad9e7be222658
  • This is the PR that makes the NamedPipe*Stream API  on Unix use Unix domain sockets.
  • The description in the commit helped me understand the issue.
• http://jonathonreinhart.blogspot.com/2012/12/named-pipes-between-c-and-python.html
  • This is a nice post  on C# <-> Python bridge using Named Pipes.
  • However,  this technique doesn’t work for Linux (when using .NET Core). Hence I am posting this article  on how to use Unix Domain sockets instead. 🙂

C# OutOfProcess Python

• WaitForExit Method (Int32):
  • true if the associated process has exited; otherwise, false.
  • https://msdn.microsoft.com/en-us/library/ty0d8k56(v=vs.110).aspx
• UseShellExecute:
  • true if the shell should be used when starting the process; false if the process should be created directly from the executable file. The default is true.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.useshellexecute(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardOutput:
  • true if output should be written to Process.StandardOutput; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardoutput(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardError:
  • true if error output should be written to Process.StandardError; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandarderror(v=vs.110).aspx
• CreateNoWindow:
  • true if the process should be started without creating a new window to contain it; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.createnowindow(v=vs.110).aspx
  • CreateNoWindow works in conjunction with UseShellExecute as follows:

    To run the process without any window:

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.CreateNoWindow = true; 
    info.UseShellExecute = false;
    Process processChild = Process.Start(info); 

    To run the child process in it’s own window (new console)

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = true; // which is the default value.
    Process processChild = Process.Start(info); // separate window

    To run the child process in the parent’s console window –-> This is what we are using

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = false; // causes consoles to share window 
    Process processChild = Process.Start(info); 

• WindowStyle :
  • is a recommendation passed to the process
  • One of the enumeration values that indicates whether the process is started in a window that is maximized, minimized, normal (neither maximized nor minimized), or not visible. The default is Normal.

Git branch cleanups

• git remote prune origin –dry-run
• (Local) git branch -d <localbranch>

 

References:

• https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches#Tracking-Branches