Tips for building better CNN models

Here are some tips for building powerful CNN models.

Things to try:

• Filter size: Smaller filters (3×3) may be more efficient
• Number of filters: Is 32 filters the right choice. Do more or fewer do better?
• Pooling vs Strided Convolution: Do you use max pooling or just stride convolutions?
• Batch normalization: Try adding spatial batch normalization after convolution layers and vanilla batch normalization after affine layers. Do your networks train faster?
• Network architecture: Can we do better with a deep network? Good architectures to try include:
  • [conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]
  • [conv-relu-conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]
  • [batchnorm-relu-conv]xN -> [affine]xM -> [softmax or SVM]
• Use TensorFlow Scope: Use TensorFlow scope and/or tf.layers to make it easier to write deeper networks. See this tutorial for making how to use tf.layers.
• Use Learning Rate Decay: As the notes point out, decaying the learning rate might help the model converge. Feel free to decay every epoch, when loss doesn’t change over an entire epoch, or any other heuristic you find appropriate. See the Tensorflow documentation for learning rate decay.
• Global Average Pooling: Instead of flattening and then having multiple affine layers, perform convolutions until your image gets small (7×7 or so) and then perform an average pooling operation to get to a 1×1 image picture (1, 1 , Filter#), which is then reshaped into a (Filter#) vector. This is used in Google’s Inception Network (See Table 1 for their architecture).
• Regularization: Add l2 weight regularization, or perhaps use Dropout as in the TensorFlow MNIST tutorial

Tips for training

For each network architecture that you try, you should tune the learning rate and regularization strength. When doing this there are a couple important things to keep in mind:

• If the parameters are working well, you should see improvement within a few hundred iterations
• Remember the coarse-to-fine approach for hyperparameter tuning: start by testing a large range of hyperparameters for just a few training iterations to find the combinations of parameters that are working at all.
• Once you have found some sets of parameters that seem to work, search more finely around these parameters. You may need to train for more epochs.
• You should use the validation set for hyperparameter search, and we’ll save the test set for evaluating your architecture on the best parameters as selected by the validation set.

Going above and beyond

If you are feeling adventurous there are many other features you can implement to try and improve your performance.

• Alternative update steps: SGD+momentum, RMSprop, and Adam, AdaGrad,  AdaDelta.
• Alternative activation functions such as leaky ReLU, parametric ReLU, ELU, or MaxOut.
• Model ensembles
• Data augmentation
• New Architectures
  • ResNets where the input from the previous layer is added to the output.
  • DenseNets where inputs into previous layers are concatenated together.
  • This blog has an in-depth overview

 

Advertisements

Factory Pattern in Python

I recently used the factory Pattern in Python.

• Gist

It was a little surprising to me comparing typical usage of this pattern in C# v/s Python.

In particular, in C# you’s typically have an Interface defining the methods and then provide an implementation of these methods in the concrete classes.

In Python, there is no interface – In the example above I embed a static  method in the base  to select the appropriate derived class.

Named Pipes. C# Python. .NET Core

I recently ran into issues  using the NamedPipeServerStream API  with .NET Core.

• In Windows,  NamedPipeServerStream creates a pipe with a defined name in a specific location on the Windows filesystem (\\.pipe\\)
  • In a Python client application,  we were able to open this pipe for communication  using  the   code  snippet  :  open(r’\\.\pipe\\’ + pipe_name, ‘r+b’, 0)

 

• However, on Linux,  the behavior for the NamedPipeServerStream API is different.
  • Looking at the source code for .NET Core,  I saw that NamedPipeServer/ClientStream in .NET Core are built on top of Unix domain sockets.
  • So, if we want to communicate with a Python client, we have to use Python’s socket module

Code:

• C# (Server) : https://gist.github.com/abgoswam/025bbf7b845259e6cbfb0671c12b51ac
• Python (Client) : https://gist.github.com/abgoswam/7305409388e7192cf7ad2cba72296aab

 

References:

• https://github.com/dotnet/corefx/commit/fcc5cc9d78b84958394395611ebad9e7be222658
  • This is the PR that makes the NamedPipe*Stream API  on Unix use Unix domain sockets.
  • The description in the commit helped me understand the issue.
• http://jonathonreinhart.blogspot.com/2012/12/named-pipes-between-c-and-python.html
  • This is a nice post  on C# <-> Python bridge using Named Pipes.
  • However,  this technique doesn’t work for Linux (when using .NET Core). Hence I am posting this article  on how to use Unix Domain sockets instead. 🙂

Temp files in Python.

Recently, I hit a nasty bug while working with temp files in Python.

The issue was manifesting itself in Windows.  Interestingly I wasnt hitting it on Linux.

• As it so turns out I would have eventually hit it on Linux too.

 

The issue was because of a leaked file descriptor.  The following link explains the issue

• https://www.logilab.org/blogentry/17873

du command (Linux)

Note:

• du command hides hidden files.  A more convenient command is ‘ncdu‘
  • sudo apt-get install ncdu
• Please refer to his link :
  • https://askubuntu.com/questions/356902/why-doesnt-this-show-the-hidden-files-folders

 

If i need to look up directory sizes  only till 1 layer, then the following command is useful:

Tip:

• the ‘-s’ flag is useful to summarize the contents on directory

 

$ du -csh *
52K Desktop
4.0K Documents
4.0K Downloads
0 media
4.0K Music
353M notebooks
4.0K Pictures
4.0K Public
4.0K Templates
947M transferLearning
4.0K Videos
1.3G total

References:

https://www.cyberciti.biz/faq/checking-directory-size-in-linux-command/

 

C# OutOfProcess Python

• WaitForExit Method (Int32):
  • true if the associated process has exited; otherwise, false.
  • https://msdn.microsoft.com/en-us/library/ty0d8k56(v=vs.110).aspx
• UseShellExecute:
  • true if the shell should be used when starting the process; false if the process should be created directly from the executable file. The default is true.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.useshellexecute(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardOutput:
  • true if output should be written to Process.StandardOutput; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardoutput(v=vs.110).aspx
• ProcessStartInfo.RedirectStandardError:
  • true if error output should be written to Process.StandardError; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandarderror(v=vs.110).aspx
• CreateNoWindow:
  • true if the process should be started without creating a new window to contain it; otherwise, false. The default is false.
  • https://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.createnowindow(v=vs.110).aspx
  • CreateNoWindow works in conjunction with UseShellExecute as follows:

    To run the process without any window:

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.CreateNoWindow = true; 
    info.UseShellExecute = false;
    Process processChild = Process.Start(info); 

    To run the child process in it’s own window (new console)

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = true; // which is the default value.
    Process processChild = Process.Start(info); // separate window

    To run the child process in the parent’s console window –-> This is what we are using

    ProcessStartInfo info = new ProcessStartInfo(fileName, arg); 
    info.UseShellExecute = false; // causes consoles to share window 
    Process processChild = Process.Start(info); 

• WindowStyle :
  • is a recommendation passed to the process
  • One of the enumeration values that indicates whether the process is started in a window that is maximized, minimized, normal (neither maximized nor minimized), or not visible. The default is Normal.

Git branch cleanups

• git remote prune origin –dry-run
• (Local) git branch -d <localbranch>

 

References:

• https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches#Tracking-Branches