linux

Virtual environments in ubuntu using venv

abgoswam linux

Tip

Activate virtual environments in ubuntu using venv

https://linuxize.com/post/how-to-create-python-virtual-environments-on-ubuntu-18-04/

Notes:

One of my machine had python3.5. In order to use python 3.6 i had to do a couple of things:

vimrc and bashrc files

abgoswam linux

For the nth time, i have had to look up the location of the .vimrc and .bashrc files on windows . so here goes:

bashrc

  • Just open git bash and do ‘ls -la’ to see the .bashrc file

vimrc

  • C:\Program Files (x86)\Vim\_vimrc
    • (basically wherever u have installed gvim on your machine)

du command (Linux)

abgoswam linux

Note:

  • To find whats contributing to the disk usage on a specific filesystem (/dev/sda2)
    • This can be tricky because there would be other filesystems mounted underneath /
    • I have found the package ‘xdiskusage‘ very useful for this.
    • This problem is discussed here

 

  • If i need to look up directory sizes  only till 1 layer, then the following command is useful:
    • the ‘-s’ flag is useful to summarize the contents on directory

 

$ du -csh *
52K Desktop
4.0K Documents
4.0K Downloads
0 media
4.0K Music
353M notebooks
4.0K Pictures
4.0K Public
4.0K Templates
947M transferLearning
4.0K Videos
1.3G total

References:

https://www.cyberciti.biz/faq/checking-directory-size-in-linux-command/

 

Ubuntu Terminal Smarts

abgoswam devproductivity, linux

I think using the Ubuntu Terminal in a smart way is crucial for day-to-day productivity.

I came across this nice link where several tricks are listed down and explained nicely.

https://help.ubuntu.com/community/UsingTheTerminal

Note: The consistent Copy-Paste mechanism  I have found working across Unity terminal, Xfce terminal, unity gvim, xfce gvim etc is  using:

  • Control + Insert : to copy
    • this is also same as ‘copy on select’ i.e. just select the text and it will be copied automatically.
  • Shift + Insert : to paste

Check it out!

Ubuntu Learnings

abgoswam linux

Remote to Ubuntu 16.04 from Windows 10:

Copy On Select:

Install Java

profile and bashrc files

abgoswam@abgoswam-ubuntu:~$ ls .bash* -lh
-rw------- 1 abgoswam abgoswam 4.1K Oct 23 15:20 .bash_history
-rw-r--r-- 1 abgoswam abgoswam  220 Oct 21 22:44 .bash_logout
-rw-r--r-- 1 abgoswam abgoswam 3.7K Oct 21 22:44 .bashrc

abgoswam@abgoswam-ubuntu:~$ ls .profile* -lh
-rw-r--r-- 1 abgoswam abgoswam 655 Oct 21 22:44 .profile

abgoswam@abgoswam-ubuntu:~$ ls -lh /etc/profile*
-rw-r--r-- 1 root root  670 Oct 23 14:55 /etc/profile
-rw-r--r-- 1 root root  884 Oct 23 15:18 /etc/profile.save

/etc/profile.d:
total 20K
-rw-r--r-- 1 root root   40 Nov 30  2015 appmenu-qt5.sh
-rw-r--r-- 1 root root  101 Jun 29 12:03 apps-bin-path.sh
-rw-r--r-- 1 root root  663 May 18 02:19 bash_completion.sh
-rw-r--r-- 1 root root 1003 Dec 29  2015 cedilla-portuguese.sh
-rw-r--r-- 1 root root 1.9K Mar 16  2016 vte-2.91.sh

abgoswam@abgoswam-ubuntu:~$ ls -lh /etc/bash*
-rw-r--r-- 1 root root 2.2K Aug 31  2015 /etc/bash.bashrc
-rw-r--r-- 1 root root   45 Aug 12  2015 /etc/bash_completion

Gvim

Making Gvim auto open in new tab:

 

Spark:

Ubuntu Linux Installation Troubleshoot

abgoswam linux

I tried installing Ubuntu 16.04 on my Lenovo X1 Carbon. It’s interesting how hard it was to install Linux on this machine.

USB stick:

  • Not possible! I spent several frustrating hours trying it out. Looks like the option to boot from a USB stick is not there in the BIOS.
  • The Ubuntu installation guide does mention that there might be issues installing from USB stick.
  • https://help.ubuntu.com/16.04/installation-guide/

NetBoot:

 

 

Netstat, Telnet

abgoswam linux

In my past life I was so used to using the ‘netstat’ command. So I found it funny when I got to re-use it again after ~5 years.

Netstat:

  • netstat -nltp
  • netstat -rn
netstat [address_family_options] [–tcp|-t] [–udp|-u] [–raw|-w] [–listening|-l] [–all|-a] [--numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–symbolic|-N] [–extend|-e[–extend|-e]] [–timers|-o] [–program|-p] [–verbose|-v] [–continuous|-c]
netstat {–route|-r} [address_family_options] [–extend|-e[–extend|-e]] [–verbose|-v] [–numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–continuous|-c]

 

There are so many more flags (see reference)

Telnet:

  • used telnet to verify connectivity with zookeeper
  • at the telnet prompt, type in ruok and press ENTER. if everything’s fine, ZooKeeper will say imok and end the Telnet session

References:

Using Vowpal Wabbit : Tips and Tricks

abgoswam linux, machinelearning, vowpalwabbit

As I play around more with the machine learning toolkit Vowpal Wabbit, I realized there are several subtle flags/functionality in the toolkit.

Here is an effort to aggregate my leanings in using this toolkit.

Tips:

  • -q [ –quadratic ] arg
    • Create and use quadratic features
    • -q is a very powerful option. It takes as an argument a pair of two letters. Its effect is to create interactions between the features of two namespaces. Suppose each example has a namespaceuser and a namespace document, then specifying -q ud will create an interaction feature for every pair of features (x,y) where x is a feature from the user namespace and y is a feature from the document namespace. If a letter matches more than one namespace then all the matching namespaces are used. In our example if there is another namespace url then interactions between url and document will also be modeled. The letter : is a wildcard to interact with all namespaces. -q a: (or -q :a) will create an interaction feature for every pair of features (x,y)where x is a feature from the namespaces starting with a and y is a feature from the all namespaces. -q :: would interact any combination of pairs of features.
  • –print
    • Use this to understand how VW constructs   the highlighted number of features.
    • When using contextual bandit mode, you will notice it gets added automatically per action
  • Feature ‘116060’
    • This is a constant feature with value 1, that essentially captures the intercept term in a linear model.
    • You may come across this feature if you look closely into the VW output.
  • Output Feature Ranking and Feature Weights Using VW
    • Is it possible to output the feature rankings after every update ?
      • try –audit
    • Is it possible to output the feature rankings at the end of training ?
      • use a combination of –-readable_model  foo and –l1 1e-3.  Any features surviving a high level of l1 regularization must be important according to the gradient.
    • Is it possible to output the feature weights after every update ?
      • possible, but this will be expensive.  In between each example you can put in a special examples with a ‘tag’ that says save_<filename>..
    • Is it possible to output the feature weights at the end of training ?
      •  that’s what –f does.  If you want it in a readable format then use –readable_model.
  • The learning rate schedule is influenced by 4 parameters, outlined here.
    • Look into the code below regarding how I am doing parameter sweeps over these 4 parameters.
    • capture
  • More to come…

 

References:

  1. https://github.com/datasciencedojo/meetup/tree/master/getting_started_with_vowpal_wabbit
  2. http://mlwave.com/tutorial-titanic-machine-learning-from-distaster/
  3. http://zinkov.com/posts/2013-08-13-vowpal-tutorial/
  4. http://stackoverflow.com/questions/24822288/correctness-of-logistic-regression-in-vowpal-wabbit

 

Code:

Bash Scripting on Windows

abgoswam linux

There are a couple of ways to make bash scripts run on windows (cygwin).  This is to take care of the \r\n issue that comes up. File editors in windows use \r\n to denote end of line.

  • Modify file ending using Notepad++  (Edit->EOL Conversion)
  • Run the Dos2Unix tool to make the file Unix compatible.

References:

 

Code: