du command (Linux)

Note:

• du command hides hidden files.  A more convenient command is ‘ncdu‘
  • sudo apt-get install ncdu
• Please refer to his link :
  • https://askubuntu.com/questions/356902/why-doesnt-this-show-the-hidden-files-folders

 

If i need to look up directory sizes  only till 1 layer, then the following command is useful:

Tip:

• the ‘-s’ flag is useful to summarize the contents on directory

 

$ du -csh *
52K Desktop
4.0K Documents
4.0K Downloads
0 media
4.0K Music
353M notebooks
4.0K Pictures
4.0K Public
4.0K Templates
947M transferLearning
4.0K Videos
1.3G total

References:

https://www.cyberciti.biz/faq/checking-directory-size-in-linux-command/

 

Ubuntu Terminal Smarts

I think using the Ubuntu Terminal in a smart way is crucial for day-to-day productivity.

I came across this nice link where several tricks are listed down and explained nicely.

https://help.ubuntu.com/community/UsingTheTerminal

Note: The consistent Copy-Paste mechanism  I have found working across Unity terminal, Xfce terminal, unity gvim, xfce gvim etc is  using:

• Control + Insert : to copy
  • this is also same as ‘copy on select’ i.e. just select the text and it will be copied automatically.
• Shift + Insert : to paste

Check it out!

Ubuntu Learnings

Remote to Ubuntu 16.04 from Windows 10:

• http://ubuntuhandbook.org/index.php/2016/07/remote-access-ubuntu-16-04/
  • VNC approach : This shows the trick of how to get past the encyption error
  • I found that uVNC was much better than TightVNC for viewing
• atom editor has some issues over rdp. but there is a hack around it
  • https://github.com/atom/atom/issues/4360
• http://www.makeuseof.com/tag/how-to-establish-simple-remote-desktop-access-between-ubuntu-and-windows/
• http://www.technig.com/remote-access-windows-10-via-ubuntu-vise-versa/
• http://www.tweaking4all.com/software/linux-software/use-xrdp-remote-access-ubuntu-14-04/
  • RDP approach : this approach would use the the xfce desktop environment 
• http://c-nergy.be/blog/?p=9962
  • RDP approach : apparently you can get the Unity desktop using these steps. But I havent tried it out

Copy On Select:

• http://askubuntu.com/questions/211292/a-terminal-which-provides-select-to-copy-and-right-click-to-paste
• http://superuser.com/questions/307339/can-i-make-ubuntus-terminal-program-automatically-copy-to-the-clipboard-selecte
  • Anything you highlight in Linux, regardless of the program, is put into a special clipboard buffer, which you can paste using your mouse’s middle (wheel) button

Install Java

• http://www.wikihow.com/Install-Oracle-Java-JDK-on-Ubuntu-Linux
  • Only care about JDK.

profile and bashrc files

• http://bencane.com/2013/09/16/understanding-a-little-more-about-etcprofile-and-etcbashrc/
  • This is a very interesting read!

abgoswam@abgoswam-ubuntu:~$ ls .bash* -lh
-rw------- 1 abgoswam abgoswam 4.1K Oct 23 15:20 .bash_history
-rw-r--r-- 1 abgoswam abgoswam  220 Oct 21 22:44 .bash_logout
-rw-r--r-- 1 abgoswam abgoswam 3.7K Oct 21 22:44 .bashrc

abgoswam@abgoswam-ubuntu:~$ ls .profile* -lh
-rw-r--r-- 1 abgoswam abgoswam 655 Oct 21 22:44 .profile

abgoswam@abgoswam-ubuntu:~$ ls -lh /etc/profile*
-rw-r--r-- 1 root root  670 Oct 23 14:55 /etc/profile
-rw-r--r-- 1 root root  884 Oct 23 15:18 /etc/profile.save

/etc/profile.d:
total 20K
-rw-r--r-- 1 root root   40 Nov 30  2015 appmenu-qt5.sh
-rw-r--r-- 1 root root  101 Jun 29 12:03 apps-bin-path.sh
-rw-r--r-- 1 root root  663 May 18 02:19 bash_completion.sh
-rw-r--r-- 1 root root 1003 Dec 29  2015 cedilla-portuguese.sh
-rw-r--r-- 1 root root 1.9K Mar 16  2016 vte-2.91.sh

abgoswam@abgoswam-ubuntu:~$ ls -lh /etc/bash*
-rw-r--r-- 1 root root 2.2K Aug 31  2015 /etc/bash.bashrc
-rw-r--r-- 1 root root   45 Aug 12  2015 /etc/bash_completion

Gvim

• http://superuser.com/questions/10588/how-to-make-cut-copy-paste-in-gvim-on-ubuntu-work-with-ctrlx-ctrlc-ctrlv
• https://www.quora.com/How-do-you-work-out-where-the-vimrc-file-is-in-Ubuntu

Making Gvim auto open in new tab:

• http://lifepluslinux.blogspot.com/2014/09/open-file-in-existing-gvim-in-new-tab.html
• http://vim.wikia.com/wiki/Launch_files_in_new_tabs_under_Unix  (just create an alias)
  • the gvim.desktop way of doing it never worked for me

 

Spark:

• http://askubuntu.com/questions/635265/how-do-i-get-pyspark-on-ubuntu
• https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python
• https://www.tutorialspoint.com/apache_spark/apache_spark_installation.htm
• https://www.youtube.com/watch?v=uhVYTNEe_-A

Ubuntu Linux Installation Troubleshoot

I tried installing Ubuntu 16.04 on my Lenovo X1 Carbon. It’s interesting how hard it was to install Linux on this machine.

USB stick:

• Not possible! I spent several frustrating hours trying it out. Looks like the option to boot from a USB stick is not there in the BIOS.
• The Ubuntu installation guide does mention that there might be issues installing from USB stick.
• https://help.ubuntu.com/16.04/installation-guide/

NetBoot:

• Took me back to my days at Cisco, doing all the gory networking stuff.
  • This is the next thing I tried out, and I spent countless hours on it!
• https://help.ubuntu.com/community/TFTP
• http://cdimage.ubuntu.com/netboot/16.04.1/?_ga=1.73326611.1880033183.1476136066
• https://help.ubuntu.com/lts/installation-guide/armhf/ch04s03.html
• https://help.ubuntu.com/community/Installation#Server and network installations
• Linux:
  • http://www.ostechnix.com/how-to-install-pxe-server-on-ubuntu-16-04/
  • http://www.ostechnix.com/install-dhcp-server-in-ubuntu-16-04/
  • https://help.ubuntu.com/community/Installation/LocalNet
• Windows:
  • https://help.ubuntu.com/community/Installation/WindowsServerNetboot

 

 

Netstat, Telnet

In my past life I was so used to using the ‘netstat’ command. So I found it funny when I got to re-use it again after ~5 years.

Netstat:

• netstat -nltp
• netstat -rn

netstat [address_family_options] [–tcp|-t] [–udp|-u] [–raw|-w] [–listening|-l] [–all|-a] [--numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–symbolic|-N] [–extend|-e[–extend|-e]] [–timers|-o] [–program|-p] [–verbose|-v] [–continuous|-c]

netstat {–route|-r} [address_family_options] [–extend|-e[–extend|-e]] [–verbose|-v] [–numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–continuous|-c]

 

There are so many more flags (see reference)

Telnet:

• used telnet to verify connectivity with zookeeper
• at the telnet prompt, type in ruok and press ENTER. if everything’s fine, ZooKeeper will say imok and end the Telnet session

References:

• http://www.computerhope.com/unix/unetstat.htm
• http://www.computerhope.com/unix/utelnet.htm

Using Vowpal Wabbit : Tips and Tricks

As I play around more with the machine learning toolkit Vowpal Wabbit, I realized there are several subtle flags/functionality in the toolkit.

Here is an effort to aggregate my leanings in using this toolkit.

Tips:

• -q [ –quadratic ] arg
  • Create and use quadratic features
  • -q is a very powerful option. It takes as an argument a pair of two letters. Its effect is to create interactions between the features of two namespaces. Suppose each example has a namespaceuser and a namespace document, then specifying -q ud will create an interaction feature for every pair of features (x,y) where x is a feature from the user namespace and y is a feature from the document namespace. If a letter matches more than one namespace then all the matching namespaces are used. In our example if there is another namespace url then interactions between url and document will also be modeled. The letter : is a wildcard to interact with all namespaces. -q a: (or -q :a) will create an interaction feature for every pair of features (x,y)where x is a feature from the namespaces starting with a and y is a feature from the all namespaces. -q :: would interact any combination of pairs of features.
• –print
  • Use this to understand how VW constructs   the highlighted number of features.

• When using contextual bandit mode, you will notice it gets added automatically per action
• Feature ‘116060’
  • This is a constant feature with value 1, that essentially captures the intercept term in a linear model.
  • You may come across this feature if you look closely into the VW output.
• Output Feature Ranking and Feature Weights Using VW
  • Is it possible to output the feature rankings after every update ?
    • try –audit
  • Is it possible to output the feature rankings at the end of training ?
    • use a combination of –-readable_model  foo and –l1 1e-3.  Any features surviving a high level of l1 regularization must be important according to the gradient.
  • Is it possible to output the feature weights after every update ?
    • possible, but this will be expensive.  In between each example you can put in a special examples with a ‘tag’ that says save_<filename>..
  • Is it possible to output the feature weights at the end of training ?
    •  that’s what –f does.  If you want it in a readable format then use –readable_model.
• The learning rate schedule is influenced by 4 parameters, outlined here.
  • Look into the code below regarding how I am doing parameter sweeps over these 4 parameters.
  • 
• More to come…

 

References:

1. https://github.com/datasciencedojo/meetup/tree/master/getting_started_with_vowpal_wabbit
2. http://mlwave.com/tutorial-titanic-machine-learning-from-distaster/
3. http://zinkov.com/posts/2013-08-13-vowpal-tutorial/
4. http://stackoverflow.com/questions/24822288/correctness-of-logistic-regression-in-vowpal-wabbit

 

Code:

• parameter_sweeps

Bash Scripting on Windows

There are a couple of ways to make bash scripts run on windows (cygwin).  This is to take care of the \r\n issue that comes up. File editors in windows use \r\n to denote end of line.

• Modify file ending using Notepad++  (Edit->EOL Conversion)
• Run the Dos2Unix tool to make the file Unix compatible.

References:

• https://danieladeniji.wordpress.com/2013/03/07/microsoft-windows-cygwin-error-r-command-not-found/
• http://stackoverflow.com/questions/15736898/running-a-shell-script-through-cygwin-on-windows
• Variable Substitution
  • http://tldp.org/LDP/abs/html/varsubn.html
• Loops:
  • http://tldp.org/LDP/abs/html/nestedloops.html

 

Code:

• parameter_sweeps.sh