Netstat, Telnet

In my past life I was so used to using the ‘netstat’ command. So I found it funny when I got to re-use it again after ~5 years.

Netstat:

  • netstat -nltp
  • netstat -rn
netstat [address_family_options] [–tcp|-t] [–udp|-u] [–raw|-w] [–listening|-l] [–all|-a] [--numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–symbolic|-N] [–extend|-e[–extend|-e]] [–timers|-o] [–program|-p] [–verbose|-v] [–continuous|-c]
netstat {–route|-r} [address_family_options] [–extend|-e[–extend|-e]] [–verbose|-v] [–numeric|-n] [–numeric-hosts] [–numeric-ports] [–numeric-users] [–continuous|-c]

 

There are so many more flags (see reference)

Telnet:

  • used telnet to verify connectivity with zookeeper
  • at the telnet prompt, type in ruok and press ENTER. if everything’s fine, ZooKeeper will say imok and end the Telnet session

References:

Using Vowpal Wabbit : Tips and Tricks

As I play around more with the machine learning toolkit Vowpal Wabbit, I realized there are several subtle flags/functionality in the toolkit.

Here is an effort to aggregate my leanings in using this toolkit.

Tips:

  • -q [ –quadratic ] arg
    • Create and use quadratic features
    • -q is a very powerful option. It takes as an argument a pair of two letters. Its effect is to create interactions between the features of two namespaces. Suppose each example has a namespaceuser and a namespace document, then specifying -q ud will create an interaction feature for every pair of features (x,y) where x is a feature from the user namespace and y is a feature from the document namespace. If a letter matches more than one namespace then all the matching namespaces are used. In our example if there is another namespace url then interactions between url and document will also be modeled. The letter : is a wildcard to interact with all namespaces. -q a: (or -q :a) will create an interaction feature for every pair of features (x,y)where x is a feature from the namespaces starting with a and y is a feature from the all namespaces. -q :: would interact any combination of pairs of features.
  • –print
    • Use this to understand how VW constructs   the highlighted number of features.
    • When using contextual bandit mode, you will notice it gets added automatically per action
  • Feature ‘116060’
    • This is a constant feature with value 1, that essentially captures the intercept term in a linear model.
    • You may come across this feature if you look closely into the VW output.
  • Output Feature Ranking and Feature Weights Using VW
    • Is it possible to output the feature rankings after every update ?
      • try –audit
    • Is it possible to output the feature rankings at the end of training ?
      • use a combination of –-readable_model  foo and –l1 1e-3.  Any features surviving a high level of l1 regularization must be important according to the gradient.
    • Is it possible to output the feature weights after every update ?
      • possible, but this will be expensive.  In between each example you can put in a special examples with a ‘tag’ that says save_<filename>..
    • Is it possible to output the feature weights at the end of training ?
      •  that’s what –f does.  If you want it in a readable format then use –readable_model.
  • The learning rate schedule is influenced by 4 parameters, outlined here.
    • Look into the code below regarding how I am doing parameter sweeps over these 4 parameters.
    • capture
  • More to come…

 

References:

  1. https://github.com/datasciencedojo/meetup/tree/master/getting_started_with_vowpal_wabbit
  2. http://mlwave.com/tutorial-titanic-machine-learning-from-distaster/
  3. http://zinkov.com/posts/2013-08-13-vowpal-tutorial/
  4. http://stackoverflow.com/questions/24822288/correctness-of-logistic-regression-in-vowpal-wabbit

 

Code:

Bash Scripting on Windows

There are a couple of ways to make bash scripts run on windows (cygwin).  This is to take care of the \r\n issue that comes up. File editors in windows use \r\n to denote end of line.

  • Modify file ending using Notepad++  (Edit->EOL Conversion)
  • Run the Dos2Unix tool to make the file Unix compatible.

References:

 

Code:

Shuffling and Splitting Operations

Shuffling is a pretty interesting operation in several scenarios. And different languages / platforms have  interesting features using the shuffling  operation.

Examples:

  • Shuffle the contents of a C# List
  • Select random lines from file  (using ‘shuf’ command in Linux)

References:

Code:

DecisionController

Redis : Basics

Recently I have been playing around with Redis. In particular I have been trying how to integrate Redis with web services in Azure.

Redis is actually more like a data structures server, supporting very interesting data structures, and operations on them.

I am listing down some data structures, and corresponding when using these data structures.

  1. Strings
  2. Lists
  3. Sets
  4. Sorted Sets
    • Useful when fast access to the middle of a large collection of elements is important
  5. Hashes
    • Hashes are maps between string fields and string values, so they are the perfect data type to represent objects.
  6. Bit arrays
  7. Hyper Log Log

 

DECR, DECRBY, DEL, EXISTS, EXPIRE, GET, GETSET, HDEL, HEXISTS, HGET, HGETALL, HINCRBY, HKEYS, HLEN, HMGET, HMSET, HSET, HVALS, INCR, INCRBY, KEYS, LINDEX, LLEN, LPOP, LPUSH,LRANGE, LREM, LSET, LTRIM, MGET, MSET, MSETNX, MULTI, PEXPIRE, RENAME, RENAMENX, RPOP, RPOPLPUSH, RPUSH, SADD, SCARD, SDIFF, SDIFFSTORE, SET, SETEX, SETNX, SINTER, SINTERSTORE, SISMEMBER, SMEMBERS, SMOVE, SORT, SPOP, SRANDMEMBER, SREM, SUNION, SUNIONSTORE, TTL, TYPE, ZADD,ZCARD, ZCOUNT, ZINCRBY, ZRANGE, ZRANGEBYSCORE, ZRANK, ZREM, ZREMRANGEBYSCORE, ZREVRANGE, ZSCORE

 

Code on GitHub:

References:

Curl Command (Linux)

$ curl http://teamonewebapiusage.azurewebsites.net/api/products
[{“Id”:1,”Name”:”Tomato Soup”,”Category”:”Groceries”,”Price”:1.0},{“Id”:2,”Name”:”Yo-yo”,”Category”:”Toys”,”Price”:3.75},{“Id”:3,”Name”:”Hammer”,”Category”:”Hardware”,”Price”:16.99}]

$ curl http://teamonewebapiusage.azurewebsites.net/api/products/3
{“Id”:3,”Name”:”Hammer”,”Category”:”Hardware”,”Price”:16.99}

 

Some more examples:

 

 

 

  • When we use curl at the command line to access websites, the traffic doesn’t show up in Fiddler. This is because curl doesn’t use the system proxy by default.  Try this:

 

 

Example from the Decision Service Web API

  • $ curl -x 127.0.0.1:8888 -H “Content-Type: application/json” -X POST -d @data.json http://localhost:29074/api/decision
    {“Action”:4,”EventId”:”26808b79af3741169863168040109301″,”TimeStamp”:”2016-02-18T09:16:20.2329418Z”}

 

Reference:

[1]  http://www.yilmazhuseyin.com/blog/dev/curl-tutorial-examples-usage/

[2] http://stackoverflow.com/questions/14978411/http-post-and-get-using-curl-in-linux

[3] http://stackoverflow.com/questions/18352190/why-i-get-411-length-required-error