These two articles helped clear up a lot of my confusion:
python
Ubuntu Docker + python3
I recently had to do a quick test of using python with ubuntu. I decided to use docker.
steps:
sudo docker run -it ubuntu bash
apt-get update
apt-get install python3-pip
# python3 --version
Python 3.8.5
to load up other stuff
sudo docker run -it -v $HOME:/work pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel bash
sudo docker run -it --ipc=host --rm -v $HOME:/work --privileged pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel bash
sudo docker run -it -v $HOME:/work py37_pytorch16_dte bash
sudo docker run -it -v $HOME:/work py37_trch16_trfmr43 bash
Beyond Integer indexing
Faced an interesting problem recently
a : (B, S, T)
b : (B, C) where 0 <= x[i, j] < S
What I want is an array of shape (B, C, T)
a = np.array(
...: [[[0,1,2,3],
...: [4,5,6,7],
...: [8,9,10,11]],
...: [[0,1,2,3],
...: [4,5,6,7],
...: [8,9,10,11]]])
b = np.array(
...: [[0,2,2],
...: [1,0, 2]])
a.shape
Out[79]: (2, 3, 4)
b.shape
Out[80]: (2, 3)
What I expect is this
array([[[ 0, 1, 2, 3],
[ 8, 9, 10, 11],
[ 8, 9, 10, 11]],
[[ 4, 5, 6, 7],
[ 0, 1, 2, 3],
[ 8, 9, 10, 11]]])
Note this is different from the typical scenario
Initially I hit some issues with integer index broadcasting. It seems it is possible to do it.
a[np.array([np.arange(2)]).T, b]
References:
Numpy RuntimeWarning
Something I learns recently..
- NumPy has its own internal warning architecture on top of Pythons, which can be specifically controlled
- So, something Numpy will just produce a
RuntimeWarning
without actually throwing an exception
Consider this:
probs = np.array([0.0, 1.0]) np.prod(probs)**(-1/len(probs))
Numpy produces a RuntimeWarning, not an exception
References:
Python output to JSON
I was recently working with multi-lingual data.
So it became essential to dump output in a format/language i could read, rather than random UTF-16 style
Tip:
BERT Trials
A very simple and up-to-date explanation of BERT
logging.basicConfig
It was quite interesting for me to learn about the Root Logger inside python logging package
https://www.pylenin.com/blogs/python-logging-guide/#logging-from-multiple-modules
Werkzeug
As I got more and more into Flask, I wanted to understand about Werkzeug.
Some helpful links:
- https://stackoverflow.com/questions/37004983/what-exactly-is-werkzeug
- https://stackoverflow.com/questions/14814201/can-i-serve-multiple-clients-using-just-flask-app-run-as-standalone
Azure ML SDK uses ngix + gUnicorn within its docker image:
Flask RESTful
Found these two posts very useful in understanding the Flask-Restful package
Converting Python objects to json.
Converting Python objects to json.
Very interesting!
https://blog.softhints.com/python-convert-object-to-json-3-examples/