ASCII, Unicode, UTF-8 hell

These two articles helped clear up a lot of my confusion:

• https://naveenr.net/unicode-character-set-and-utf-8-utf-16-utf-32-encoding/
• https://pynative.com/python-json-encode-unicode-and-non-ascii-characters-as-is/

Ubuntu Docker + python3

I recently had to do a quick test of using python with ubuntu. I decided to use docker.

steps:

sudo docker run -it ubuntu bash

apt-get update
apt-get install python3-pip

# python3 --version
Python 3.8.5

to load up other stuff

sudo docker run -it -v $HOME:/work pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel bash

sudo docker run -it --ipc=host --rm -v $HOME:/work --privileged pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel bash

sudo docker run -it -v $HOME:/work py37_pytorch16_dte bash

sudo docker run -it -v $HOME:/work py37_trch16_trfmr43 bash

Beyond Integer indexing

Faced an interesting problem recently

a : (B, S, T)
b : (B, C)  where 0 <= x[i, j] < S

What I want is an array of shape (B, C, T)

a = np.array(
   ...:    [[[0,1,2,3], 
   ...:      [4,5,6,7],
   ...:      [8,9,10,11]],
   ...:     [[0,1,2,3],
   ...:      [4,5,6,7],
   ...:      [8,9,10,11]]])

b = np.array(
   ...:    [[0,2,2],
   ...:     [1,0, 2]])

a.shape
Out[79]: (2, 3, 4)

b.shape
Out[80]: (2, 3)

What I expect is this

array([[[ 0,  1,  2,  3],
        [ 8,  9, 10, 11],
        [ 8,  9, 10, 11]],
       [[ 4,  5,  6,  7],
        [ 0,  1,  2,  3],
        [ 8,  9, 10, 11]]])

Note this is different from the typical scenario

Initially I hit some issues with integer index broadcasting. It seems it is possible to do it.

a[np.array([np.arange(2)]).T, b]

References:

• https://stackoverflow.com/questions/46124469/shape-mismatch-indexing-arrays-could-not-be-broadcast-together-with-shapes
• https://docs.scipy.org/doc/numpy-1.13.0/user/basics.indexing.html#indexing-multi-dimensional-arrays

Numpy RuntimeWarning

Something I learns recently..

• NumPy has its own internal warning architecture on top of Pythons, which can be specifically controlled
• So, something Numpy will just produce a RuntimeWarning  without actually throwing an exception

Consider this:

probs = np.array([0.0, 1.0])
np.prod(probs)**(-1/len(probs))

Numpy produces a RuntimeWarning, not an exception

References:

• https://stackoverflow.com/questions/20161899/scipy-curve-fit-error-divide-by-zero-encountered
• https://stackoverflow.com/questions/10519237/python-how-to-avoid-numpy-runtimewarning-in-function-definition

Python output to JSON

I was recently working with multi-lingual data.

So it became essential to dump output in a format/language i could read, rather than random UTF-16 style

Tip:

• https://stackoverflow.com/questions/12309269/how-do-i-write-json-data-to-a-file

BERT Trials

A very simple and up-to-date explanation of BERT

• Painless Fine-Tuning of BERT in Pytorch

• NER with BERT in Action

View at Medium.com

logging.basicConfig

It was quite interesting for me to learn about the Root Logger  inside python logging package

https://www.pylenin.com/blogs/python-logging-guide/#logging-from-multiple-modules

 

Werkzeug

As I got more and more into Flask, I wanted to understand about Werkzeug.

Some helpful links:

• https://stackoverflow.com/questions/37004983/what-exactly-is-werkzeug
• https://stackoverflow.com/questions/14814201/can-i-serve-multiple-clients-using-just-flask-app-run-as-standalone

Azure ML SDK  uses ngix + gUnicorn  within its docker image:

• https://liupeirong.github.io/amlDockerImage/

 

 

Flask RESTful

Found these two posts very useful in understanding the Flask-Restful package

• https://blog.miguelgrinberg.com/post/designing-a-restful-api-with-python-and-flask
• https://blog.miguelgrinberg.com/post/designing-a-restful-api-using-flask-restful

Converting Python objects to json.

Converting Python objects to json.

Very interesting!

https://blog.softhints.com/python-convert-object-to-json-3-examples/