scatterplot with correlation coefficient , p-value

Single

from scipy import stats

result_0928 = stats.pearsonr(df_0928["dt_1"], y=df_0928["prev_deltaMs"])
print("coef: {0}".format(result_0928.statistic))
print("p-value: {0}".format(result_0928.pvalue))

import matplotlib.pyplot as plt

plt.scatter(df_0928["dt_1"], 
            df_0928["prev_deltaMs"], 
            c ="blue",
            label="pearson coef:{0} p-value:{1}".format(round(result_0928.statistic, 3), round(result_0928.pvalue,8)))
            
plt.xlabel("dt_1")
plt.ylabel("prev_deltaMs")
plt.title("09/28.  pearson coef:{0} p-value:{1}".format(
    round(result_0928.statistic, 3), 
    round(result_0928.pvalue, 7)))

# plt.legend()
plt.show()

Subplots

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 4))

ax[0].scatter(df_0928["dt_1"], df_0928["prev_deltaMs"])
ax[0].title.set_text("09/28.  pearson coef:{0} p-value:{1}".format(
    round(result_0928.statistic, 3), 
    round(result_0928.pvalue, 15)))
ax[0].set(xlabel='dt_1', ylabel='prev_deltaMs')
    

ax[1].scatter(df_0928["dt_1"], df_0928["prev_deltaMs"])
ax[1].title.set_text("09/28.  pearson coef:{0} p-value:{1}".format(
    round(result_0928.statistic, 3), 
    round(result_0928.pvalue,15)))
ax[1].set(xlabel='dt_1', ylabel='prev_deltaMs')
    
fig.subplots_adjust(wspace=.4)    
plt.show()

Histogram matplotlib

see the counts (accept_n) and edge boundaries (accept_bins) when plotting histograms in matplotlib

N=50
M=max(data_accept.dt_1)
print(M)
accept_n, accept_bins, _ = plt.hist(data_accept.dt_1, N, range=[0, N], label='accept') 
reject_n, reject_bins, _ = plt.hist(data_reject.dt_1, N, range=[0, N], label='reject')
print(accept_n)
print(accept_bins)
print(reject_n)
print(reject_bins)
plt.legend()
plt.show()

References

• https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.hist.html
• https://pythonspot.com/matplotlib-histogram/
• https://www.geeksforgeeks.org/matplotlib-pyplot-hist-in-python/

ASCII, Unicode, UTF-8 hell

These two articles helped clear up a lot of my confusion:

• https://naveenr.net/unicode-character-set-and-utf-8-utf-16-utf-32-encoding/
• https://pynative.com/python-json-encode-unicode-and-non-ascii-characters-as-is/

Ubuntu Docker + python3

I recently had to do a quick test of using python with ubuntu. I decided to use docker.

steps:

sudo docker run -it ubuntu bash

apt-get update
apt-get install python3-pip

# python3 --version
Python 3.8.5

to load up other stuff

sudo docker run -it -v $HOME:/work pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel bash

sudo docker run -it --ipc=host --rm -v $HOME:/work --privileged pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel bash

sudo docker run -it -v $HOME:/work py37_pytorch16_dte bash

sudo docker run -it -v $HOME:/work py37_trch16_trfmr43 bash

Beyond Integer indexing

Faced an interesting problem recently

a : (B, S, T)
b : (B, C)  where 0 <= x[i, j] < S

What I want is an array of shape (B, C, T)

a = np.array(
   ...:    [[[0,1,2,3], 
   ...:      [4,5,6,7],
   ...:      [8,9,10,11]],
   ...:     [[0,1,2,3],
   ...:      [4,5,6,7],
   ...:      [8,9,10,11]]])

b = np.array(
   ...:    [[0,2,2],
   ...:     [1,0, 2]])

a.shape
Out[79]: (2, 3, 4)

b.shape
Out[80]: (2, 3)

What I expect is this

array([[[ 0,  1,  2,  3],
        [ 8,  9, 10, 11],
        [ 8,  9, 10, 11]],
       [[ 4,  5,  6,  7],
        [ 0,  1,  2,  3],
        [ 8,  9, 10, 11]]])

Note this is different from the typical scenario

Initially I hit some issues with integer index broadcasting. It seems it is possible to do it.

a[np.array([np.arange(2)]).T, b]

References:

• https://stackoverflow.com/questions/46124469/shape-mismatch-indexing-arrays-could-not-be-broadcast-together-with-shapes
• https://docs.scipy.org/doc/numpy-1.13.0/user/basics.indexing.html#indexing-multi-dimensional-arrays

PyTest live logging in PyCharm

PyTest does allow output to be ‘live printed’ 

Also, it possible to see logging output in PyTest

Checkout these two links:

• https://intellij-support.jetbrains.com/hc/en-us/community/posts/360007644040-Show-logging-output-when-using-pytest
• https://docs.pytest.org/en/latest/capture.html#setting-capturing-methods-or-disabling-capturing

Updating the transformers package

Few steps I follow each time i update the transformers package

git pull
pip install --upgrade .
pip install -r ./examples/requirements.txt

that’s

Latex multirow and column

Short and intuitive example

• https://texblog.org/2012/12/21/multi-column-and-multi-row-cells-in-latex-tables/

Numpy RuntimeWarning

Something I learns recently..

• NumPy has its own internal warning architecture on top of Pythons, which can be specifically controlled
• So, something Numpy will just produce a RuntimeWarning  without actually throwing an exception

Consider this:

probs = np.array([0.0, 1.0])
np.prod(probs)**(-1/len(probs))

Numpy produces a RuntimeWarning, not an exception

References:

• https://stackoverflow.com/questions/20161899/scipy-curve-fit-error-divide-by-zero-encountered
• https://stackoverflow.com/questions/10519237/python-how-to-avoid-numpy-runtimewarning-in-function-definition

Gradient accumulation in PyTorch

Need to understand:

• https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255
• https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20
• https://gist.github.com/thomwolf/ac7a7da6b1888c2eeac8ac8b9b05d3d3