The numpy np.corrcoef function is frustrating, because it returns the correlation value of every input vector with every other, when often we just want the single correlation coefficient between two vectors.
Returning all the coefficients, as numpy does, may also be inefficient.
Here are some timings for a custom correlation routine, to check whether it is faster, when avoiding calculating all the coefficients.
import numpy as np
import numpy.linalg as npl
vec1 = np.random.normal(size=(100000,))
vec2 = np.random.normal(size=(100000,))
We can use the IPython timeit magic command to time execution of np.corrcoef on these vectors:
%timeit np.corrcoef(vec1, vec2)[0, 1]
The formula for Pearson's product-moment correlation coefficient from wikipedia is:
$$ r = r_{xy} =\frac{\sum ^n _{i=1}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum ^n _{i=1}(x_i - \bar{x})^2} \sqrt{\sum ^n _{i=1}(y_i - \bar{y})^2}} $$where $\bar{x}$ is the mean:
$$ \bar{x} = \frac{1}{n} \sum ^n _{i=1} x_i $$def mycorr(x, y):
x_mean0 = x.ravel() - x.mean()
y_mean0 = y.ravel() - y.mean()
corr_top = x_mean0.dot(y_mean0)
# Use dot product on self to do sum of squares efficiently
corr_bottom = (np.sqrt(x_mean0.dot(x_mean0)) *
np.sqrt(y_mean0.dot(y_mean0)))
return corr_top / corr_bottom
%timeit mycorr(vec1, vec2)
The custom routine is twice as fast as np.corrcoef for this simple case of just two vectors.