Orange Machine Learning (Python), the charm of Machine Learning

I asked about a good visualization tool on Kaggle, and D33B pointed out to Orange. Despite not being what I was asking for, checking the tool out revealed it to be awesome (M. Horbal felt that way too).

You will never win a kaggle competition with Orange, but it will certainly help you deal with data and build models very, very quickly and intuitively. In fact, I used it to quickly build a prototype model for a company's data which was very nasty (loads of missing values, numbers with quotation marks...). I quickly build a linear regression and visualized some scatterplots and conditional distributions. All of it with this nice workflow

In the image above, the only datasource is the file element. For the analysis pipeline, we first select the attributes that have a number of values in the independent variables, since these variables incrementally have less and less values. We are interested in keeping most of the values and still visualize the relationship between the attributes. After selecting the attributes, we tell Orange to prune the data before injecting it into the several elements after that. We want to see the conditional distributions in case we can get rid of non-informative attributes, we want to study potential linear relationships, see their correlations via a distance map and perform a linear regression (also ridge and lasso). On the other hand, we also want to study the regressors in depth and for that we select only the regressor attrubutes in the pipeline below.

Definitely a piece of software to have in your toolset.

Pairs trading and colinearity

We sometimes recognize a couple of assets as being co-related. However, the dependence regime changes over time, making this co-relation non-linear and depending on, let's say, a phase. A more robust concept is multiple co-linearity, which implies that a linear combination of the returns of those assets is linearly related, and has a constant mean and variance.

Let's say that two assets are co-linear and that the returns of one of them have been consistently larger than the other. It makes sense to sell short the asset with larger past returns and buy the asset with smaller past returns. With this, we would have a quantitative model to measure large discrepancies of the return of the linear combination, for example, execute this strategy when the absolute value of the return exceeds twice the standard deviation. This gives a statistical arbitrage oportunity.

One example that I like to use is the pair EURUSD and GC (NYMEX Gold 100oz) and I used that to get a hold on some coins. Another perhaps more interesting application would be corn and wheat. They seem to have periods when one is the loved child of agricultural commodity traders. They are normally worth the around the same, being wheat historically more expensive. Corn catched up and had a period that was more expensive, but wheat had recently a rally and got to be USD100 more expensive per contract. Obviously, the gap close down to a difference of USD20. It then widened and is now sitting around USD40-USD50. This simple model would have yield a potentical change of USD80 per contract.

Least squares in C

Just for fun.

I show here how to perform simple regression with least squares.

We implement the closed formulae that result from
$$\min_{\alpha,\,\beta}Q(\alpha,\beta)$$
where
$$Q(\alpha,\beta) = \sum_{i=1}^n\hat{\varepsilon}_i^{\,2} = \sum_{i=1}^n (y_i - \alpha - \beta x_i)^2 $$
We thus compute
$$\frac{\partial}{\partial \beta} Q(\alpha,\beta)$$
and
$$\frac{\partial}{\partial \alpha} Q(\alpha,\beta)$$
Which results in
 $$\hat\beta  = \frac{ \sum_{i=1}^{n} (x_{i}-\bar{x})(y_{i}-\bar{y}) }{ \sum_{i=1}^{n} (x_{i}-\bar{x})^2 }
             = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 }
              = \frac{ \operatorname{Cov}[x,y] }{ \operatorname{Var}[x] }
$$

$$\hat\alpha  = \bar{y} - \hat\beta\,\bar{x}$$

So here is the code:

#include "stdafx.h"
#include <stdlib.h>

double sum(double *x, int n) {
	double s=0;

	for (int i=0; i<n; i++) {
		s+=x[i];
	}

	return s;
}


double dot(double *x, double *y, int n) {
	double s=0;

	for (int i=0; i<n; i++) {
		s+=x[i]*y[i];
	}

	return s;
}

void ols(double *x, double *y, int n, double *beta, double *alpha) {
	double sx = sum(x,n);
	double sy = sum(y,n);
	double sx2 = dot(x,x,n);
	double sxy = dot(x,y,n);
	double sxx = sx*sx;

	*beta = (sxy-sx*sy/n) / (sx2-sxx/n);
	*alpha = sy/n - *beta*sx/n;
}

And you can download the Visual C++ project here.