Basis Function Regression

Check Linear Regression post here.

1. Basis Function

In mathematics, a basis function is an element of a particular basis for a function space. Every continuous function in the function space can be represented as a linear combination of basis functions, just as every vector in a vector space can be represented as a linear combination of basis vectors.

One trick you can use to adapt linear regression to nonlinear relationships between variables is to transform the data according to basis functions.

This is still a linear model, the linearity refers to the fact that the coefficients an never multiply or divide each other. 

What we have effectively done is taken our one-dimensional x values and projected them into a higher dimension, so that a linear fit can fit more complicated relationships between x and y

Polynomial basis functions

The PolynomialFeatures transformer has converted one-dimensional array into a three dimensional (polynominal) array by taking the exponent of each value. The new higher dimensional data representation can then be plugged into a linear regression.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

x = np.array([1, 2, 3, 4, 5])
y = np.array([4, 2, 1, 3, 7])
plt.scatter(x, y);
plt.show()
X = x[:, np.newaxis]
poly = PolynomialFeatures(degree=3, include_bias=False)
X2 = poly.fit_transform(X)
model = LinearRegression().fit(X2, y)
yfit = model.predict(X2)
plt.scatter(x, y)
plt.plot(x, yfit);
plt.show()

Gaussian basis functions

Other basis functions can be used. Instead of using a sum of polynomial try to use a sum of Gaussians.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

rng = np.random.RandomState(1)
x = 10 * rng.rand(50)
y = np.sin(x) + 0.1 * rng.randn(50)

center_of_gauss = np.linspace(x.min(), x.max(), 20).astype(np.float)
std = (center_of_gauss[1] - center_of_gauss[0])*0.7

def get_gauss(x):
    X = x[:, np.newaxis]
    gauss = np.exp(-0.5 * (((X-center_of_gauss)/1.41*std) ** 2))
    return gauss

model = LinearRegression().fit(get_gauss(x), y)
xfit = np.linspace(0, 10, 1000)
yfit = model.predict(get_gauss(xfit))
plt.scatter(x, y)
plt.plot(xfit, yfit)
plt.xlim(0, 10);
plt.show()