# January 29, 2014

## Some Non-Linear Functions that can be Linearized for Least-Square Regression

The reason polynomial regression works is because it is able to create a system of linear equations. Despite the fact polynomials are not all linear, the method for solving the coefficients generates linear equations. Consider the following polynomial:

The coefficients for this equation (*a*, *b*, and *c*) can be determined if there are three known *x* and corresponding *y* values. For a moment let us forget about the regression part and focus on solving the system exactly. We have three unknown coefficients, and thus will require three different points to solve this equation. Let us solve the coefficients for the known points: (5,38); (10,123); and (15,258). First, create 3 equations from the known points by substituting in the known values of *x* and *y*:

Expand and arrange:

The three resulting equations are all linear, and therefore we can solve this system of equation using a matrix. Turn the system of equations into matrix form:

Solve using matrix inversion:

(Solution)

The resulting coefficients are *a*=1, *b*=2, *c*=3.

So despite the fact the starting equation is not linear, the known values are for the non-linear terms. We could rewrite the starting equation as follows:

Where all upper case letters are constants. In this case *A* = *x*^{2}, *B*=*x*. When written this way, the equation is clearly linear. This is why we can find coefficients for a non-linear polynomial equation with polynomial regression. All the non-linear operations are preformed on the data before we solve for the coefficients.

Several non-linear equations can be linearized in this fashion. Take a simple example:

Since we are solving for *a* and *b* with known values of *x* and *y*, the values relating to *x* and *y* become constants. That means the natural log goes away, and the resulting equation is linear. We can demonstrate using two known points: (1, 3); and (e, 7). This will create the following system of equations:

Solve the logs:

Simplify:

Knowing the *b* = 3, we can quickly find *a* = 4 by substituting the known *b* into the second equation. Starting with the original equation one can see it will be linear fairly easily. Again, we can rewrite the starting equation so it looks linear:

Where *A* = ln *x*.

Sometimes, however, equations that don't look linear can be linearized. Take for instance:

This does not at all look like it is linear. However we can use some algebra to change things around. It doesn't matter what we do to *x* and *y* as they just turn into constants. We need to manipulate this equation doing whatever non-linear operations are required to get *a* and *b* in a linear form—or at least close. Here we want to get *b* and *x* out of the exponent. This can be accomplished by taking the natural log of both sides.

Split using log identities.

Pull out the exponent using log identities.

And the natural log of *e* is 1, which results in:

This still isn't completely linear because we need to solve for *a*. However it is close.

This is linear and we can solve it for some known point. After we solve the system of equations, we need to solve for *a* using the value of *α*. Let us demonstrate using the points: (0,5) and (ln(5),125).

Place into a matrix:

And solve:

Now we have values for *α* and *b*. We just need to use the value for *A* to solve for *a*.

Take the exponential of each side:

Reduce:

Now fill in the known value of *α*:

Solve:

So despite the equation being non-linear, it can be solved in a linear fashion.

This is useful because if we can do this for an ordinary system of equations, we can also do it for an overdetermined system like we do for polynomial regression. Let us take the an earlier example and solve an overdetermined case using least-squares.

First, the equation:

The square of the residual:

Substitute in the function *f*:

To minimize the square of the residual, take the derivative and set it equal to zero:

Set this to zero:

Factor out the constant 2:

The two goes away when we divide two into both sides. Now break up the summations:

Pull out the constants and reduce:

Rearrange a bit:

And place into matrix form:

We now have a least-square solution to our starting function. This isn't too different from our normal equation for linear regression. In fact, we can place any two coefficient equation that can be linearized into least-square regression. The input must have the form:

Where each of the terms is a function of their lower-case part. We we work out the least-square math, the result will be:

We can use this as a template to solve for an other function we can make linear:

We already saw that this can be turned into:

Now place calculate the variables:

And place this in our matrix:

Note that the matrix when solved will give us the natural log of *a*. That will have to be undone in order after the matrix has been solved by taking the exponential of ln( *a *). Normally I would leave the equation in matrix form, but let us take this one step further so the this last step can be demonstrated.

Solve:

Take out of matrix form:

Now to solve for *a* in the second equation, take the exponential of both sides.

Where exp( *x* ) = *e*^{x}.

Here is an example plot of the algorithm being used on a set of 50 noisy data points.

So we have a method to linearize some two coefficient non-linear functions and preform least-square regression using them. I was looking for some equations that had more than two coefficients I could demonstrate but did not find any. That will have to wait for an other day.