Machine Learning:Basic Conception

How did the error come？

It comes from bias and variance.

What is estimator

We do not know $\hat{f}$.From training data,we get $f^{\ast }$.And $f^{\ast }$ is an estimator of $\hat{f}$.It depends on bias+variance.

How to get mean in data?

We can select a set of sub-data.Like { x 1 , x 2 , . . . , x N } \{x^1,x^2,...,x^N\} {x1,x2,...,xN}.We assume the mean of the data is $\mu$.We can get below:
$$m=\frac{1}{N}\sum _n{x}^n\ne \mu$$
$$E[m]=E[\frac{1}{N}\sum _n{x}^n]=\frac{1}{N}\sum _{n}E[x^n]=\mu$$

How to get variance in data?

Actually the same as mean.We can get below:
$$m=\frac{1}{N}\sum _n{x}^n$$
$$s^2=\frac{1}{N}\sum _n(x^n-m{)}^2$$
$$E[s^2]=\frac{N-1}{N}{\sigma }^2$$
And we know that if the larger the $N$,$E[s^2]$ is more equal to ${\sigma }^2$.We need low variance and low bias.

Bias is import

There is something can explain this:
$$E[f^{\ast }]=\overline{f}\ne \hat{f}$$
There is a distance between $\overline{f}$ and $\hat{f}$ called bias.Bias will keep your model away from the real model.But variance will not.

Error is build by bias and variance

Bias like the distance between $\overline{f}$and$\hat{f}$.And variance like different models we design,which are influenced by training data.

The complexity and variance

Actually, more complexity the model is,larger variance is .And complex model will be seriously influenced by training data.

The complexity and bias

Actually, more complexity the model is,smaller bias is.So we can get a conclusion:small complexity,large bias and small variance,large conplexity,small bias and large variance!

The explanation to overfitting

The origin of over-fitting is because the complexity is too high, although the bias becomes smaller but the variance becomes larger, resulting in too large error.Of course, if you handle the complexity well, you can control the bias and variance to minimize the error.

Underfitting

The reason for underfitting is that the complexity is too small, which leads to too big bias and too big error, but the error value caused by underfitting will be much smaller than overfitting.

How to know that our model is overfitting or underfitting?

If the error between the training data and your model is high, intuitively speaking, the model cannot cover the training data well, which is underfitting.If your model can cover the training data well, but the error value of the test data is a bit large, it is overfitting.

How can we handle overfitting or underfitting?

If our model underfitting,we need to redesign.Like add more features as input.A more complex model.If our model overfitting,we can add more trainning data.It very effective,but not practical.Or Regularization.It means make our model more smooth.But it will influence bias.

What you should concern

Testing set is not real ‘Testing Set’.Two different set will generate the different error.Therefore,testing data has public testing data and private testing data.We have public testing data,but not private testing data.Maybe you get the low error in public testing data.But in private testing data,our error will be larger.

How can we do it better?

We need to consider Cross Validation.

What is Cross Validation?

We need to separate part of the training data into verification data, which can reduce the impact of private testing data.

Do not tune your model by public testing data!!!

If you find that the error value of public testing data is too large, go back and modify the model. This is very not recommended, because you regard public testing data more heavily than private testing data.

How can we do it better?

Another validation called ‘N-fold Cross Validation’.

N-fold Cross Validation

Divide the training data into N pieces for traversal. The process is to modify one of them into verification data, get the error value for each model, and generate the error array of the model, and then calculate the average value of the error array for comparison, the error is the smallest Is the model we choose, and then verify the public testing data.