如何使用交叉验证来确定使用训练、验证和测试集的最终模型

Question

*种子为重复性设置为123，我运行的是3.63 R。

library(pacman)
pacman::p_load(readr, caret, tidyverse, dplyr)

# Download the dataset:
download.file("https://raw.githubusercontent.com/christianmckinnon/StackQ/master/LifeExpectancyData.csv", "LifeExpectancyData.csv")

# Read in the data:
led <-read_csv("LifeExpectancyData.csv")

# Check for NAs
sum(is.na(led))
# Set all NAs to 0
led[is.na(led)] <- 0

# Rename `Life expectancy` to life_exp to avoid using spaces
led <-led %>% rename(life_exp = `Life expectancy`)

# Partition training and test sets
set.seed(123, sample.kind = "Rounding")
test_index <- createDataPartition(y = led$life_exp, times = 1, p = 0.2, list = F)
led_train <- led[-test_index,]
led_test <- led[test_index,]

# Add RMSE as unit of error measurement
RMSE <-function(true_ratings, predicted_ratings){
  sqrt(mean((true_ratings - predicted_ratings)^2))
}

# Create a linear model
led_lm <- lm(life_exp ~ GDP, data = led_train)
# Create prediction
lm_preds <-predict(led_lm, led_test)
# Check RMSE
RMSE(led_test$life_exp, lm_preds)
# The linear Model achieves an RMSE of 9.81141

# Create a Random Forest Model with Repeated Cross Validation
led_cv <- trainControl(method = "repeatedcv", number = 5, repeats = 3,
                      search = "random")
# Set the seed for reproducibility:
set.seed(123, sample.kind = "Rounding")
train_rf <- train(life_exp ~ GDP, data = led_train,
                  method = "rf", ntree = 150, trControl = led_cv,
                  tuneLength = 5, nSamp = 1000, 
                  preProcess = c("center","scale"))
# Create Prediction
rf_preds <-predict(train_rf, led_test)
# Check RMSE
RMSE(led_test$life_exp, rf_preds)
# The rf Model achieves an RMSE of 9.828415

# kNN Model:
knn_cv <-trainControl(method = "repeatedcv", repeats = 1)
# Set the seed for reproducibility:
set.seed(123, sample.kind = "Rounding")
train_knn <- train(life_exp ~ GDP, method = "knn", data = led_train,
                   tuneLength = 10, trControl = knn_cv,
                   preProcess = c("center","scale"))
# Create the Prediction:
knn_preds <-predict(train_knn, led_test)
# Check the RMSE:
RMSE(led_test$life_exp, knn_preds)
# The kNN model achieves the lowest RMSE of 8.923281

Answer 1

我的方法如下。最终的模型应该使用所有的数据。我不确定是什么动机不包括所有的数据在最终的模型。你只是在丢弃预测力。

对于交叉验证，只需将数据拆分为训练和测试数据即可。然后为完整模型选择性能最好的建模方法，再创建完整模型。

当前代码的更大问题是交叉验证方法可能会导致两个结果：虚假的准确性和潜在的虚假模型比较。在交叉验证中需要处理时间自相关。例如，如果我的训练数据集有英国2014年和2016年的特征，那么你就会期望像一个随机森林这样的东西能够高精度地预测2015年的预期寿命。这可能就是您在当前交叉验证类型中所要测量的全部内容。最好创建一个独立的数据集，以便训练和测试中的国家是不同的，或者将其划分为明显不同的时间段。确切的方法取决于您希望模型预测的内容