pipeline - 如何在Pipeline中使用XGBoost进行GPU训练？

Question

XGBoost如何用GPU进行训练？

结合了pipeline、FunctionTransformer、GridSearchCV、XGBoost做训练，能在XGBoost做训练中设置 device='cuda'开启训练，但是训练数据却无法送到 GPU，请问如何解决呢？

cat_columns = X.select_dtypes(include=['object']).columns.values # 注意：这里不要用train_df，否则会出现 KeyErr: Premium Amount
num_columns = X.select_dtypes(include=np.number).columns.values

print(cat_columns)
print(num_columns)

preprocess = ColumnTransformer([
    ('categorical', make_pipeline(transform_datatime, transform_drop_columns, SimpleImputer(strategy='most_frequent'), OneHotEncoder()), cat_columns),
    ('numerical', make_pipeline(transform_drop_columns, SimpleImputer(strategy='mean'), StandardScaler()), num_columns)
])

# 定义完整的pipeline
pipeline = Pipeline(steps=[
    ('preprocess', preprocess),
    ('regressor', xgbRegressor(tree_method='hist', objective='reg:squarederror', n_jobs=-1))
])

# 定义超参数网格
param_grid = {
    'regressor__n_estimators': [200, 300],
    'regressor__max_depth': [5, 7],
    'regressor__learning_rate': [0.05, 0.1],
    'regressor__subsample': [0.8, 1.0]
}

# 设置GridSearchCV
grid_search = GridSearchCV(pipeline, param_grid, cv=2, scoring='neg_mean_squared_error', verbose=2)

grid_search.fit(X, y)

参考了官方文档，但是依然无法将训练数据设置到 GPU 上。

params = dict()
params["device"] = "cuda"
params["tree_method"] = "hist"
Xy = xgboost.QuantileDMatrix(X, y)
xgboost.train(params, Xy)

Answer 1

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.model_selection import GridSearchCV
from xgboost import XGBRegressor
import numpy as np

# 选择分类和数值列
cat_columns = X.select_dtypes(include=['object']).columns.values
num_columns = X.select_dtypes(include=np.number).columns.values

print(cat_columns)
print(num_columns)

# 定义预处理步骤
preprocess = ColumnTransformer([
    ('categorical', make_pipeline(transform_datetime, transform_drop_columns, SimpleImputer(strategy='most_frequent'), OneHotEncoder()), cat_columns),
    ('numerical', make_pipeline(transform_drop_columns, SimpleImputer(strategy='mean'), StandardScaler()), num_columns)
])

# 定义完整的管道
pipeline = Pipeline(steps=[
    ('preprocess', preprocess),
    ('regressor', XGBRegressor(tree_method='hist', objective='reg:squarederror', n_jobs=-1))
])

# 定义超参数网格
param_grid = {
    'regressor__n_estimators': [200, 300],
    'regressor__max_depth': [5, 7],
    'regressor__learning_rate': [0.05, 0.1],
    'regressor__subsample': [0.8, 1.0]
}

# 设置 GridSearchCV
grid_search = GridSearchCV(pipeline, param_grid, cv=2, scoring='neg_mean_squared_error', verbose=2)

# 拟合模型
grid_search.fit(X, y)

Answer 2

在您的Pipeline中使用XGBoost进行GPU训练时，确保数据能够正确传递到GPU上，您需要确保几个关键点被正确设置。以下是如何在您的Pipeline中配置XGBoost以使用GPU进行训练的步骤和解释：

步骤 1: 安装支持GPU的XGBoost

首先，确保您安装了支持GPU的XGBoost版本。这通常意味着您需要安装xgboost-cuda或者通过特定的安装命令来确保CUDA支持。

pip install xgboost --install-option=--enable-cuda

步骤 2: 配置XGBoostRegressor以使用GPU

在您的Pipeline中，xgbRegressor需要被配置为使用GPU。虽然您已经在params中设置了device='cuda'，但这种方式在Pipeline中并不直接适用，因为Pipeline中的xgbRegressor需要直接接受这些参数。您可以通过在初始化xgbRegressor时直接传递这些参数来实现。

修改后的代码

import xgboost as xgb
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import SimpleImputer, OneHotEncoder, StandardScaler
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
import numpy as np

假设X和y已经被定义并加载

cat_columns = X.select_dtypes(include=['object']).columns.values
num_columns = X.select_dtypes(include=np.number).columns.values

print(cat_columns)
print(num_columns)

preprocess = ColumnTransformer([

('categorical', make_pipeline(SimpleImputer(strategy='most_frequent'), OneHotEncoder()), cat_columns),
('numerical', make_pipeline(SimpleImputer(strategy='mean'), StandardScaler()), num_columns)

])

定义XGBoostRegressor并配置为使用GPU

xgb_regressor = xgb.XGBRegressor(

tree_method='hist',
objective='reg:squarederror',
use_label_encoder=False,  # 注意：对于XGBoost 1.3.0及以上版本，需要设置
gpu_id=0,                 # 指定GPU设备ID
predictor='gpu_predictor' # 明确指定使用GPU预测器

)

定义完整的pipeline

pipeline = Pipeline(steps=[

('preprocess', preprocess),
('regressor', xgb_regressor)

])

定义超参数网格

param_grid = {

'regressor__n_estimators': [200, 300],
'regressor__max_depth': [5, 7],
'regressor__learning_rate': [0.05, 0.1],
'regressor__subsample': [0.8, 1.0]

}

设置GridSearchCV

grid_search = GridSearchCV(pipeline, param_grid, cv=2, scoring='neg_mean_squared_error', verbose=2)

grid_search.fit(X, y)

### 解释

1. **GPU支持**：确保安装了支持CUDA的XGBoost。
2. **XGBoostRegressor参数**：
   - `gpu_id`：指定使用的GPU设备ID。
   - `predictor`：设置为`'gpu_predictor'`以在预测时使用GPU。
   - 注意`use_label_encoder`参数，这是为了避免在新版本XGBoost中可能遇到的标签编码问题。
3. **Pipeline配置**：直接在`xgbRegressor`的初始化中设置GPU相关参数，这样Pipeline中的每一步都能正确地处理和传递数据。