「一周算法实践」Task1.2 模型构建之集成模型

背景介绍

由Datawhale组织的「一周算法实践」活动，通过短期实践一个比较完整的数据挖掘项目，迅速了解相关实际过程

GitHub - Datawhale Datamining Practice - awyd234

任务描述

【任务1.2 - 模型构建】构建随机森林、GBDT、XGBoost和LightGBM这4个模型，评分方式任意。

1. 随机森林

1
2
3
4
5
6
7
from sklearn.ensemble import RandomForestClassifier

random_forest_clf = RandomForestClassifier(random_state=2018)
random_forest_clf.fit(x_train, y_train)
print('RandomForest fit finished, score: {}'.format(random_forest_clf.score(x_test,y_test)))
predictions = random_forest_clf.predict_proba(x_test)
print('RandomForest fit finished, auc: {}'.format(cal_auc(y_test, predictions)))

Output

1
2
RandomForest fit finished, score: 0.7694463910301331
RandomForest fit finished, auc: 0.7202878365831012

2. GBDT

1
2
3
4
5
6
7
from sklearn.ensemble import GradientBoostingClassifier

gbdt_clf = GradientBoostingClassifier(random_state=2018)
gbdt_clf.fit(x_train, y_train)
print('GBDT fit finished, score: {}'.format(gbdt_clf.score(x_test, y_test)))
predictions = gbdt_clf.predict_proba(x_test)
print('GBDT fit finished, auc: {}'.format(cal_auc(y_test, predictions)))

Output

1
2
GBDT fit finished, score: 0.7806587245970568
GBDT fit finished, auc: 0.7638284665060039

3.XGBoost

XGBoost官方文档

3.1 官方介绍

XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. It implements machine learning algorithms under the Gradient Boosting framework. XGBoost provides a parallel tree boosting (also known as GBDT, GBM) that solve many data science problems in a fast and accurate way. The same code runs on major distributed environment (Hadoop, SGE, MPI) and can solve problems beyond billions of examples.

3.2 环境准备

安装相关第三方包

1
pip install -i install -i https://pypi.doubanio.com/simple/ xgboost

3.3 代码运行

1
2
3
4
5
6
7
from xgboost import XGBClassifier

xgbt_clf = XGBClassifier(random_state=2018)
xgbt_clf.fit(x_train, y_train)
print('XGBT fit finished, score: {}'.format(xgbt_clf.score(x_test, y_test)))
predictions = xgbt_clf.predict_proba(x_test)
print('XGBT fit finished, auc: {}'.format(cal_auc(y_test, predictions)))

Output

1
2
XGBT fit finished, score: 0.7855641205325858
XGBT fit finished, auc: 0.7713634419371329

4.LightGBM

LightGBM官方文档

4.1 官方介绍

LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed and efficient with the following advantages:

  • Faster training speed and higher efficiency.

  • Lower memory usage.

  • Better accuracy.

  • Support of parallel and GPU learning.

  • Capable of handling large-scale data.

4.2 环境准备

安装第三方包

1
pip install -i install -i https://pypi.doubanio.com/simple/ lightgbm

4.3 遇到的错误

然而当尝试import时

1
from lightgbm import LGBMClassifier

出现了下列错误提示

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Traceback (most recent call last):
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/main.py", line 12, in <module>
    from lightgbm import LGBMClassifier
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/lightgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/lightgbm/basic.py", line 34, in <module>
    _LIB = _load_lib()
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/lightgbm/basic.py", line 29, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 426, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from: /Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found

在GitHub Issue上寻找了一种方法尝试进行解决，此前CSDN也搜索过类似问题，解释是LightGBM编译依赖OpenMP，Apple Clang不支持，于是尝试使用以下方式进行解决

1
2
3
4
5
6
pip uninstall lightgbm
git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
export CXX=g++-8 CC=gcc-8
mkdir build ; cd build
cmake ..
make -j4

我的命令行没有g++-8和gcc-8，于是更改成了

1
export CXX=g++ CC=gcc

当执行cmake ..时，再次出错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
-- The C compiler identification is AppleClang 9.1.0.9020039
-- The CXX compiler identification is AppleClang 9.1.0.9020039
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/gcc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/g++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:32 (cmake_minimum_required):
  CMake 3.12 or higher is required.  You are running version 3.8.2

这个是cmake版本问题，于是更新了下cmake

1
brew upgrade cmake

当删除了build目录，重新cmake ..时，再次出错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
-- The C compiler identification is AppleClang 9.1.0.9020039
-- The CXX compiler identification is AppleClang 9.1.0.9020039
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/gcc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/g++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /usr/local/Cellar/cmake/3.13.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.13.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.13.1/share/cmake/Modules/FindOpenMP.cmake:473 (find_package_handle_standard_args)
  CMakeLists.txt:78 (find_package)

于是，尝试执行了

1
brew install libomp

此时再次执行cmake，再次出错，但这时import不报错了，所以先放着，估计是和libomp的安装有关，和git clone应该没关系，因为我是用virtualenv开了虚拟环境跑的

4.4 代码实现

1
2
3
4
5
6
7
from lightgbm import LGBMClassifier

light_gbm_clf = LGBMClassifier(random_state=2018)
light_gbm_clf.fit(x_train, y_train)
print('LightGBM fit finished, score: {}'.format(light_gbm_clf.score(x_test, y_test)))
predictions = light_gbm_clf.predict_proba(x_test)
print('LightGBM fit finished, auc: {}'.format(cal_auc(y_test, predictions)))

Output

1
2
LightGBM fit finished, score: 0.7701471618780659
LightGBM fit finished, auc: 0.7574019592501017

4.5 参数分析

通过进入LGBMClassifier内部，其继承自LGBMModel，于是分析LGBMModel的初始化函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
Parameters
----------
boosting_type : string, optional (default='gbdt')
    'gbdt', traditional Gradient Boosting Decision Tree.
    'dart', Dropouts meet Multiple Additive Regression Trees.
    'goss', Gradient-based One-Side Sampling.
    'rf', Random Forest.
num_leaves : int, optional (default=31)
    Maximum tree leaves for base learners.
max_depth : int, optional (default=-1)
    Maximum tree depth for base learners, -1 means no limit.
learning_rate : float, optional (default=0.1)
    Boosting learning rate.
    You can use ``callbacks`` parameter of ``fit`` method to shrink/adapt learning rate
    in training using ``reset_parameter`` callback.
    Note, that this will ignore the ``learning_rate`` argument in training.
n_estimators : int, optional (default=100)
    Number of boosted trees to fit.
subsample_for_bin : int, optional (default=200000)
    Number of samples for constructing bins.
objective : string, callable or None, optional (default=None)
    Specify the learning task and the corresponding learning objective or
    a custom objective function to be used (see note below).
    Default: 'regression' for LGBMRegressor, 'binary' or 'multiclass' for LGBMClassifier, 'lambdarank' for LGBMRanker.
class_weight : dict, 'balanced' or None, optional (default=None)
    Weights associated with classes in the form ``{class_label: weight}``.
    Use this parameter only for multi-class classification task;
    for binary classification task you may use ``is_unbalance`` or ``scale_pos_weight`` parameters.
    The 'balanced' mode uses the values of y to automatically adjust weights
    inversely proportional to class frequencies in the input data as ``n_samples / (n_classes * np.bincount(y))``.
    If None, all classes are supposed to have weight one.
    Note, that these weights will be multiplied with ``sample_weight`` (passed through the ``fit`` method)
    if ``sample_weight`` is specified.
min_split_gain : float, optional (default=0.)
    Minimum loss reduction required to make a further partition on a leaf node of the tree.
min_child_weight : float, optional (default=1e-3)
    Minimum sum of instance weight (hessian) needed in a child (leaf).
min_child_samples : int, optional (default=20)
    Minimum number of data needed in a child (leaf).
subsample : float, optional (default=1.)
    Subsample ratio of the training instance.
subsample_freq : int, optional (default=0)
    Frequence of subsample, <=0 means no enable.
colsample_bytree : float, optional (default=1.)
    Subsample ratio of columns when constructing each tree.
reg_alpha : float, optional (default=0.)
    L1 regularization term on weights.
reg_lambda : float, optional (default=0.)
    L2 regularization term on weights.
random_state : int or None, optional (default=None)
    Random number seed.
    If None, default seeds in C++ code will be used.
n_jobs : int, optional (default=-1)
    Number of parallel threads.
silent : bool, optional (default=True)
    Whether to print messages while running boosting.
importance_type : string, optional (default='split')
    The type of feature importance to be filled into ``feature_importances_``.
    If 'split', result contains numbers of times the feature is used in a model.
    If 'gain', result contains total gains of splits which use the feature.
**kwargs
    Other parameters for the model.
    Check http://lightgbm.readthedocs.io/en/latest/Parameters.html for more parameters.

    Note
    ----
    \*\*kwargs is not supported in sklearn, it may cause unexpected issues.

Attributes
----------
n_features_ : int
    The number of features of fitted model.
classes_ : array of shape = [n_classes]
    The class label array (only for classification problem).
n_classes_ : int
    The number of classes (only for classification problem).
best_score_ : dict or None
    The best score of fitted model.
best_iteration_ : int or None
    The best iteration of fitted model if ``early_stopping_rounds`` has been specified.
objective_ : string or callable
    The concrete objective used while fitting this model.
booster_ : Booster
    The underlying Booster of this model.
evals_result_ : dict or None
    The evaluation results if ``early_stopping_rounds`` has been specified.
feature_importances_ : array of shape = [n_features]
    The feature importances (the higher, the more important the feature).

Note
----
A custom objective function can be provided for the ``objective`` parameter.
In this case, it should have the signature
``objective(y_true, y_pred) -> grad, hess`` or
``objective(y_true, y_pred, group) -> grad, hess``:

    y_true : array-like of shape = [n_samples]
        The target values.
    y_pred : array-like of shape = [n_samples] or shape = [n_samples * n_classes] (for multi-class task)
        The predicted values.
    group : array-like
        Group/query data, used for ranking task.
    grad : array-like of shape = [n_samples] or shape = [n_samples * n_classes] (for multi-class task)
        The value of the gradient for each sample point.
    hess : array-like of shape = [n_samples] or shape = [n_samples * n_classes] (for multi-class task)
        The value of the second derivative for each sample point.

For multi-class task, the y_pred is group by class_id first, then group by row_id.
If you want to get i-th row y_pred in j-th class, the access way is y_pred[j * num_data + i]
and you should group grad and hess in this way as well.
  • boosting_type
    • 有时也称为boosting或者boost，默认gbdt
    • 本例子中，如果直接设置gbdt，和直接用GBDT跑出来的结果不一致，可能和其他默认参数设置有关
赏

谢谢你请我吃糖果

扫一扫，分享到微信

微信分享二维码

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

Python后端开发，业余前端，自学CV中