「一周算法实践」Task2 模型评估

背景介绍

由Datawhale组织的「一周算法实践」活动，通过短期实践一个比较完整的数据挖掘项目，迅速了解相关实际过程

GitHub - Datawhale Datamining Practice - awyd234

任务描述

【任务2 - 模型评估】记录7个模型（在Task1的基础上）关于accuracy、precision，recall和F1-score、auc值的评分表格，并画出Roc曲线。

评价指标

对二分类问题常用的评估指标是精度(precision)、召回率(recall)、F1值(F1-score)，通常以关注的类为正类，其他类为负类

实践过程

画ROC曲线需要用到matplotlib

1
pip install -i install -i https://pypi.doubanio.com/simple/ matplotlib

参考004-Sm1les同学博客的写法，将训练过程和评价方法单独定义了一个函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
from sklearn import metrics
from sklearn import tree
from matplotlib import pyplot as plt
from sklearn.preprocessing import StandardScaler


def fit_and_evaluate_model(model, x_train_standard, y_train, x_test_standard, y_test, model_name):
    '''
    train model and evaluate it
    :param model: 
    :param x_train_standard: 
    :param y_train: 
    :param x_test_standard: 
    :param y_test: 
    :param model_name: 
    :return: 
    '''
    clf.fit(x_train_standard, y_train)

    # 模型对训练集的预测值
    y_train_prediction_value = clf.predict(x_train_standard)

    # 模型对测试集的预测值
    y_test_prediction_value = clf.predict(x_test_standard)

    # 模型对训练集的各类预测概率
    y_train_prediction_prob = clf.predict_proba(x_train_standard)
    # 模型对测试集的各类预测概率
    y_test_prediction_prob = clf.predict_proba(x_test_standard)

    train_accuracy_score = metrics.accuracy_score(y_train, y_train_prediction_value)
    test_accuracy_score = metrics.accuracy_score(y_test, y_test_prediction_value)

    train_precision_score = metrics.precision_score(y_train, y_train_prediction_value)
    test_precision_score = metrics.precision_score(y_test, y_test_prediction_value)

    train_recall = metrics.recall_score(y_train, y_train_prediction_value)
    test_recall = metrics.recall_score(y_test, y_test_prediction_value)

    train_f1_score = metrics.f1_score(y_train, y_train_prediction_value)
    test_f1_score = metrics.f1_score(y_test, y_test_prediction_value)

    train_auc_score = metrics.roc_auc_score(y_train, y_train_prediction_prob[:, 1])
    test_auc_score = metrics.roc_auc_score(y_test, y_test_prediction_prob[:, 1])

    print("------------------------{model_name}------------------------".format(**locals()))
    print("Train Set:")
    print("Accuracy: {train_accuracy_score}".format(**locals()))
    print("Precision: {train_precision_score}".format(**locals()))
    print("F1_score: {train_f1_score}".format(**locals()))
    print("Recall: {train_recall}".format(**locals()))
    print("Auc: {train_auc_score}".format(**locals()))

    print("Test Set:")
    print("Accuracy: {test_accuracy_score}".format(**locals()))
    print("Precision: {test_precision_score}".format(**locals()))
    print("F1_score: {test_f1_score}".format(**locals()))
    print("Recall: {test_recall}".format(**locals()))
    print("Auc: {test_auc_score}".format(**locals()))

    # ROC曲线绘制
    train_fprs, train_tprs, train_thresholds = metrics.roc_curve(y_train, y_train_prediction_prob[:, 1])
    test_fprs, test_tprs, test_thresholds = metrics.roc_curve(y_test, y_test_prediction_prob[:, 1])
    plt.plot(train_fprs, train_tprs)
    plt.plot(test_fprs, test_tprs)
    plt.plot([0, 1], [0, 1], "--")
    plt.title("{model_name} ROC curve".format(**locals()))
    plt.xlabel("FPR")
    plt.ylabel("TPR")
    plt.legend(
        labels=["Train Set AUC:" + str(round(train_accuracy_score, 5)), "Test Set AUC:" + str(round(test_accuracy_score, 5))],
        loc="lower right")
    plt.show()

    print("-----------------------------------------------------------")

生成结果

1. 逻辑回归

1.1 代码实现

1
2
lr = LogisticRegression(random_state=2018)
fit_and_evaluate_model(lr, x_train_standard, y_train, x_test_standard, y_test, 'LogisticRegression')

1.2 评价指标结果

Evaluation Train Set Test Set
Accuracy 0.8049293657950105 0.7876664330763841
Precision 0.7069351230425056 0.6609195402298851
F1-score 0.4933645589383294 0.4315196998123827
Recall 0.37889688249400477 0.3203342618384401
Auc 0.8198240444948494 0.7657454644090431

1.3 ROC曲线

2. SVM

2.1 代码实现

1
2
svm_clf = SVC(random_state=2018, probability=True)
fit_and_evaluate_model(svm_clf, x_train_standard, y_train, x_test_standard, y_test, 'SVM')

2.2 评价指标结果

Evaluation Train Set Test Set
Accuracy 0.8428013225127743 0.7806587245970568
Precision 0.9146666666666666 0.7017543859649122
F1-score 0.5674110835401158 0.3382663847780127
Recall 0.41127098321342925 0.22284122562674094
Auc 0.9217853154299663 0.7531297924947575

2.3 ROC曲线

3. DecisionTree

3.1 代码实现

1
2
dt_clf = tree.DecisionTreeClassifier(random_state=2018)
fit_and_evaluate_model(dt_clf, x_train_standard, y_train, x_test_standard, y_test, 'DecisionTree')

3.2 评价指标结果

Evaluation Train Set Test Set
Accuracy 1.0 0.6860546601261388
Precision 1.0 0.3850129198966408
F1-score 1.0 0.39946380697050937
Recall 1.0 0.415041782729805
Auc 1.0 0.5960976703911197

3.3 ROC曲线

4. RandomForest

4.1 代码实现

1
2
random_forest_clf = RandomForestClassifier(random_state=2018)
fit_and_evaluate_model(random_forest_clf, x_train_standard, y_train, x_test_standard, y_test, 'RandomForest')

4.2 评价指标结果

Evaluation Train Set Test Set
Accuracy 0.9849714457469192 0.7680448493342676
Precision 1.0 0.5875
F1-score 0.9690976514215082 0.3622350674373796
Recall 0.9400479616306955 0.2618384401114206
Auc 0.9995264919231882 0.7194427926095167

4.3 ROC曲线

5. GBDT

5.1 代码实现

1
2
gbdt_clf = GradientBoostingClassifier(random_state=2018)
fit_and_evaluate_model(gbdt_clf, x_train_standard, y_train, x_test_standard, y_test, 'GBDT')

5.2 评价指标结果

Evaluation Train Set Test Set
Accuracy 0.8623384430417794 0.7806587245970568
Precision 0.8836734693877552 0.6116504854368932
F1-score 0.6540785498489425 0.44601769911504424
Recall 0.5191846522781774 0.35097493036211697
Auc 0.9207464353427005 0.7632677120173599

5.3 ROC曲线

6. XGBoost

6.1 代码实现

1
2
xgbt_clf = XGBClassifier(random_state=2018)
fit_and_evaluate_model(xgbt_clf, x_train_standard, y_train, x_test_standard, y_test, 'XGBoost')

6.2 评价指标结果

Evaluation Train Set Test Set
Accuracy 0.8539224526600541 0.7841625788367204
Precision 0.875 0.624390243902439
F1-score 0.6255778120184899 0.45390070921985815
Recall 0.486810551558753 0.3565459610027855
Auc 0.9174710772897927 0.7708522424963224

6.3 ROC曲线

7. LightGBM

7.1 代码实现

1
2
light_gbm_clf = LGBMClassifier(random_state=2018)
fit_and_evaluate_model(light_gbm_clf, x_train_standard, y_train, x_test_standard, y_test, 'LightGBM')

7.2 评价指标结果

Evaluation Train Set Test Set
Accuracy 0.9957920048091373 0.7701471618780659
Precision 1.0 0.5688888888888889
F1-score 0.9915356711003627 0.4383561643835616
Recall 0.9832134292565947 0.3565459610027855
Auc 0.9999826853318788 0.7535210165566023

7.3 ROC曲线

遇到的问题

1. matplot的import问题

当尝试引入matplotlib时

1
from matplotlib import pyplot as plt

出现了以下错误

1
2
3
4
5
6
7
8
9
10
11
12
Traceback (most recent call last):
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/main.py", line 14, in <module>
    from matplotlib import pyplot as plt
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/matplotlib/pyplot.py", line 2374, in <module>
    switch_backend(rcParams["backend"])
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/matplotlib/pyplot.py", line 207, in switch_backend
    backend_mod = importlib.import_module(backend_name)
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/windtrack/Interests/ai_related_learning/datawhale_dm_practice/.env/lib/python3.6/site-packages/matplotlib/backends/backend_macosx.py", line 14, in <module>
    from matplotlib.backends import _macosx
ImportError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

于是在网上搜到了一种解决方案，在自己的工作目录的.matplotlib目录里，新建了文件matplotlibrc并写入值backend: TkAgg

1
echo 'backend: TkAgg' > ~/.matplotlib/matplotlibrc

此时再次执行则可以正常进行

2. 逻辑回归和SVM中部分评价结果为0

据说，与数据未调参有关，导致两个分母都为0有关，可以暂时加上数据归一化

1
2
3
4
5
6
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(x_train)
x_train_standard = scaler.transform(x_train)
x_test_standard = scaler.transform(x_test)
赏

谢谢你请我吃糖果

扫一扫，分享到微信

微信分享二维码

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

Python后端开发，业余前端，自学CV中