[Python] 波士頓房價的7種模型(線性擬合、二次多項式、Ridge、Lasso、SVM、決策樹、隨機森林)的訓練效果對比

哆啦夢樂園發表於2021-03-17

1. 載入資料

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import cross_val_score
from collections import Counter

from sklearn.datasets import load_boston
sns.set_style('darkgrid')

boston_dataset = load_boston()
dataset = pd.DataFrame(boston_dataset.data, columns = boston_dataset.feature_names)
dataset.head()

輸出：

      CRIM    ZN  INDUS  CHAS    NOX  ...  RAD    TAX  PTRATIO       B  LSTAT
0  0.00632  18.0   2.31   0.0  0.538  ...  1.0  296.0     15.3  396.90   4.98
1  0.02731   0.0   7.07   0.0  0.469  ...  2.0  242.0     17.8  396.90   9.14
2  0.02729   0.0   7.07   0.0  0.469  ...  2.0  242.0     17.8  392.83   4.03
3  0.03237   0.0   2.18   0.0  0.458  ...  3.0  222.0     18.7  394.63   2.94
4  0.06905   0.0   2.18   0.0  0.458  ...  3.0  222.0     18.7  396.90   5.33

[5 rows x 13 columns]

列解釋Columns:

CRIM: 按城鎮劃分的人均犯罪率
ZN: 大於25000平方英尺的住宅用地比例
INDUS: 每個城鎮非零售業務英畝比例
CHAS : 查爾斯河流啞變數 (靠近河流為1; 否則為0)
NOX: 一氧化氮濃度 (百萬分之)
RM: 每個住宅的房間數
AGE: 1940年之前建造的自由單位
DIS: 與5個波士頓就業中心的加權距離
RAD: 高速公里同行能力指數
PTRATIO: 按城鎮劃分的師生比例
B: 1000(Bk — 0.63)², 按城鎮劃分的非裔人口結構比例
LSTAT: 低收入人口百分比
MEDV: 自有住房的中位數價值(單位：1000美元)

上圖看到沒有MEDV這個我們要預測的列，先加進來。

dataset['MEDV'] = boston_dataset.target

現在再看就有了。

2. 資料分析

2.1 預處理

看看有沒有缺失值

dataset.isnull().sum()

設定特徵和標籤

X = dataset.iloc[:, 0:13].values
y = dataset.iloc[:, 13].values.reshape(-1,1)

分割訓練和測試

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 25)

看看分割結果

print("Shape of X_train: ",X_train.shape)
print("Shape of X_test: ", X_test.shape)
print("Shape of y_train: ",y_train.shape)
print("Shape of y_test",y_test.shape)
輸出：
Shape of X_train:  (354, 13)
Shape of X_test:  (152, 13)
Shape of y_train:  (354, 1)
Shape of y_test (152, 1)

2.2 視覺化

#相關係數矩陣，即給出了任意兩個變數之間的相關係數
corr = dataset.corr()
import matplotlib.pyplot as plt
import seaborn as sns
fig, ax = plt.subplots(figsize=(10, 10))
sns.heatmap(corr, cmap='RdBu', annot=True, fmt=".2f")
plt.xticks(range(len(corr.columns)), corr.columns)
plt.yticks(range(len(corr.columns)), corr.columns)
plt.show()

sns.pairplot(dataset)
plt.show()

3. 訓練模型

3.1 線性擬合

from sklearn.linear_model import LinearRegression
regressor_linear = LinearRegression()
regressor_linear.fit(X_train, y_train)

看看此時的預測得分

from sklearn.metrics import r2_score
# 交叉驗證：將資料集分為10折，做一次交叉驗證，實際上它是計算了十次，將每一折都當做一次測試集，其餘九折當做訓練集，這樣迴圈十次。通過傳入的模型，訓練十次，最後將十次結果求平均值。
cv_linear = cross_val_score(estimator = regressor_linear, X = X_train, y = y_train, cv = 10)

# R2 score，即決定係數，反映因變數的全部變異能通過迴歸關係被自變數解釋的比例
y_pred_linear_train = regressor_linear.predict(X_train)
r2_score_linear_train = r2_score(y_train, y_pred_linear_train)

y_pred_linear_test = regressor_linear.predict(X_test)
r2_score_linear_test = r2_score(y_test, y_pred_linear_test)

# RMSE一般指均方根誤差。均方根誤差亦稱標準誤差。
rmse_linear = (np.sqrt(mean_squared_error(y_test, y_pred_linear_test)))

print("CV: ", cv_linear.mean())
print('R2_score (train): ', r2_score_linear_train)
print('R2_score (test): ', r2_score_linear_test)
print("RMSE: ", rmse_linear)

輸出：

CV:  0.6984854476156042
R2_score (train):  0.7435787589010061
R2_score (test):  0.7133593313710366
RMSE:  4.6472797457242

3.2 多項式迴歸(二次)

from sklearn.preprocessing import PolynomialFeatures
poly_reg = PolynomialFeatures(degree = 2)
X_poly = poly_reg.fit_transform(X_train)
poly_reg.fit(X_poly, y_train)
regressor_poly2 = LinearRegression()
regressor_poly2.fit(X_poly, y_train)

看預測得分

from sklearn.metrics import r2_score
cv_poly2 = cross_val_score(estimator = regressor_poly2, X = X_train, y = y_train, cv = 10)

y_pred_poly2_train = regressor_poly2.predict(poly_reg.fit_transform(X_train))
r2_score_poly2_train = r2_score(y_train, y_pred_poly2_train)

y_pred_poly2_test = regressor_poly2.predict(poly_reg.fit_transform(X_test))
r2_score_poly2_test = r2_score(y_test, y_pred_poly2_test)

rmse_poly2 = (np.sqrt(mean_squared_error(y_test, y_pred_poly2_test)))
print('CV: ', cv_poly2.mean())
print('R2_score (train): ', r2_score_poly2_train)
print('R2_score (test): ', r2_score_poly2_test)
print("RMSE: ", rmse_poly2)

輸出略，最後彙總

3.3 脊迴歸(Ridge Regression),又叫嶺迴歸

lasso 迴歸和嶺迴歸（ridge regression）其實就是在標準線性迴歸的基礎上分別加入 L1 和 L2 正則化（regularization）

from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures

steps = [
    ('scalar', StandardScaler()),
    ('poly', PolynomialFeatures(degree=2)),
    ('model', Ridge(alpha=3.8, fit_intercept=True))
]

ridge_pipe = Pipeline(steps)
ridge_pipe.fit(X_train, y_train)

評估

from sklearn.metrics import r2_score

cv_ridge = cross_val_score(estimator = ridge_pipe, X = X_train, y = y_train.ravel(), cv = 10)

y_pred_ridge_train = ridge_pipe.predict(X_train)
r2_score_ridge_train = r2_score(y_train, y_pred_ridge_train)

y_pred_ridge_test = ridge_pipe.predict(X_test)
r2_score_ridge_test = r2_score(y_test, y_pred_ridge_test)

rmse_ridge = (np.sqrt(mean_squared_error(y_test, y_pred_ridge_test)))
print('CV: ', cv_ridge.mean())
print('R2_score (train): ', r2_score_ridge_train)
print('R2_score (test): ', r2_score_ridge_test)
print("RMSE: ", rmse_ridge)

3.4 Lasso 迴歸

from sklearn.linear_model import Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures

steps = [
    ('scalar', StandardScaler()),
    ('poly', PolynomialFeatures(degree=2)),
    ('model', Lasso(alpha=0.012, fit_intercept=True, max_iter=3000))
]

lasso_pipe = Pipeline(steps)
lasso_pipe.fit(X_train, y_train)

評估

from sklearn.metrics import r2_score

# Predicting Cross Validation Score
cv_lasso = cross_val_score(estimator = lasso_pipe, X = X_train, y = y_train, cv = 10)

# Predicting R2 Score the Test set results
y_pred_lasso_train = lasso_pipe.predict(X_train)
r2_score_lasso_train = r2_score(y_train, y_pred_lasso_train)

# Predicting R2 Score the Test set results
y_pred_lasso_test = lasso_pipe.predict(X_test)
r2_score_lasso_test = r2_score(y_test, y_pred_lasso_test)

# Predicting RMSE the Test set results
rmse_lasso = (np.sqrt(mean_squared_error(y_test, y_pred_lasso_test)))
print('CV: ', cv_lasso.mean())
print('R2_score (train): ', r2_score_lasso_train)
print('R2_score (test): ', r2_score_lasso_test)
print("RMSE: ", rmse_lasso)

3.5 支援向量迴歸 Support Vector Regression

支援向量分類的方法能被推廣到解決迴歸問題，稱為支援向量迴歸

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc_X = StandardScaler()
sc_y = StandardScaler()
X_scaled = sc_X.fit_transform(X_train)
y_scaled = sc_y.fit_transform(y_train.reshape(-1,1))

# Fitting the SVR Model to the dataset
from sklearn.svm import SVR
regressor_svr = SVR(kernel = 'rbf', gamma = 'scale')
regressor_svr.fit(X_scaled, y_scaled.ravel())

評估

from sklearn.metrics import r2_score

# Predicting Cross Validation Score
cv_svr = cross_val_score(estimator = regressor_svr, X = X_scaled, y = y_scaled.ravel(), cv = 10)

# Predicting R2 Score the Train set results
y_pred_svr_train = sc_y.inverse_transform(regressor_svr.predict(sc_X.transform(X_train)))
r2_score_svr_train = r2_score(y_train, y_pred_svr_train)

# Predicting R2 Score the Test set results
y_pred_svr_test = sc_y.inverse_transform(regressor_svr.predict(sc_X.transform(X_test)))
r2_score_svr_test = r2_score(y_test, y_pred_svr_test)

# Predicting RMSE the Test set results
rmse_svr = (np.sqrt(mean_squared_error(y_test, y_pred_svr_test)))
print('CV: ', cv_svr.mean())
print('R2_score (train): ', r2_score_svr_train)
print('R2_score (test): ', r2_score_svr_test)
print("RMSE: ", rmse_svr)

3.6 決策樹迴歸 Decision Tree Regression

from sklearn.tree import DecisionTreeRegressor
regressor_dt = DecisionTreeRegressor(random_state = 0)
regressor_dt.fit(X_train, y_train)

評估

from sklearn.metrics import r2_score

# Predicting Cross Validation Score
cv_dt = cross_val_score(estimator = regressor_dt, X = X_train, y = y_train, cv = 10)

# Predicting R2 Score the Train set results
y_pred_dt_train = regressor_dt.predict(X_train)
r2_score_dt_train = r2_score(y_train, y_pred_dt_train)

# Predicting R2 Score the Test set results
y_pred_dt_test = regressor_dt.predict(X_test)
r2_score_dt_test = r2_score(y_test, y_pred_dt_test)

# Predicting RMSE the Test set results
rmse_dt = (np.sqrt(mean_squared_error(y_test, y_pred_dt_test)))
print('CV: ', cv_dt.mean())
print('R2_score (train): ', r2_score_dt_train)
print('R2_score (test): ', r2_score_dt_test)
print("RMSE: ", rmse_dt)

3.7 隨機森林迴歸 Random Forest Regression

from sklearn.ensemble import RandomForestRegressor
regressor_rf = RandomForestRegressor(n_estimators = 500, random_state = 0)
regressor_rf.fit(X_train, y_train.ravel())

評估

from sklearn.metrics import r2_score

# Predicting Cross Validation Score
cv_rf = cross_val_score(estimator = regressor_rf, X = X_scaled, y = y_train.ravel(), cv = 10)

# Predicting R2 Score the Train set results
y_pred_rf_train = regressor_rf.predict(X_train)
r2_score_rf_train = r2_score(y_train, y_pred_rf_train)

# Predicting R2 Score the Test set results
y_pred_rf_test = regressor_rf.predict(X_test)
r2_score_rf_test = r2_score(y_test, y_pred_rf_test)

# Predicting RMSE the Test set results
rmse_rf = (np.sqrt(mean_squared_error(y_test, y_pred_rf_test)))
print('CV: ', cv_rf.mean())
print('R2_score (train): ', r2_score_rf_train)
print('R2_score (test): ', r2_score_rf_test)
print("RMSE: ", rmse_rf)

4. 評估結果彙總

models = [('Linear Regression', rmse_linear, r2_score_linear_train, r2_score_linear_test, cv_linear.mean()),
          ('Polynomial Regression (2nd)', rmse_poly2, r2_score_poly2_train, r2_score_poly2_test, cv_poly2.mean()),
          ('Ridge Regression', rmse_ridge, r2_score_ridge_train, r2_score_ridge_test, cv_ridge.mean()),
          ('Lasso Regression', rmse_lasso, r2_score_lasso_train, r2_score_lasso_test, cv_lasso.mean()),
          ('Support Vector Regression', rmse_svr, r2_score_svr_train, r2_score_svr_test, cv_svr.mean()),
          ('Decision Tree Regression', rmse_dt, r2_score_dt_train, r2_score_dt_test, cv_dt.mean()),
          ('Random Forest Regression', rmse_rf, r2_score_rf_train, r2_score_rf_test, cv_rf.mean())   
         ]

看看錶格

predict = pd.DataFrame(data = models, columns=['Model', 'RMSE', 'R2_Score(training)', 'R2_Score(test)', 'Cross-Validation'])
predict

輸出：

No	Model	RMSE	R2_Score(training)	R2_Score(test)	Cross-Validation
0	Linear Regression	4.647280	0.743579	0.713359	0.698485
1	Polynomial Regression (2nd)	4.194313	0.930656	0.766513	0.698485
2	Ridge Regression	2.853062	0.922818	0.891965	0.763563
3	Lasso Regression	2.811451	0.923402	0.895094	0.750544
4	Support Vector Regression	3.838898	0.874272	0.804407	0.782601
5	Decision Tree Regression	5.723785	1.000000	0.565183	0.613295
6	Random Forest Regression	3.211470	0.976717	0.863118	0.818677

5. 視覺化評估結果

f, axe = plt.subplots(1,1, figsize=(18,6))

predict.sort_values(by=['Cross-Validation'], ascending=False, inplace=True)

sns.barplot(x='Cross-Validation', y='Model', data = predict, ax = axe)
#axes[0].set(xlabel='Region', ylabel='Charges')
axe.set_xlabel('Cross-Validaton Score', size=16)
axe.set_ylabel('Model')
axe.set_xlim(0,1.0)
plt.show()

f, axes = plt.subplots(2,1, figsize=(14,10))

predict.sort_values(by=['R2_Score(training)'], ascending=False, inplace=True)

sns.barplot(x='R2_Score(training)', y='Model', data = predict, palette='Blues_d', ax = axes[0])
#axes[0].set(xlabel='Region', ylabel='Charges')
axes[0].set_xlabel('R2 Score (Training)', size=16)
axes[0].set_ylabel('Model')
axes[0].set_xlim(0,1.0)

predict.sort_values(by=['R2_Score(test)'], ascending=False, inplace=True)

sns.barplot(x='R2_Score(test)', y='Model', data = predict, palette='Reds_d', ax = axes[1])
#axes[0].set(xlabel='Region', ylabel='Charges')
axes[1].set_xlabel('R2 Score (Test)', size=16)
axes[1].set_ylabel('Model')
axes[1].set_xlim(0,1.0)

plt.show()

predict.sort_values(by=['RMSE'], ascending=False, inplace=True)

f, axe = plt.subplots(1,1, figsize=(18,6))
sns.barplot(x='Model', y='RMSE', data=predict, ax = axe)
axe.set_xlabel('Model', size=16)
axe.set_ylabel('RMSE', size=16)

plt.show()

機器學習之決策樹與隨機森林模型
2017-10-17
機器學習隨機森林模型
決策樹和隨機森林
2020-12-11
隨機森林
【機器學習】--決策樹和隨機森林
2018-03-27
機器學習隨機森林
波士頓房價預測
2024-03-28
基於sklearn的波士頓房價預測_線性迴歸學習筆記
2021-04-02
筆記
決策樹、邏輯迴歸、線性迴歸使用時注意事項以及模型過擬合策略
2016-06-27
邏輯迴歸模型
從決策樹到隨機森林：樹型演算法的原理與實現
2017-07-31
隨機森林演算法
Python實現隨機森林RF並對比自變數的重要性
2023-02-16
Python隨機森林變數
決策樹VS隨機森林——應該使用哪種演算法？（附程式碼&連結）
2020-06-11
隨機森林演算法
機器學習(五)：通俗易懂決策樹與隨機森林及程式碼實踐
2021-02-25
機器學習隨機森林
決策樹模型(3)決策樹的生成與剪枝
2024-03-28
模型
機器學習——線性迴歸-KNN-決策樹(例項)
2018-10-09
機器學習KNN
學習SVM（一） SVM模型訓練與分類的OpenCV實現
2017-03-29
模型OpenCV
Java 使用 Apache commons-math3 線性擬合、非線性擬合例項（帶效果圖）
2021-08-06
JavaApache
AdaBoost 演算法-分析波士頓房價資料集
2020-12-17
演算法
R語言中使用線性模型、迴歸決策樹自動組合特徵因子水平
2020-08-21
R語言模型特徵
機器學習——決策樹模型：Python實現
2020-11-09
機器學習模型Python
機器學習——決策樹模型
2023-12-26
機器學習模型
從線性模型到決策樹再到深度學習的分位數迴歸
2018-10-26
模型深度學習
聊聊基於Alink庫的隨機森林模型
2023-10-02
隨機森林模型
matlab練習程式（最小二乘多項式擬合）
2018-10-10
Matlab
分類——決策樹模型
2020-10-23
模型
隨機森林RF模型超引數的最佳化：Python實現
2023-02-17
隨機森林模型Python
Python隨機森林 - CodeProject
2017-11-15
Python隨機森林Project
通俗易懂--決策樹演算法、隨機森林演算法講解(演算法+案例)
2018-12-18
演算法隨機森林
決策樹演算法預測森林植被
2016-05-07
演算法
【問題3】：Kaggle練習題《房價預測》----分別採用的嶺迴歸，隨機森林，bagging模型，AdaBoost，XgBoost等。
2019-02-13
隨機森林模型
Wenet多機多卡分散式訓練
2023-01-09
分散式
使用 SVM 和決策樹進行整合學習
2024-03-11
隨機森林的祕密
2018-07-30
隨機森林
基於Theano的深度學習框架keras及配合SVM訓練模型
2020-04-06
深度學習框架Keras模型
4擬合線性函式的k和b
2020-11-12
函式
隨手刷屏的波士頓動力機器人，用3D列印解決了哪些問題
2018-03-24
機器人3D
支援向量機（Support Vector Machine，SVM）——　線性SVM
2018-12-28
Mac
隨機之美，隨機森林
2016-04-13
隨機森林
解讀乾貨：詞語對齊的注意力機制，提升中文預訓練模型效果
2022-01-13
模型
何時線性迴歸，聚類或決策樹？
2018-02-20
聚類
Python機器學習：決策樹001什麼是決策樹
2020-12-24
Python機器學習