機器學習-資料分析模板

Objective

我們的任務：預測一個人的收入能否超過五萬美元
人口普查資料集: https://archive.ics.uci.edu/ml/datasets/adult

Data Set Information:

Extraction was done by Barry Becker from the 1994 Census database. A set of reasonably clean records was extracted using the following conditions: ((AAGE>16) && (AGI>100) && (AFNLWGT>1)&& (HRSWK>0))

Prediction task is to determine whether a person makes over 50K a year.

Attribute Information:

Listing of attributes:
50K, <=50K.
age: continuous.
workclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.
fnlwgt: continuous.
education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.
education-num: continuous.
marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.
occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.
relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.
race: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.
sex: Female, Male.
capital-gain: continuous.
capital-loss: continuous.
hours-per-week: continuous.
native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.

INPUT

# Load Training and Test Data Sets
headers = ['age', 'workclass', 'fnlwgt', 
           'education', 'education-num', 
           'marital-status', 'occupation', 
           'relationship', 'race', 'sex', 
           'capital-gain', 'capital-loss', 
           'hours-per-week', 'native-country', 
           'predclass']
training_raw = pd.read_csv('dataset/adult.data', 
                       header=None, 
                       names=headers, 
                       sep=',\s', 
                       na_values=["?"], 
                       engine='python')
test_raw = pd.read_csv('dataset/adult.test', 
                      header=None, 
                      names=headers, 
                      sep=',\s', 
                      na_values=["?"], 
                      engine='python', 
                      skiprows=1)

header預設是0，即取第零行資料為columns；header=None，即不從資料中取columns
names=headers，即為columns賦值

import pandas as pd
#可以列印說明文件
print (help(pd.read_csv))

# Join Datasets
dataset_raw = training_raw.append(test_raw)
dataset_raw.reset_index(inplace=True)
dataset_raw.drop('index',inplace=True,axis=1)
dataset_raw.head()

由於合併資料，導致index亂了，所以reset_index(inplace=True)

屬性：
age 年齡 double
workclass 工作型別 string
fnlwgt 序號 string
education 教育程度 string
education_num 受教育時間 double
maritial_status 婚姻狀況 string
occupation 職業 string
relationship 關係 string
race 種族 string
sex 性別 string
capital_gain 資本收益 string
capital_loss 資本損失 string
hours_per_week 每週工作小時數 double
native_country 原籍 string
income 收入 string

單特徵與缺失值展示

關於特徵，我們可以分析單特徵，也可以分析不同特徵之間的關係，首先來看單特徵

特徵簡單分為兩種：類別型和數值型

• Numerical: 都是數
• Categorical: 種類或者字串

# 會展示所有數值型的特徵
dataset_raw.describe()

# 展示所有種型別特徵,注意是大寫的o
dataset_raw.describe(include=['O'])

# 缺失值顯示
missingno.matrix(dataset_raw, figsize = (30,5))

Feature Cleaning, Engineering

清洗: 資料預處理工作:

• 缺失值: 對缺失值進行填充
• 特殊值: 一些錯誤導致的特殊值，例如 ±Inf, NA NaN
• 離群點: 這些點可能會對結果產生影響，先把它們找出來
• 錯誤值: 比如人的年齡不可能出現負數

特徵工程: There are multiple techniques for feature engineering:

• 特徵分解: 比如將時間資料2014-09-20T20:45:40Z 轉換成天，小時等資訊.
• 離散化: 我們可以選擇離散一些我們所擁有的連續變數，因為一些演算法會執行得更快。但是會對結果產生什麼樣的影響呢？需要比較離散和非離散的建模結果
• dataset_bin => 連續值被離散化的資料集
• dataset_con => 非離散化的資料集
• 特徵組合: 將不同的特徵組合成一個新特徵

缺失值問題: 我們可以填補缺失值，在許多不同的方式：：

• 額外的資料補充: 有點難弄
• 均值填充: 這樣可以不改變當前資料集整體的均值
• 迴歸模型預測: 建立一個迴歸模型去得到預測值

標籤轉換

如果收入大於 $50K. 那麼就是1 反之就是0

# Let's fix the Class Feature
dataset_raw.loc[dataset_raw['predclass'] == '>50K', 'predclass'] = 1
dataset_raw.loc[dataset_raw['predclass'] == '>50K.', 'predclass'] = 1
dataset_raw.loc[dataset_raw['predclass'] == '<=50K', 'predclass'] = 0
dataset_raw.loc[dataset_raw['predclass'] == '<=50K.', 'predclass'] = 0

dataset_bin['predclass'] = dataset_raw['predclass']
dataset_con['predclass'] = dataset_raw['predclass']

#資料不太均衡的
plt.style.use('seaborn-whitegrid')
fig = plt.figure(figsize=(20,1)) 
sns.countplot(y="predclass", data=dataset_bin);

單變數

Feature: Age

dataset_bin['age'] = pd.cut(dataset_raw['age'], 10) # 將連續值進行切分
dataset_con['age'] = dataset_raw['age'] # non-discretised

#左圖是切分後的結果 右圖是根據不同的收入等級劃分
plt.style.use('seaborn-whitegrid')
fig = plt.figure(figsize=(20,5)) 
plt.subplot(1, 2, 1)
sns.countplot(y="age", data=dataset_bin);
plt.subplot(1, 2, 2)
sns.distplot(dataset_con.loc[dataset_con['predclass'] == 1]['age'], kde_kws={"label": ">$50K"});
sns.distplot(dataset_con.loc[dataset_con['predclass'] == 0]['age'], kde_kws={"label": "<$50K"});

Feature: Workclas

# 工作的種類
plt.style.use('seaborn-whitegrid')
plt.figure(figsize=(20,3)) 
sns.countplot(y="workclass", data=dataset_raw);

# 種類有些太多了，有些類似的要進行合併
dataset_raw.loc[dataset_raw['workclass'] == 'Without-pay', 'workclass'] = 'Not Working'
dataset_raw.loc[dataset_raw['workclass'] == 'Never-worked', 'workclass'] = 'Not Working'
dataset_raw.loc[dataset_raw['workclass'] == 'Federal-gov', 'workclass'] = 'Fed-gov'
dataset_raw.loc[dataset_raw['workclass'] == 'State-gov', 'workclass'] = 'Non-fed-gov'
dataset_raw.loc[dataset_raw['workclass'] == 'Local-gov', 'workclass'] = 'Non-fed-gov'
dataset_raw.loc[dataset_raw['workclass'] == 'Self-emp-not-inc', 'workclass'] = 'Self-emp'
dataset_raw.loc[dataset_raw['workclass'] == 'Self-emp-inc', 'workclass'] = 'Self-emp'

dataset_bin['workclass'] = dataset_raw['workclass']
dataset_con['workclass'] = dataset_raw['workclass']

plt.style.use('seaborn-whitegrid')
fig = plt.figure(figsize=(20,2)) 
sns.countplot(y="workclass", data=dataset_bin);

Feature: Occupation職業

# 職業也是同樣的問題，重複的太多了
plt.style.use('seaborn-whitegrid')
plt.figure(figsize=(20,5)) 
sns.countplot(y="occupation", data=dataset_raw);

# Create buckets for Occupation
dataset_raw.loc[dataset_raw['occupation'] == 'Adm-clerical', 'occupation'] = 'Admin'
dataset_raw.loc[dataset_raw['occupation'] == 'Armed-Forces', 'occupation'] = 'Military'
dataset_raw.loc[dataset_raw['occupation'] == 'Craft-repair', 'occupation'] = 'Manual Labour'
dataset_raw.loc[dataset_raw['occupation'] == 'Exec-managerial', 'occupation'] = 'Office Labour'
dataset_raw.loc[dataset_raw['occupation'] == 'Farming-fishing', 'occupation'] = 'Manual Labour'
dataset_raw.loc[dataset_raw['occupation'] == 'Handlers-cleaners', 'occupation'] = 'Manual Labour'
dataset_raw.loc[dataset_raw['occupation'] == 'Machine-op-inspct', 'occupation'] = 'Manual Labour'
dataset_raw.loc[dataset_raw['occupation'] == 'Other-service', 'occupation'] = 'Service'
dataset_raw.loc[dataset_raw['occupation'] == 'Priv-house-serv', 'occupation'] = 'Service'
dataset_raw.loc[dataset_raw['occupation'] == 'Prof-specialty', 'occupation'] = 'Professional'
dataset_raw.loc[dataset_raw['occupation'] == 'Protective-serv', 'occupation'] = 'Military'
dataset_raw.loc[dataset_raw['occupation'] == 'Sales', 'occupation'] = 'Office Labour'
dataset_raw.loc[dataset_raw['occupation'] == 'Tech-support', 'occupation'] = 'Office Labour'
dataset_raw.loc[dataset_raw['occupation'] == 'Transport-moving', 'occupation'] = 'Manual Labour'

dataset_bin['occupation'] = dataset_raw['occupation']
dataset_con['occupation'] = dataset_raw['occupation']

plt.style.use('seaborn-whitegrid')
fig = plt.figure(figsize=(20,3))
sns.countplot(y="occupation", data=dataset_bin);

Feature: Native Country

# 同樣進行合併
plt.style.use('seaborn-whitegrid')
plt.figure(figsize=(20,10)) 
sns.countplot(y="native-country", data=dataset_raw);

Feature: Education

# 教育也進行合併
plt.style.use('seaborn-whitegrid')
plt.figure(figsize=(20,5)) 
sns.countplot(y="education", data=dataset_raw);

Feature: Marital Status

Feature: Final Weight

# 同樣分成10份
dataset_bin['fnlwgt'] = pd.cut(dataset_raw['fnlwgt'], 10)
dataset_con['fnlwgt'] = dataset_raw['fnlwgt']

Feature: Capital Gain

#資本增益
dataset_bin['capital-gain'] = pd.cut(dataset_raw['capital-gain'], 5)
dataset_con['capital-gain'] = dataset_raw['capital-gain']

plt.style.use('seaborn-whitegrid')
fig = plt.figure(figsize=(20,3)) 
plt.subplot(1, 2, 1)
sns.countplot(y="capital-gain", data=dataset_bin);
plt.subplot(1, 2, 2)
sns.distplot(dataset_con['capital-gain']);

Feature: Capital Loss

# 資本損失
dataset_bin['capital-loss'] = pd.cut(dataset_raw['capital-loss'], 5)
dataset_con['capital-loss'] = dataset_raw['capital-loss']

plt.style.use('seaborn-whitegrid')
fig = plt.figure(figsize=(20,3)) 
plt.subplot(1, 2, 1)
sns.countplot(y="capital-loss", data=dataset_bin);
plt.subplot(1, 2, 2)
sns.distplot(dataset_con['capital-loss']);

Features: Race, Sex, Relationship

# 這些就直接用了
dataset_con['sex'] = dataset_bin['sex'] = dataset_raw['sex']
dataset_con['race'] = dataset_bin['race'] = dataset_raw['race']
dataset_con['relationship'] = dataset_bin['relationship'] = dataset_raw['relationship']`

雙變數

接下來要看特徵之間的關係了

特徵編碼 Feature Encoding

對特徵進行編碼，因為機器學習只認識數字
常見編碼方式Additional Resources: http://pbpython.com/categorical-encoding.html

兩種常見的編碼方式：

• one hot encoding
• label encoding (如果有10種可能，就編碼為0-9)

兩種資料集：

• dataset_bin => 連續值被離散化的資料集
• dataset_con => 非離散化的資料集

One Hot Encodes

# One Hot Encodes 
one_hot_cols = dataset_bin.columns.tolist()
one_hot_cols.remove('predclass')
dataset_bin_enc = pd.get_dummies(dataset_bin, columns=one_hot_cols)

dataset_bin_enc.head()

pd.get_dummie就能得到one hot編碼

Label Encode

# Label Encode 
dataset_con_test = dataset_con

dataset_con_test['workclass'] = dataset_con['workclass'].factorize()[0]
dataset_con_test['occupation'] = dataset_con['occupation'].factorize()[0]
dataset_con_test['native-country'] = dataset_con['native-country'].factorize()[0]

dataset_con_enc = dataset_con_test.apply(LabelEncoder().fit_transform)

dataset_con_enc.head()

如果注視掉中間三個語句dataset_con_test[‘workclass’] 、dataset_con_test[‘occupation’] 、dataset_con_test[‘native-country’] ，就會報錯
TypeError: ("’<’ not supported between instances of ‘str’ and ‘float’", ‘occurred at index workclass’)
原因是這三個屬性中有nan值，所以報錯

# Label Encode 
dataset_con_test = dataset_con

# dataset_con_test['workclass'] = dataset_con['workclass'].factorize()[0]
# dataset_con_test['occupation'] = dataset_con['occupation'].factorize()[0]
# dataset_con_test['native-country'] = dataset_con['native-country'].factorize()[0]

dataset_con_enc = dataset_con_test.apply(LabelEncoder().fit_transform)

dataset_con_enc.head()

參考factorize()用法，該函式使作用於series ，factorize()[0]作用類似於label encode，會將nan轉為-1（LabelEncoder不能處理空值），如下圖

label encoding 例子