Pandas之:Pandas高階教程以鐵達尼號真實資料為例

flydean發表於2021-06-07

Pandas之:Pandas高階教程以鐵達尼號真實資料為例

簡介

今天我們會講解一下Pandas的高階教程,包括讀寫檔案、選取子集和圖形表示等。

讀寫檔案

資料處理的一個關鍵步驟就是讀取檔案進行分析,然後將分析處理結果再次寫入檔案。

Pandas支援多種檔案格式的讀取和寫入:

In [108]: pd.read_
 read_clipboard() read_excel()     read_fwf()       read_hdf()       read_json        read_parquet     read_sas         read_sql_query   read_stata
 read_csv         read_feather()   read_gbq()       read_html        read_msgpack     read_pickle      read_sql         read_sql_table   read_table

接下來我們會以Pandas官網提供的Titanic.csv為例來講解Pandas的使用。

Titanic.csv提供了800多個泰坦利特號上乘客的資訊,是一個891 rows x 12 columns的矩陣。

我們使用Pandas來讀取這個csv:

In [5]: titanic=pd.read_csv("titanic.csv")

read_csv方法會將csv檔案轉換成為pandas 的DataFrame

預設情況下我們直接使用DF變數,會預設展示前5行和後5行資料:

In [3]: titanic
Out[3]: 
     PassengerId  Survived  Pclass                                               Name     Sex  ...  Parch            Ticket     Fare Cabin  Embarked
0              1         0       3                            Braund, Mr. Owen Harris    male  ...      0         A/5 21171   7.2500   NaN         S
1              2         1       1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  ...      0          PC 17599  71.2833   C85         C
2              3         1       3                             Heikkinen, Miss. Laina  female  ...      0  STON/O2. 3101282   7.9250   NaN         S
3              4         1       1       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  ...      0            113803  53.1000  C123         S
4              5         0       3                           Allen, Mr. William Henry    male  ...      0            373450   8.0500   NaN         S
..           ...       ...     ...                                                ...     ...  ...    ...               ...      ...   ...       ...
886          887         0       2                              Montvila, Rev. Juozas    male  ...      0            211536  13.0000   NaN         S
887          888         1       1                       Graham, Miss. Margaret Edith  female  ...      0            112053  30.0000   B42         S
888          889         0       3           Johnston, Miss. Catherine Helen "Carrie"  female  ...      2        W./C. 6607  23.4500   NaN         S
889          890         1       1                              Behr, Mr. Karl Howell    male  ...      0            111369  30.0000  C148         C
890          891         0       3                                Dooley, Mr. Patrick    male  ...      0            370376   7.7500   NaN         Q

[891 rows x 12 columns]

可以使用head(n)和tail(n)來指定特定的行數:

In [4]: titanic.head(8)
Out[4]: 
   PassengerId  Survived  Pclass                                               Name     Sex  ...  Parch            Ticket     Fare Cabin  Embarked
0            1         0       3                            Braund, Mr. Owen Harris    male  ...      0         A/5 21171   7.2500   NaN         S
1            2         1       1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  ...      0          PC 17599  71.2833   C85         C
2            3         1       3                             Heikkinen, Miss. Laina  female  ...      0  STON/O2. 3101282   7.9250   NaN         S
3            4         1       1       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  ...      0            113803  53.1000  C123         S
4            5         0       3                           Allen, Mr. William Henry    male  ...      0            373450   8.0500   NaN         S
5            6         0       3                                   Moran, Mr. James    male  ...      0            330877   8.4583   NaN         Q
6            7         0       1                            McCarthy, Mr. Timothy J    male  ...      0             17463  51.8625   E46         S
7            8         0       3                     Palsson, Master. Gosta Leonard    male  ...      1            349909  21.0750   NaN         S

[8 rows x 12 columns]

使用dtypes可以檢視每一列的資料型別:

In [5]: titanic.dtypes
Out[5]: 
PassengerId      int64
Survived         int64
Pclass           int64
Name            object
Sex             object
Age            float64
SibSp            int64
Parch            int64
Ticket          object
Fare           float64
Cabin           object
Embarked        object
dtype: object

使用to_excel可以將DF轉換為excel檔案,使用read_excel可以再次讀取excel檔案:

In [11]: titanic.to_excel('titanic.xlsx', sheet_name='passengers', index=False)

In [12]: titanic = pd.read_excel('titanic.xlsx', sheet_name='passengers')

使用info()可以來對DF進行一個初步的統計:

In [14]: titanic.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB

DF的選擇

選擇列資料

DF的head或者tail方法只能顯示所有的列資料,下面的方法可以選擇特定的列資料。

In [15]: ages = titanic["Age"]

In [16]: ages.head()
Out[16]:
0    22.0
1    38.0
2    26.0
3    35.0
4    35.0
Name: Age, dtype: float64

每一列都是一個Series:

In [6]: type(titanic["Age"])
Out[6]: pandas.core.series.Series

In [7]: titanic["Age"].shape
Out[7]: (891,)

還可以多選:

In [8]: age_sex = titanic[["Age", "Sex"]]

In [9]: age_sex.head()
Out[9]: 
    Age     Sex
0  22.0    male
1  38.0  female
2  26.0  female
3  35.0  female
4  35.0    male

如果選擇多列的話,返回的結果就是一個DF型別:

In [10]: type(titanic[["Age", "Sex"]])
Out[10]: pandas.core.frame.DataFrame

In [11]: titanic[["Age", "Sex"]].shape
Out[11]: (891, 2)

選擇行資料

上面我們講到了怎麼選擇列資料,下面我們來看看怎麼選擇行資料:

選擇客戶年齡大於35歲的:

In [12]: above_35 = titanic[titanic["Age"] > 35]

In [13]: above_35.head()
Out[13]: 
    PassengerId  Survived  Pclass                                               Name     Sex  ...  Parch    Ticket     Fare Cabin  Embarked
1             2         1       1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  ...      0  PC 17599  71.2833   C85         C
6             7         0       1                            McCarthy, Mr. Timothy J    male  ...      0     17463  51.8625   E46         S
11           12         1       1                           Bonnell, Miss. Elizabeth  female  ...      0    113783  26.5500  C103         S
13           14         0       3                        Andersson, Mr. Anders Johan    male  ...      5    347082  31.2750   NaN         S
15           16         1       2                   Hewlett, Mrs. (Mary D Kingcome)   female  ...      0    248706  16.0000   NaN         S

[5 rows x 12 columns]

使用isin選擇Pclass在2和3的所有客戶:

In [16]: class_23 = titanic[titanic["Pclass"].isin([2, 3])]
In [17]: class_23.head()
Out[17]: 
   PassengerId  Survived  Pclass                            Name     Sex   Age  SibSp  Parch            Ticket     Fare Cabin Embarked
0            1         0       3         Braund, Mr. Owen Harris    male  22.0      1      0         A/5 21171   7.2500   NaN        S
2            3         1       3          Heikkinen, Miss. Laina  female  26.0      0      0  STON/O2. 3101282   7.9250   NaN        S
4            5         0       3        Allen, Mr. William Henry    male  35.0      0      0            373450   8.0500   NaN        S
5            6         0       3                Moran, Mr. James    male   NaN      0      0            330877   8.4583   NaN        Q
7            8         0       3  Palsson, Master. Gosta Leonard    male   2.0      3      1            349909  21.0750   NaN        S

上面的isin等於:

In [18]: class_23 = titanic[(titanic["Pclass"] == 2) | (titanic["Pclass"] == 3)]

篩選Age不是空的:

In [20]: age_no_na = titanic[titanic["Age"].notna()]

In [21]: age_no_na.head()
Out[21]: 
   PassengerId  Survived  Pclass                                               Name     Sex  ...  Parch            Ticket     Fare Cabin  Embarked
0            1         0       3                            Braund, Mr. Owen Harris    male  ...      0         A/5 21171   7.2500   NaN         S
1            2         1       1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  ...      0          PC 17599  71.2833   C85         C
2            3         1       3                             Heikkinen, Miss. Laina  female  ...      0  STON/O2. 3101282   7.9250   NaN         S
3            4         1       1       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  ...      0            113803  53.1000  C123         S
4            5         0       3                           Allen, Mr. William Henry    male  ...      0            373450   8.0500   NaN         S

[5 rows x 12 columns]

同時選擇行和列

我們可以同時選擇行和列。

使用loc和iloc可以進行行和列的選擇,他們兩者的區別是loc是使用名字進行選擇,iloc是使用數字進行選擇。

選擇age>35的乘客名:

In [23]: adult_names = titanic.loc[titanic["Age"] > 35, "Name"]

In [24]: adult_names.head()
Out[24]: 
1     Cumings, Mrs. John Bradley (Florence Briggs Th...
6                               McCarthy, Mr. Timothy J
11                             Bonnell, Miss. Elizabeth
13                          Andersson, Mr. Anders Johan
15                     Hewlett, Mrs. (Mary D Kingcome) 
Name: Name, dtype: object

loc中第一個值表示行選擇,第二個值表示列選擇。

使用iloc進行選擇:

In [25]: titanic.iloc[9:25, 2:5]
Out[25]: 
    Pclass                                 Name     Sex
9        2  Nasser, Mrs. Nicholas (Adele Achem)  female
10       3      Sandstrom, Miss. Marguerite Rut  female
11       1             Bonnell, Miss. Elizabeth  female
12       3       Saundercock, Mr. William Henry    male
13       3          Andersson, Mr. Anders Johan    male
..     ...                                  ...     ...
20       2                 Fynney, Mr. Joseph J    male
21       2                Beesley, Mr. Lawrence    male
22       3          McGowan, Miss. Anna "Annie"  female
23       1         Sloper, Mr. William Thompson    male
24       3        Palsson, Miss. Torborg Danira  female

[16 rows x 3 columns]

使用plots作圖

怎麼將DF轉換成為多樣化的圖形展示呢?

要想在命令列中使用matplotlib作圖,那麼需要啟動ipython的QT環境:

ipython qtconsole --pylab=inline

直接使用plot來展示一下上面我們讀取的乘客資訊:

import matplotlib.pyplot as plt

import pandas as pd

titanic = pd.read_excel('titanic.xlsx', sheet_name='passengers')

titanic.plot()

橫座標就是DF中的index,列座標是各個列的名字。注意上面的列只展示的是數值型別的。

我們只展示age資訊:

titanic['Age'].plot()

預設的是柱狀圖,我們可以轉換圖形的形式,比如點圖:

titanic.plot.scatter(x="PassengerId",y="Age", alpha=0.5)

選擇資料中的PassengerId作為x軸,age作為y軸:

除了散點圖,還支援很多其他的影像:

[method_name for method_name in dir(titanic.plot) if not method_name.startswith("_")]
Out[11]: 
['area',
 'bar',
 'barh',
 'box',
 'density',
 'hexbin',
 'hist',
 'kde',
 'line',
 'pie',
 'scatter']

再看一個box圖:

titanic['Age'].plot.box()

可以看到,乘客的年齡大多集中在20-40歲之間。

還可以將選擇的多列分別作圖展示:

titanic.plot.area(figsize=(12, 4), subplots=True)

指定特定的列:

titanic[['Age','Pclass']].plot.area(figsize=(12, 4), subplots=True)

還可以先畫圖,然後填充:

fig, axs = plt.subplots(figsize=(12, 4));

先畫一個空的圖,然後對其進行填充:

titanic['Age'].plot.area(ax=axs);

axs.set_ylabel("Age");

fig

使用現有的列建立新的列

有時候,我們需要對現有的列進行變換,以得到新的列,比如我們想新增一個Age2列,它的值是Age列+10,則可以這樣:

titanic["Age2"]=titanic["Age"]+10;

titanic[["Age","Age2"]].head()
Out[34]: 
    Age  Age2
0  22.0  32.0
1  38.0  48.0
2  26.0  36.0
3  35.0  45.0
4  35.0  45.0

還可以對列進行重新命名:

titanic_renamed = titanic.rename(
   ...:     columns={"Age": "Age2",
   ...:              "Pclass": "Pclas2"})

列名轉換為小寫:

titanic_renamed = titanic_renamed.rename(columns=str.lower)

進行統計

我們來統計下乘客的平均年齡:

titanic["Age"].mean()
Out[35]: 29.69911764705882

選擇中位數:

titanic[["Age", "Fare"]].median()
Out[36]: 
Age     28.0000
Fare    14.4542
dtype: float64

更多資訊:

titanic[["Age", "Fare"]].describe()
Out[37]: 
              Age        Fare
count  714.000000  891.000000
mean    29.699118   32.204208
std     14.526497   49.693429
min      0.420000    0.000000
25%     20.125000    7.910400
50%     28.000000   14.454200
75%     38.000000   31.000000
max     80.000000  512.329200

使用agg指定特定的聚合方法:

titanic.agg({'Age': ['min', 'max', 'median', 'skew'],'Fare': ['min', 'max', 'median', 'mean']})
Out[38]: 
              Age        Fare
max     80.000000  512.329200
mean          NaN   32.204208
median  28.000000   14.454200
min      0.420000    0.000000
skew     0.389108         NaN

可以使用groupby:

titanic[["Sex", "Age"]].groupby("Sex").mean()
Out[39]: 
              Age
Sex              
female  27.915709
male    30.726645

groupby所有的列:

titanic.groupby("Sex").mean()
Out[40]: 
        PassengerId  Survived    Pclass        Age     SibSp     Parch  
Sex                                                                      
female   431.028662  0.742038  2.159236  27.915709  0.694268  0.649682   
male     454.147314  0.188908  2.389948  30.726645  0.429809  0.235702   

groupby之後還可以選擇特定的列:

titanic.groupby("Sex")["Age"].mean()
Out[41]: 
Sex
female    27.915709
male      30.726645
Name: Age, dtype: float64

可以分類進行count:

titanic["Pclass"].value_counts()
Out[42]: 
3    491
1    216
2    184
Name: Pclass, dtype: int64

上面等同於:

titanic.groupby("Pclass")["Pclass"].count()

DF重組

可以根據某列進行排序:

titanic.sort_values(by="Age").head()
Out[43]: 
     PassengerId  Survived  Pclass                             Name     Sex  \
803          804         1       3  Thomas, Master. Assad Alexander    male   
755          756         1       2        Hamalainen, Master. Viljo    male   
644          645         1       3           Baclini, Miss. Eugenie  female   
469          470         1       3    Baclini, Miss. Helene Barbara  female   
78            79         1       2    Caldwell, Master. Alden Gates    male   

根據多列排序:

titanic.sort_values(by=['Pclass', 'Age'], ascending=False).head()
Out[44]: 
     PassengerId  Survived  Pclass                       Name     Sex   Age  \
851          852         0       3        Svensson, Mr. Johan    male  74.0   
116          117         0       3       Connors, Mr. Patrick    male  70.5   
280          281         0       3           Duane, Mr. Frank    male  65.0   
483          484         1       3     Turkula, Mrs. (Hedwig)  female  63.0   
326          327         0       3  Nysveen, Mr. Johan Hansen    male  61.0   

選擇特定的行和列資料,下面的例子我們將會選擇性別為女性的部分資料:

female=titanic[titanic['Sex']=='female']

female_subset=female[["Age","Pclass","PassengerId","Survived"]].sort_values(["Pclass"]).groupby(["Pclass"]).head(2)

female_subset
Out[58]: 
      Age  Pclass  PassengerId  Survived
1    38.0       1            2         1
356  22.0       1          357         1
726  30.0       2          727         1
443  28.0       2          444         1
855  18.0       3          856         1
654  18.0       3          655         0

使用pivot可以進行軸的轉換:

female_subset.pivot(columns="Pclass", values="Age")
Out[62]: 
Pclass     1     2     3
1       38.0   NaN   NaN
356     22.0   NaN   NaN
443      NaN  28.0   NaN
654      NaN   NaN  18.0
726      NaN  30.0   NaN
855      NaN   NaN  18.0

female_subset.pivot(columns="Pclass", values="Age").plot()

本文已收錄於 http://www.flydean.com/02-python-pandas-advanced/

最通俗的解讀,最深刻的乾貨,最簡潔的教程,眾多你不知道的小技巧等你來發現!

歡迎關注我的公眾號:「程式那些事」,懂技術,更懂你!

相關文章