[譯] TensorFlow 教程 #02 - 卷積神經網路

活魚眼發表於2017-07-07

01 - 簡單線性模型

by Magnus Erik Hvass Pedersen / GitHub / Videos on YouTube
中文翻譯 thrillerist / Github

如有轉載,請附上本文連結。


介紹

先前的教程展示了一個簡單的線性模型,對MNIST資料集中手寫數字的識別率達到了91%。

在這個教程中,我們會在TensorFlow中實現一個簡單的卷積神經網路,它能達到大約99%的分類準確率,如果你做了一些建議的練習,準確率還可能更高。

卷積神經網路在一張輸入圖片上移動一個小的濾波器。這意味著在遍歷整張影像來識別模式時,要重複使用這些濾波器。這讓卷積神經網路在擁有相同數量的變數時比全連線網路(Fully-Connected)更強大,也讓卷積神經網路訓練得更快。

你應該熟悉基本的線性代數、Python和Jupyter Notebook編輯器。如果你是TensorFlow新手,在本教程之前應該先學習第一篇教程。

流程圖

下面的圖表直接顯示了之後實現的卷積神經網路中資料的傳遞。

from IPython.display import Image
Image('images/02_network_flowchart.png')複製程式碼

輸入影像在第一層卷積層裡使用權重過濾器處理。結果在16張新圖裡,每張代表了卷積層裡一個過濾器(的處理結果)。影像經過降取樣,解析度從28x28減少到14x14。

16張小圖在第二個卷積層中處理。這16個通道以及這層輸出的每個通道都需要一個過濾權重。總共有36個輸出,所以在第二個卷積層有16 x 36 = 576個濾波器。輸出圖再一次降取樣到7x7個畫素。

第二個卷積層的輸出是36張7x7畫素的影像。它們被轉換到一個長為7 x 7 x 36 = 1764的向量中去,它作為一個有128個神經元(或元素)的全連線網路的輸入。這些又輸入到另一個有10個神經元的全連線層中,每個神經元代表一個類別,用來確定影像的類別,即影像上的數字。

卷積濾波一開始是隨機挑選的,因此分類也是隨機完成的。根據交叉熵(cross-entropy)來測量輸入圖預測值和真實類別間的錯誤。然後優化器用鏈式法則自動地將這個誤差在卷積網路中傳遞,更新濾波權重來提升分類質量。這個過程迭代了幾千次,直到分類誤差足夠低。

這些特定的濾波權重和中間影像是一個優化結果,和你執行程式碼所看到的可能會有所不同。

注意,這些在TensorFlow上的計算是在一部分影像上執行,而非單獨的一張圖,這使得計算更有效。也意味著在TensorFlow上實現時,這個流程圖實際上會有更多的資料維度。

卷積層

下面的圖片展示了在第一個卷積層中處理影像的基本思想。輸入圖片描繪了數字7,這裡顯示了它的四張拷貝,我們可以很清晰的看到濾波器是如何在影像的不同位置移動。在濾波器的每個位置上,計算濾波器以及濾波器下方影像畫素的點乘,得到輸出影像的一個畫素。因此,在整張輸入影像上移動時,會有一張新的影像生成。

紅色的濾波權重表示濾波器對輸入圖的黑色畫素有正響應,藍色的代表有負響應。

在這個例子中,很明顯這個濾波器識別數字7的水平線段,在輸出圖中可以看到它對線段的強烈響應。

Image('images/02_convolution.png')複製程式碼

濾波器遍歷輸入圖的移動步長稱為stride。在水平和豎直方向各有一個stride。

在下面的原始碼中,兩個方向的stride都設為1,這說明濾波器從輸入影像的左上角開始,下一步移動到右邊1個畫素去。當濾波器到達影像的右邊時,它會返回最左邊,然後向下移動1個畫素。持續這個過程,直到濾波器到達輸入影像的右下角,同時,也生成了整張輸出圖片。

當濾波器到達輸入圖的右端或底部時,它會用零(白色畫素)來填充。因為輸出圖要和輸入圖一樣大。

此外,卷積層的輸出可能會傳遞給修正線性單元(ReLU),它用來保證輸出是正值,將負值置為零。輸出還會用最大池化(max-pooling)進行降取樣,它使用了2x2的小視窗,只保留畫素中的最大值。這讓輸入圖解析度減小一半,比如從28x28到14x14。

第二個卷積層更加複雜,因為它有16個輸入通道。我們想給每個通道一個單獨的濾波,因此需要16個。另外,我們想從第二個卷積層得到36個輸出,因此總共需要16 x 36 = 576個濾波器。要理解這些如何工作可能有些困難。

匯入

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math複製程式碼

使用Python3.5.2(Anaconda)開發,TensorFlow版本是:

tf.__version__複製程式碼

'0.12.0-rc0'

神經網路的配置

方便起見,在這裡定義神經網路的配置,你可以很容易找到或改變這些數值,然後重新執行Notebook。

# Convolutional Layer 1.
filter_size1 = 5          # Convolution filters are 5 x 5 pixels.
num_filters1 = 16         # There are 16 of these filters.

# Convolutional Layer 2.
filter_size2 = 5          # Convolution filters are 5 x 5 pixels.
num_filters2 = 36         # There are 36 of these filters.

# Fully-connected layer.
fc_size = 128             # Number of neurons in fully-connected layer.複製程式碼

載入資料

MNIST資料集大約12MB,如果沒在資料夾中找到就會自動下載。

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)複製程式碼

Extracting data/MNIST/train-images-idx3-ubyte.gz
Extracting data/MNIST/train-labels-idx1-ubyte.gz
Extracting data/MNIST/t10k-images-idx3-ubyte.gz
Extracting data/MNIST/t10k-labels-idx1-ubyte.gz

現在已經載入了MNIST資料集,它由70,000張影像和對應的標籤(比如影像的類別)組成。資料集分成三份互相獨立的子集。我們在教程中只用訓練集和測試集。

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))複製程式碼

Size of:
-Training-set: 55000
-Test-set: 10000
-Validation-set: 5000

型別標籤使用One-Hot編碼,這意外每個標籤是長為10的向量,除了一個元素之外,其他的都為零。這個元素的索引就是類別的數字,即相應圖片中畫的數字。我們也需要測試資料集類別數字的整型值,用下面的方法來計算。

data.test.cls = np.argmax(data.test.labels, axis=1)複製程式碼

資料維度

在下面的原始碼中,有很多地方用到了資料維度。它們只在一個地方定義,因此我們可以在程式碼中使用這些數字而不是直接寫數字。

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10複製程式碼

用來繪製圖片的幫助函式

這個函式用來在3x3的柵格中畫9張影像,然後在每張影像下面寫出真實類別和預測類別。

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()複製程式碼

繪製幾張影像來看看資料是否正確

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)複製程式碼

TensorFlow圖

TensorFlow的全部目的就是使用一個稱之為計算圖(computational graph)的東西,它會比直接在Python中進行相同計算量要高效得多。TensorFlow比Numpy更高效,因為TensorFlow瞭解整個需要執行的計算圖,然而Numpy只知道某個時間點上唯一的數學運算。

TensorFlow也能夠自動地計算需要優化的變數的梯度,使得模型有更好的表現。這是由於圖是簡單數學表示式的結合,因此整個圖的梯度可以用鏈式法則推匯出來。

TensorFlow還能利用多核CPU和GPU,Google也為TensorFlow製造了稱為TPUs(Tensor Processing Units)的特殊晶片,它比GPU更快。

一個TensorFlow圖由下面幾個部分組成,後面會詳細描述:

  • 佔位符變數(Placeholder)用來改變圖的輸入。
  • 模型變數(Model)將會被優化,使得模型表現得更好。
  • 模型本質上就是一些數學函式,它根據Placeholder和模型的輸入變數來計算一些輸出。
  • 一個cost度量用來指導變數的優化。
  • 一個優化策略會更新模型的變數。

另外,TensorFlow圖也包含了一些除錯狀態,比如用TensorBoard列印log資料,本教程不涉及這些。

建立新變數的幫助函式

函式用來根據給定大小建立TensorFlow變數,並將它們用隨機值初始化。需注意的是在此時並未完成初始化工作,僅僅是在TensorFlow圖裡定義它們。

def new_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))複製程式碼
def new_biases(length):
    return tf.Variable(tf.constant(0.05, shape=[length]))複製程式碼

建立卷積層的幫助函式

這個函式為TensorFlow在計算圖裡建立了新的卷積層。這裡並沒有執行什麼計算,只是在TensorFlow圖裡新增了數學公式。

假設輸入的是四維的張量,各個維度如下:

  1. 影像數量
  2. 每張影像的Y軸
  3. 每張影像的X軸
  4. 每張影像的通道數

輸入通道可能是彩色通道,當輸入是前面的卷積層生成的時候,它也可能是濾波通道。

輸出是另外一個4通道的張量,如下:

  1. 影像數量,與輸入相同
  2. 每張影像的Y軸。如果用到了2x2的池化,是輸入影像寬高的一半。
  3. 每張影像的X軸。同上。
  4. 卷積濾波生成的通道數。
def new_conv_layer(input,              # The previous layer.
                   num_input_channels, # Num. channels in prev. layer.
                   filter_size,        # Width and height of each filter.
                   num_filters,        # Number of filters.
                   use_pooling=True):  # Use 2x2 max-pooling.

    # Shape of the filter-weights for the convolution.
    # This format is determined by the TensorFlow API.
    shape = [filter_size, filter_size, num_input_channels, num_filters]

    # Create new weights aka. filters with the given shape.
    weights = new_weights(shape=shape)

    # Create new biases, one for each filter.
    biases = new_biases(length=num_filters)

    # Create the TensorFlow operation for convolution.
    # Note the strides are set to 1 in all dimensions.
    # The first and last stride must always be 1,
    # because the first is for the image-number and
    # the last is for the input-channel.
    # But e.g. strides=[1, 2, 2, 1] would mean that the filter
    # is moved 2 pixels across the x- and y-axis of the image.
    # The padding is set to 'SAME' which means the input image
    # is padded with zeroes so the size of the output is the same.
    layer = tf.nn.conv2d(input=input,
                         filter=weights,
                         strides=[1, 1, 1, 1],
                         padding='SAME')

    # Add the biases to the results of the convolution.
    # A bias-value is added to each filter-channel.
    layer += biases

    # Use pooling to down-sample the image resolution?
    if use_pooling:
        # This is 2x2 max-pooling, which means that we
        # consider 2x2 windows and select the largest value
        # in each window. Then we move 2 pixels to the next window.
        layer = tf.nn.max_pool(value=layer,
                               ksize=[1, 2, 2, 1],
                               strides=[1, 2, 2, 1],
                               padding='SAME')

    # Rectified Linear Unit (ReLU).
    # It calculates max(x, 0) for each input pixel x.
    # This adds some non-linearity to the formula and allows us
    # to learn more complicated functions.
    layer = tf.nn.relu(layer)

    # Note that ReLU is normally executed before the pooling,
    # but since relu(max_pool(x)) == max_pool(relu(x)) we can
    # save 75% of the relu-operations by max-pooling first.

    # We return both the resulting layer and the filter-weights
    # because we will plot the weights later.
    return layer, weights複製程式碼

轉換一個層的幫助函式

卷積層生成了4維的張量。我們會在卷積層之後新增一個全連線層,因此我們需要將這個4維的張量轉換成可被全連線層使用的2維張量。

def flatten_layer(layer):
    # Get the shape of the input layer.
    layer_shape = layer.get_shape()

    # The shape of the input layer is assumed to be:
    # layer_shape == [num_images, img_height, img_width, num_channels]

    # The number of features is: img_height * img_width * num_channels
    # We can use a function from TensorFlow to calculate this.
    num_features = layer_shape[1:4].num_elements()

    # Reshape the layer to [num_images, num_features].
    # Note that we just set the size of the second dimension
    # to num_features and the size of the first dimension to -1
    # which means the size in that dimension is calculated
    # so the total size of the tensor is unchanged from the reshaping.
    layer_flat = tf.reshape(layer, [-1, num_features])

    # The shape of the flattened layer is now:
    # [num_images, img_height * img_width * num_channels]

    # Return both the flattened layer and the number of features.
    return layer_flat, num_features複製程式碼

建立一個全連線層的幫助函式

這個函式為TensorFlow在計算圖中建立了一個全連線層。這裡也不進行任何計算,只是往TensorFlow圖中新增數學公式。

輸入是大小為[num_images, num_inputs]的二維張量。輸出是大小為[num_images, num_outputs]的2維張量。

def new_fc_layer(input,          # The previous layer.
                 num_inputs,     # Num. inputs from prev. layer.
                 num_outputs,    # Num. outputs.
                 use_relu=True): # Use Rectified Linear Unit (ReLU)?

    # Create new weights and biases.
    weights = new_weights(shape=[num_inputs, num_outputs])
    biases = new_biases(length=num_outputs)

    # Calculate the layer as the matrix multiplication of
    # the input and weights, and then add the bias-values.
    layer = tf.matmul(input, weights) + biases

    # Use ReLU?
    if use_relu:
        layer = tf.nn.relu(layer)

    return layer複製程式碼

佔位符 (Placeholder)變數

Placeholder是作為圖的輸入,每次我們執行圖的時候都可能會改變它們。將這個過程稱為feeding placeholder變數,後面將會描述它。

首先我們為輸入影像定義placeholder變數。這讓我們可以改變輸入到TensorFlow圖中的影像。這也是一個張量(tensor),代表一個多維向量或矩陣。資料型別設定為float32,形狀設為[None, img_size_flat]None代表tensor可能儲存著任意數量的影像,每張圖象是一個長度為img_size_flat的向量。

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')複製程式碼

卷積層希望x被編碼為4維張量,因此我們需要將它的形狀轉換至[num_images, img_height, img_width, num_channels]。注意img_height == img_width == img_size,如果第一維的大小設為-1, num_images的大小也會被自動推匯出來。轉換運算如下:

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])複製程式碼

接下來我們為輸入變數x中的影像所對應的真實標籤定義placeholder變數。變數的形狀是[None, num_classes],這代表著它儲存了任意數量的標籤,每個標籤是長度為num_classes的向量,本例中長度為10。

y_true = tf.placeholder(tf.float32, shape=[None, 10], name='y_true')複製程式碼

我們也可以為class-number提供一個placeholder,但這裡用argmax來計算它。這裡只是TensorFlow中的一些操作,沒有執行什麼運算。

y_true_cls = tf.argmax(y_true, dimension=1)複製程式碼

卷積層 1

建立第一個卷積層。將x_image當作輸入,建立num_filters1個不同的濾波器,每個濾波器的寬高都與 filter_size1相等。最終我們會用2x2的max-pooling將影像降取樣,使它的尺寸減半。

layer_conv1, weights_conv1 = \
    new_conv_layer(input=x_image,
                   num_input_channels=num_channels,
                   filter_size=filter_size1,
                   num_filters=num_filters1,
                   use_pooling=True)複製程式碼

檢查卷積層輸出張量的大小。它是(?,14, 14, 16),這代表著有任意數量的影像(?代表數量),每張影像有14個畫素的寬和高,有16個不同的通道,每個濾波器各有一個通道。

layer_conv1複製程式碼

卷積層 2

建立第二個卷積層,它將第一個卷積層的輸出作為輸入。輸入通道的數量對應著第一個卷積層的濾波數。

layer_conv2, weights_conv2 = \
    new_conv_layer(input=layer_conv1,
                   num_input_channels=num_filters1,
                   filter_size=filter_size2,
                   num_filters=num_filters2,
                   use_pooling=True)複製程式碼

核對一下這個卷積層輸出張量的大小。它的大小是(?, 7, 7, 36),其中?也代表著任意數量的影像,每張圖有7畫素的寬高,每個濾波器有36個通道。

layer_conv2複製程式碼

轉換層

這個卷積層輸出一個4維張量。現在我們想將它作為一個全連線網路的輸入,這就需要將它轉換成2維張量。

layer_flat, num_features = flatten_layer(layer_conv2)複製程式碼

這個張量的大小是(?, 1764),意味著共有一定數量的影像,每張影像被轉換成長為1764的向量。其中1764 = 7 x 7 x 36。

layer_flat複製程式碼
num_features複製程式碼

1764

全連線層 1

往網路中新增一個全連線層。輸入是一個前面卷積得到的被轉換過的層。全連線層中的神經元或節點數為fc_size。我們可以用ReLU來學習非線性關係。

layer_fc1 = new_fc_layer(input=layer_flat,
                         num_inputs=num_features,
                         num_outputs=fc_size,
                         use_relu=True)複製程式碼

全連線層的輸出是一個大小為(?,128)的張量,?代表著一定數量的影像,並且fc_size == 128。

layer_fc1複製程式碼

全連線層 2

新增另外一個全連線層,它的輸出是一個長度為10的向量,它確定了輸入圖是屬於哪個類別。這層並沒有用到ReLU。

layer_fc2 = new_fc_layer(input=layer_fc1,
                         num_inputs=fc_size,
                         num_outputs=num_classes,
                         use_relu=False)複製程式碼
layer_fc2複製程式碼

預測類別

第二個全連線層估算了輸入圖有多大的可能屬於10個類別中的其中一個。然而,這是很粗略的估計並且很難解釋,因為數值可能很小或很大,因此我們會對它們做歸一化,將每個元素限制在0到1之間,並且相加為1。這用一個稱為softmax的函式來計算的,結果儲存在y_pred中。

y_pred = tf.nn.softmax(layer_fc2)複製程式碼

類別數字是最大元素的索引。

y_pred_cls = tf.argmax(y_pred, dimension=1)複製程式碼

優化損失函式

為了使模型更好地對輸入影像進行分類,我們必須改變weightsbiases變數。首先我們需要對比模型y_pred的預測輸出和期望輸出的y_true,來了解目前模型的效能如何。

交叉熵(cross-entropy)是在分類中使用的效能度量。交叉熵是一個常為正值的連續函式,如果模型的預測值精準地符合期望的輸出,它就等於零。因此,優化的目的就是通過改變網路層的變數來最小化交叉熵。

TensorFlow有一個內建的計算交叉熵的函式。這個函式內部計算了softmax,所以我們要用layer_fc2的輸出而非直接用y_pred,因為y_pred上已經計算了softmax。

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
                                                        labels=y_true)複製程式碼

我們為每個影像分類計算了交叉熵,所以有一個當前模型在每張圖上表現的度量。但是為了用交叉熵來指導模型變數的優化,我們需要一個額外的標量值,因此簡單地利用所有影像分類交叉熵的均值。

cost = tf.reduce_mean(cross_entropy)複製程式碼

優化方法

既然我們有一個需要被最小化的損失度量,接著就可以建立優化一個優化器。這個例子中,我們使用的是梯度下降的變體AdamOptimizer

優化過程並不是在這裡執行。實際上,還沒計算任何東西,我們只是往TensorFlow圖中新增了優化器,以便之後的操作。

optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cost)複製程式碼

效能度量

我們需要另外一些效能度量,來向使用者展示這個過程。

這是一個布林值向量,代表預測型別是否等於每張圖片的真實型別。

correct_prediction = tf.equal(y_pred_cls, y_true_cls)複製程式碼

上面的計算先將布林值向量型別轉換成浮點型向量,這樣子False就變成0,True變成1,然後計算這些值的平均數,以此來計算分類的準確度。

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))複製程式碼

執行TensorFlow

建立TensorFlow會話(session)

一旦建立了TensorFlow圖,我們需要建立一個TensorFlow會話,用來執行圖。

session = tf.Session()複製程式碼

初始化變數

我們需要在開始優化weights和biases變數之前對它們進行初始化。

session.run(tf.global_variables_initializer())複製程式碼

用來優化迭代的幫助函式

在訓練集中有50,000張圖。用這些影像計算模型的梯度會花很多時間。因此我們利用隨機梯度下降的方法,它在優化器的每次迭代裡只用到了一小部分的影像。

如果記憶體耗盡導致電腦當機或變得很慢,你應該試著減少這些數量,但同時可能還需要更優化的迭代。

train_batch_size = 64複製程式碼

函式執行了多次的優化迭代來逐步地提升網路層的變數。在每次迭代中,從訓練集中選擇一批新的資料,然後TensorFlow用這些訓練樣本來執行優化器。每100次迭代會列印出相關資訊。

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))複製程式碼

用來繪製錯誤樣本的幫助函式

函式用來繪製測試集中被誤分類的樣本。

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])複製程式碼

繪製混淆(confusion)矩陣的幫助函式

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()複製程式碼

展示效能的幫助函式

函式用來列印測試集上的分類準確度。

為測試集上的所有圖片計算分類會花費一段時間,因此我們直接用這個函式來呼叫上面的結果,這樣就不用每次都重新計算了。

這個函式可能會佔用很多電腦記憶體,這也是為什麼將測試集分成更小的幾個部分。如果你的電腦記憶體比較小或當機了,就要試著降低batch-size。

# Split the test-set into smaller batches of this size.
test_batch_size = 256

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # Number of images in the test-set.
    num_test = len(data.test.images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_test, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_test:
        # The ending index for the next batch is denoted j.
        j = min(i + test_batch_size, num_test)

        # Get the images from the test-set between index i and j.
        images = data.test.images[i:j, :]

        # Get the associated labels.
        labels = data.test.labels[i:j, :]

        # Create a feed-dict with these images and labels.
        feed_dict = {x: images,
                     y_true: labels}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Convenience variable for the true class-numbers of the test-set.
    cls_true = data.test.cls

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / num_test

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, correct_sum, num_test))

    # Plot some examples of mis-classifications, if desired.
    if show_example_errors:
        print("Example errors:")
        plot_example_errors(cls_pred=cls_pred, correct=correct)

    # Plot the confusion matrix, if desired.
    if show_confusion_matrix:
        print("Confusion Matrix:")
        plot_confusion_matrix(cls_pred=cls_pred)複製程式碼

優化之前的效能

測試集上的準確度很低,這是由於模型只做了初始化,並沒做任何優化,所以它只是對影像做隨機分類。

print_test_accuracy()複製程式碼

Accuracy on Test-Set: 10.9% (1093 / 10000)

1次迭代後的效能

做了一次優化後,此時優化器的學習率很低,效能其實並沒有多大提升。

optimize(num_iterations=1)複製程式碼

Optimization Iteration: 1, Training Accuracy: 6.2%
Time usage: 0:00:00

print_test_accuracy()複製程式碼

Accuracy on Test-Set: 13.0% (1296 / 10000)

100次迭代優化後的效能

100次優化迭代之後,模型顯著地提升了分類的準確度。

optimize(num_iterations=99) # We already performed 1 iteration above.複製程式碼

Time usage: 0:00:00

print_test_accuracy(show_example_errors=True)複製程式碼

Accuracy on Test-Set: 66.6% (6656 / 10000)
Example errors:

1000次優化迭代後的效能

1000次優化迭代之後,模型在測試集上的準確度超過了90%。

optimize(num_iterations=900) # We performed 100 iterations above.複製程式碼

Optimization Iteration: 101, Training Accuracy: 71.9%
Optimization Iteration: 201, Training Accuracy: 76.6%
Optimization Iteration: 301, Training Accuracy: 71.9%
Optimization Iteration: 401, Training Accuracy: 85.9%
Optimization Iteration: 501, Training Accuracy: 89.1%
Optimization Iteration: 601, Training Accuracy: 95.3%
Optimization Iteration: 701, Training Accuracy: 90.6%
Optimization Iteration: 801, Training Accuracy: 92.2%
Optimization Iteration: 901, Training Accuracy: 95.3%
Time usage: 0:00:03

print_test_accuracy(show_example_errors=True)複製程式碼

Accuracy on Test-Set: 93.1% (9308 / 10000)
Example errors:

10,000次優化迭代後的效能

經過10,000次優化迭代後,測試集上的分類準確率高達99%。

optimize(num_iterations=9000) # We performed 1000 iterations above.複製程式碼

Optimization Iteration: 1001, Training Accuracy: 98.4%
Optimization Iteration: 1101, Training Accuracy: 93.8%
Optimization Iteration: 1201, Training Accuracy: 92.2%
Optimization Iteration: 1301, Training Accuracy: 95.3%
Optimization Iteration: 1401, Training Accuracy: 93.8%
Optimization Iteration: 1501, Training Accuracy: 93.8%
Optimization Iteration: 1601, Training Accuracy: 92.2%
Optimization Iteration: 1701, Training Accuracy: 92.2%
Optimization Iteration: 1801, Training Accuracy: 89.1%
Optimization Iteration: 1901, Training Accuracy: 95.3%
Optimization Iteration: 2001, Training Accuracy: 93.8%
Optimization Iteration: 2101, Training Accuracy: 98.4%
Optimization Iteration: 2201, Training Accuracy: 92.2%
Optimization Iteration: 2301, Training Accuracy: 95.3%
Optimization Iteration: 2401, Training Accuracy: 100.0%
Optimization Iteration: 2501, Training Accuracy: 96.9%
Optimization Iteration: 2601, Training Accuracy: 93.8%
Optimization Iteration: 2701, Training Accuracy: 100.0%
Optimization Iteration: 2801, Training Accuracy: 95.3%
Optimization Iteration: 2901, Training Accuracy: 95.3%
Optimization Iteration: 3001, Training Accuracy: 96.9%
Optimization Iteration: 3101, Training Accuracy: 96.9%
Optimization Iteration: 3201, Training Accuracy: 95.3%
Optimization Iteration: 3301, Training Accuracy: 96.9%
Optimization Iteration: 3401, Training Accuracy: 98.4%
Optimization Iteration: 3501, Training Accuracy: 100.0%
Optimization Iteration: 3601, Training Accuracy: 98.4%
Optimization Iteration: 3701, Training Accuracy: 95.3%
Optimization Iteration: 3801, Training Accuracy: 95.3%
Optimization Iteration: 3901, Training Accuracy: 95.3%
Optimization Iteration: 4001, Training Accuracy: 100.0%
Optimization Iteration: 4101, Training Accuracy: 93.8%
Optimization Iteration: 4201, Training Accuracy: 95.3%
Optimization Iteration: 4301, Training Accuracy: 100.0%
Optimization Iteration: 4401, Training Accuracy: 96.9%
Optimization Iteration: 4501, Training Accuracy: 100.0%
Optimization Iteration: 4601, Training Accuracy: 100.0%
Optimization Iteration: 4701, Training Accuracy: 100.0%
Optimization Iteration: 4801, Training Accuracy: 98.4%
Optimization Iteration: 4901, Training Accuracy: 98.4%
Optimization Iteration: 5001, Training Accuracy: 98.4%
Optimization Iteration: 5101, Training Accuracy: 100.0%
Optimization Iteration: 5201, Training Accuracy: 95.3%
Optimization Iteration: 5301, Training Accuracy: 96.9%
Optimization Iteration: 5401, Training Accuracy: 100.0%
Optimization Iteration: 5501, Training Accuracy: 100.0%
Optimization Iteration: 5601, Training Accuracy: 100.0%
Optimization Iteration: 5701, Training Accuracy: 96.9%
Optimization Iteration: 5801, Training Accuracy: 98.4%
Optimization Iteration: 5901, Training Accuracy: 100.0%
Optimization Iteration: 6001, Training Accuracy: 95.3%
Optimization Iteration: 6101, Training Accuracy: 96.9%
Optimization Iteration: 6201, Training Accuracy: 100.0%
Optimization Iteration: 6301, Training Accuracy: 96.9%
Optimization Iteration: 6401, Training Accuracy: 100.0%
Optimization Iteration: 6501, Training Accuracy: 98.4%
Optimization Iteration: 6601, Training Accuracy: 98.4%
Optimization Iteration: 6701, Training Accuracy: 95.3%
Optimization Iteration: 6801, Training Accuracy: 100.0%
Optimization Iteration: 6901, Training Accuracy: 98.4%
Optimization Iteration: 7001, Training Accuracy: 95.3%
Optimization Iteration: 7101, Training Accuracy: 100.0%
Optimization Iteration: 7201, Training Accuracy: 100.0%
Optimization Iteration: 7301, Training Accuracy: 100.0%
Optimization Iteration: 7401, Training Accuracy: 100.0%
Optimization Iteration: 7501, Training Accuracy: 100.0%
Optimization Iteration: 7601, Training Accuracy: 96.9%
Optimization Iteration: 7701, Training Accuracy: 98.4%
Optimization Iteration: 7801, Training Accuracy: 95.3%
Optimization Iteration: 7901, Training Accuracy: 100.0%
Optimization Iteration: 8001, Training Accuracy: 100.0%
Optimization Iteration: 8101, Training Accuracy: 98.4%
Optimization Iteration: 8201, Training Accuracy: 98.4%
Optimization Iteration: 8301, Training Accuracy: 100.0%
Optimization Iteration: 8401, Training Accuracy: 96.9%
Optimization Iteration: 8501, Training Accuracy: 98.4%
Optimization Iteration: 8601, Training Accuracy: 98.4%
Optimization Iteration: 8701, Training Accuracy: 100.0%
Optimization Iteration: 8801, Training Accuracy: 100.0%
Optimization Iteration: 8901, Training Accuracy: 98.4%
Optimization Iteration: 9001, Training Accuracy: 95.3%
Optimization Iteration: 9101, Training Accuracy: 100.0%
Optimization Iteration: 9201, Training Accuracy: 100.0%
Optimization Iteration: 9301, Training Accuracy: 96.9%
Optimization Iteration: 9401, Training Accuracy: 96.9%
Optimization Iteration: 9501, Training Accuracy: 98.4%
Optimization Iteration: 9601, Training Accuracy: 100.0%
Optimization Iteration: 9701, Training Accuracy: 96.9%
Optimization Iteration: 9801, Training Accuracy: 98.4%
Optimization Iteration: 9901, Training Accuracy: 98.4%
Time usage: 0:00:26

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)複製程式碼

Accuracy on Test-Set: 98.8% (9880 / 10000)
Example errors:

Confusion Matrix:
[[ 973 0 1 0 0 1 1 0 3 1]
[ 0 1129 2 1 0 0 1 1 1 0]
[ 1 2 1023 2 0 0 0 2 2 0]
[ 1 0 1 1002 0 3 0 1 2 0]
[ 0 1 0 0 974 0 1 0 2 4]
[ 2 0 0 3 0 882 2 0 1 2]
[ 4 1 0 0 1 4 948 0 0 0]
[ 1 4 11 2 0 0 0 1004 2 4]
[ 3 0 4 2 1 2 0 0 960 2]
[ 3 4 1 0 7 5 0 2 2 985]]

權重和層的視覺化

為了理解為什麼卷積神經網路可以識別手寫數字,我們將會對卷積濾波和輸出影像進行視覺化。

繪製卷積權重的幫助函式

def plot_conv_weights(weights, input_channel=0):
    # Assume weights are TensorFlow ops for 4-dim variables
    # e.g. weights_conv1 or weights_conv2.

    # Retrieve the values of the weight-variables from TensorFlow.
    # A feed-dict is not necessary because nothing is calculated.
    w = session.run(weights)

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Number of filters used in the conv. layer.
    num_filters = w.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i<num_filters:
            # Get the weights for the i'th filter of the input channel.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = w[:, :, input_channel, i]

            # Plot image.
            ax.imshow(img, vmin=w_min, vmax=w_max,
                      interpolation='nearest', cmap='seismic')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()複製程式碼

繪製卷積層輸出的幫助函式

def plot_conv_layer(layer, image):
    # Assume layer is a TensorFlow op that outputs a 4-dim tensor
    # which is the output of a convolutional layer,
    # e.g. layer_conv1 or layer_conv2.

    # Create a feed-dict containing just one image.
    # Note that we don't need to feed y_true because it is
    # not used in this calculation.
    feed_dict = {x: [image]}

    # Calculate and retrieve the output values of the layer
    # when inputting that image.
    values = session.run(layer, feed_dict=feed_dict)

    # Number of filters used in the conv. layer.
    num_filters = values.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot the output images of all the filters.
    for i, ax in enumerate(axes.flat):
        # Only plot the images for valid filters.
        if i<num_filters:
            # Get the output image of using the i'th filter.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap='binary')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()複製程式碼

輸入影像

繪製影像的幫助函式

def plot_image(image):
    plt.imshow(image.reshape(img_shape),
               interpolation='nearest',
               cmap='binary')

    plt.show()複製程式碼

如下所示,繪製一張測試集中的影像。

image1 = data.test.images[0]
plot_image(image1)複製程式碼

繪製測試集裡的另一張影像。

image2 = data.test.images[13]
plot_image(image2)複製程式碼

卷積層 1

現在繪製第一個卷積層的濾波權重。

其中正值權重是紅色的,負值為藍色。

plot_conv_weights(weights=weights_conv1)複製程式碼

將這些卷積濾波新增到第一張輸入影像,得到以下輸出,它們也作為第二個卷積層的輸入。注意這些影像被降取樣到14 x 14畫素,即原始輸入圖解析度的一半。

plot_conv_layer(layer=layer_conv1, image=image1)複製程式碼

下面是將卷積濾波新增到第二張影像的結果。

plot_conv_layer(layer=layer_conv1, image=image2)複製程式碼

從這些影像很難看出卷積濾波的作用是什麼。顯然,它們生成了輸入影像的一些變體,就像光線從不同角度打到影像上併產生陰影一樣。

卷積層 2

現在繪製第二個卷積層的濾波權重。

第一個卷積層有16個輸出通道,代表著第二個卷基層有16個輸入。第二個卷積層的每個輸入通道也有一些權重濾波。我們先繪製第一個通道的權重濾波。

同樣的,正值是紅色,負值是藍色。

plot_conv_weights(weights=weights_conv2, input_channel=0)複製程式碼

第二個卷積層共有16個輸入通道,我們可以同樣地畫出其他影像。這裡我們畫出第二個通道的影像。

plot_conv_weights(weights=weights_conv2, input_channel=1)複製程式碼

由於這些濾波是高維度的,很難理解它們是如何應用的。

給第一個卷積層的輸出加上這些濾波,得到下面的影像。

這些影像被降取樣至7 x 7的畫素,即上一個卷積層輸出的一半。

plot_conv_layer(layer=layer_conv2, image=image1)複製程式碼

這是給第二張影像加上濾波權重的結果。

plot_conv_layer(layer=layer_conv2, image=image2)複製程式碼

從這些影像來看,似乎第二個卷積層會檢測輸入影像中的線段和模式,這對輸入圖中的區域性變化不那麼敏感。

關閉TensorFlow會話

現在我們已經用TensorFlow完成了任務,關閉session,釋放資源。

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()複製程式碼

總結

我們看到卷積神經網路在識別手寫數字上的表現要比教程#01中簡單線性模型要好得多。卷積神經網路可能達到99%的分類準確率,如果你做一些調整,還可能表現得更好,而簡單線性模型只有91%的正確率。

然而,卷積神經網路實現起來更復雜,並且光看權重濾波也不好理解為什麼它能奏效或者失敗。

因此我們需要一個更簡單的實現卷積神經網路的方式,同時也要尋找一種更好的方法來對它們內部工作原理進行視覺化。

練習

下面使一些可能會讓你提升TensorFlow技能的一些建議練習。為了學習如何更合適地使用TensorFlow,實踐經驗是很重要的。

在你對這個Notebook進行修改之前,可能需要先備份一下。

  • 如果你不改變任何引數,多次執行Notebook,會得到完成一樣的結果嗎?隨機性的來源是什麼?

  • 再進行10,000次優化。結果有變好麼?

  • 改變優化器的學習率。

  • 改變層次的屬性,比如卷積濾波器數量、濾波器的大小、全連線層中的神經元數量等等。

  • 在全連線層之後新增一個drop-out層。在計算分類準確率的時候,drop-out層可能為0,因此你需要一個placeholder變數。

  • 改變ReLU和max-pooling的順序。它的計算結果相同麼?最快的計算方法是什麼?節省了多少計算量?這也適用於Sigmoid-function和average-pooling嗎?

  • 新增一個或多個卷積層和全連線層。這對效能有幫助嗎?

  • 能得到良好結果的最小可能配置是什麼?

  • 試著在最後一個全連線層中使用ReLU。效能有變化嗎?為什麼?

  • 卷積層裡不用pooling。這對分類準確率和訓練時間有影響嗎?

  • 在卷積層裡用2x2的stride代替max-pooling?有什麼變化嗎?

  • 不看原始碼,自己重寫程式。

  • 向朋友解釋程式如何工作。

相關文章