[action]tensorflow深度學習實戰 (4) 實現簡單卷積神經網路
我們首先匯入已經清洗好的資料。
這個清洗過程在之前的博文實戰(1)。
# These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
from __future__ import print_function
import numpy as np
import tensorflow as tf
from six.moves import cPickle as pickle
from six.moves import range
pickle_file = 'notMNIST.pickle'
with open(pickle_file, 'rb') as f:
save = pickle.load(f)
train_dataset = save['train_dataset']
train_labels = save['train_labels']
valid_dataset = save['valid_dataset']
valid_labels = save['valid_labels']
test_dataset = save['test_dataset']
test_labels = save['test_labels']
del save # hint to help gc free up memory
print('Training set', train_dataset.shape, train_labels.shape)
print('Validation set', valid_dataset.shape, valid_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)卷積神經網路需要的圖片矩陣是三維的，這一點需要做一下改變。(利用np.reshape)
其次，所有的label都是one-shot的。
image_size = 28
num_labels = 10
num_channels = 1 # grayscale
# it is a picture in gray scale originally we only need add a dimention
def reformat(dataset, labels):
dataset = dataset.reshape((-1, image_size, image_size, num_channels)).astype(np.float32)
labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
return dataset, labels
train_dataset, train_labels = reformat(train_dataset, train_labels)
valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
test_dataset, test_labels = reformat(test_dataset, test_labels)
print('Training set', train_dataset.shape, train_labels.shape)
print('Validation set', valid_dataset.shape, valid_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)
def accuracy(predictions, labels):
return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1)) / predictions.shape[0])Training set (200000, 28, 28, 1) (200000, 10) Validation set (10000, 28, 28, 1) (10000, 10) Test set (10000, 28, 28, 1) (10000, 10)
1.簡單卷積神經網路
tensorflow提供了卷積函式nn.conv2d。
我們來看一下這個函式的文件，學習下如何使用。
Signature: tf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=None, data_format=None, name=None)Docstring:Computes a 2-D convolution given 4-Dinput and filter tensors.
Given an input tensor of shape [batch, in_height, in_width, in_channels]and a filter / kernel tensor of shape[filter_height, filter_width, in_channels, out_channels], this opperforms the following:
Flattens the filter to a 2-D matrix with shape[filter_height * filter_width * in_channels, output_channels].
Extracts image patches from the input tensor to form avirtual tensor of shape [batch, out_height, out_width, filter_height * filter_width * in_channels].
For each patch, right-multiplies the filter matrix and the image patchvector.
In detail, with the default NHWC format,
output[b, i, j, k] =
sum_{di, dj, q} input[b, strides[1] * i + di, strides[2] * j + dj, q] *
filter[di, dj, q, k]
Must have strides[0] = strides[3] = 1. For the most common case of the samehorizontal and vertices strides,strides = [1, stride, stride, 1].
Args:
input:
A Tensor. Must be one of the following types:half, float32, float64.
filter: A Tensor. Must have the same type asinput.
strides: A list of ints. 1-D of length 4. The stride of the sliding window for each dimension ofinput. Must be in the same order as the dimension specified with format.
padding: A string from:"SAME", "VALID". The type of padding algorithm to use.
use_cudnn_on_gpu: An optionalbool. Defaults to True.
data_format: An optional string from: "NHWC", "NCHW". Defaults to "NHWC". Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: [batch, in_height, in_width, in_channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, in_channels, in_height, in_width]. name: A name for the operation (optional).
Returns: A Tensor. Has the same type asinput.Type: function輸入input就是我們的一組圖片資料，這裡是4維的。
filter：過濾器，也叫權重矩陣，要與input的型別一樣。
stride：步長，含有四個整數的列表，一般第一個和第四個都應該為1，中間兩個分別代表畫素掃描的間隔。
padding：輸入分割的方式，valid和same
最為重要的使理解conv2d對我們的輸入引數都做了什麼變化，這樣我們才能知道如何設定好輸入引數，滿足conv2d的需要。
在conv2d中，
假設inpute的四個維度是[batch, in_height, in_width, in_channels]，
filter的四個維度是[filter_height, filter_width, in_channels, out_channels]。
進入conv2d函式後，filter被拉伸為2維陣列.
新的二維陣列A的維度是[filter_height＊filter_width＊in_channels, out_channels].
其次，input陣列依然為4維陣列，但是維度發生了變化。
input產生新的4維陣列B的維度是[batch, out_height, out_width, filter_height * filter_width * in_channels]。
最後進行乘法B＊A。
所以在設定filter時應注意filter的維度以及input的維度的設定，否則conv2d無法進行運算。
領會了其中的緣由就可以設定好自己的卷積神經網路。
batch_size = 128
patch_size = 5 # padding image pixels by 5*5
depth = 16 # depth
num_hidden = 64 # num of node in hidden layer
graph = tf.Graph()
with graph.as_default():
# Input data.
tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size, image_size, num_channels)) # num_channels=1 grayscale
tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)
# Variables.
layer1_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, num_channels, depth], stddev=0.1))
layer1_biases = tf.Variable(tf.zeros([depth]))
layer2_weights = tf.Variable(tf.truncated_normal( [patch_size, patch_size, depth, depth], stddev=0.1))
layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))
layer3_weights = tf.Variable(tf.truncated_normal([image_size // 4 * image_size // 4 * depth, num_hidden], stddev=0.1))
layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
layer4_weights = tf.Variable(tf.truncated_normal([num_hidden, num_labels], stddev=0.1))
layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))
# Model.
def model(data):
# data (batch, 28, 28, 1)
# weights reshaped to (patch_size*patch_size*num_channels, depth)
# data reshaped to (batch, 14, 14, patch_size*patch_size*num_channels)
# conv shape (batch, 14, 14, depth)
conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME') # convolution
hidden = tf.nn.relu(conv + layer1_biases)
# weights shape (patch_size, patch_size, depth, depth)
# weights reshaped into (patch_size*patch_size* depth, depth)
# hidden reshaped into (batch, 7, 7, patch_size*patch_size* depth)
# conv shape (batch, 7, 7, depth)
conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME') # convolution
hidden = tf.nn.relu(conv + layer2_biases)
# hidden shape (batch, 7, 7, depth)
shape = hidden.get_shape().as_list()
reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]])
# reshape (batch, 7*7*depth)
# weights shape( 28//4 * 28//4*depth, num_hidden)
# hidden shape(batch, num_hidden)
hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
# return tensor (batch, num_labels)
return tf.matmul(hidden, layer4_weights) + layer4_biases
# Training computation.
logits = model(tf_train_dataset)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))
# Optimizer.
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(loss)
# Predictions for the training, validation, and test data.
train_prediction = tf.nn.softmax(logits)
valid_prediction = tf.nn.softmax(model(tf_valid_dataset))
test_prediction = tf.nn.softmax(model(tf_test_dataset))num_steps = 1001
with tf.Session(graph=graph) as session:
tf.initialize_all_variables().run()
print('Initialized')
for step in range(num_steps):
offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
batch_data = train_dataset[offset:(offset + batch_size), :, :, :]
batch_labels = train_labels[offset:(offset + batch_size), :]
feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
_, l, predictions = session.run(
[optimizer, loss, train_prediction], feed_dict=feed_dict)
if (step % 50 == 0):
print('Minibatch loss at step %d: %f' % (step, l))
print('Minibatch accuracy: %.1f%%' % accuracy(predictions, batch_labels))
print('Validation accuracy: %.1f%%' % accuracy(
valid_prediction.eval(), valid_labels))
print('Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels))2. 池化 pooling
2. 池化 poolingpooling會充分利用輸入的資訊，減少資訊流失，但同樣增加了計算量。
tensorflow提供了max pooling 以及 average pooling函式，計算基本類似，只不過將求最大值改為了求平均值。
nn.max_pool函式的文件如下：
Signature: tf.nn.max_pool(value, ksize, strides, padding, data_format='NHWC', name=None)Docstring:Performs the max pooling on the input.
Args: value: A 4-D Tensor with shape[batch, height, width, channels] and type tf.float32.
ksize: A list of ints that has length >= 4. The size of the window for each dimension of the input tensor.
strides: A list of ints that has length >= 4. The stride of the sliding window for each dimension of the input tensor.
padding: A string, either 'VALID' or'SAME'. The padding algorithm. See the comment here
data_format: A string. 'NHWC' and 'NCHW' are supported.
name: Optional name for the operation.
Returns: A Tensor with type tf.float32. The max pooled output tensor. Type: functionstride與conv2d中的定義一樣，理解為掃描步長。它的存在，定義了，max_pool函式返回的tensor的維度。
在寫卷積神經網路時，最好先定義好模型函式(如上文的model函式)，這樣再在前面補充權重矩陣的維度。沒有寫普通神經網路那麼簡潔(定義權重矩陣的時候,模型就定好了)。
# 87.5%
batch_size = 128
patch_size = 5 # padding image pixels by 5*5
depth = 16 # depth
num_hidden = 64 # num of node in hidden layer
graph = tf.Graph()
with graph.as_default():
# Input data.
tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size, image_size, num_channels)) # num_channels=1 grayscale
tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)
# Variables.
layer1_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, num_channels, depth], stddev=0.1))
layer1_biases = tf.Variable(tf.zeros([depth]))
layer2_weights = tf.Variable(tf.truncated_normal( [patch_size, patch_size, depth, depth], stddev=0.1))
layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))
layer3_weights = tf.Variable(tf.truncated_normal([28//7 * 28//7 * depth, num_hidden], stddev=0.1))
layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
layer4_weights = tf.Variable(tf.truncated_normal([num_hidden, num_labels], stddev=0.1))
layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))
# Model.
def model(data):
# data (batch, 28, 28, 1)
# weights reshaped to (patch_size*patch_size*num_channels, depth)
# data reshaped to (batch, 14, 14, patch_size*patch_size*num_channels)
# conv shape (batch, 14, 14, depth)
conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME') # convolution
hidden = tf.nn.relu(conv + layer1_biases)
# weights shape (patch_size, patch_size, depth, depth)
# weights reshaped into (patch_size*patch_size* depth, depth)
# hidden reshaped into (batch, 7, 7, patch_size*patch_size* depth)
# conv shape (batch, 7, 7, depth)
conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME') # convolution
# conv shape (batch, 7, 7, depth)
#print('conv1 shape', conv.get_shape().as_list())
conv = tf.nn.max_pool(conv, [1,2,2,1], [1,2,2,1], padding='SAME') # strides change dimensions
#print('conv2 shape', conv.get_shape().as_list())
hidden = tf.nn.relu(conv + layer2_biases)
# hidden shape (batch, 4, 4, depth)
shape = hidden.get_shape().as_list()
reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]])
# reshape (batch,4*4*depth)
# weights shape( 4 * 4*depth, num_hidden)
# hidden shape(batch, num_hidden)
#print('reshape shape', reshape.get_shape().as_list())
#print('layer3_weights', layer3_weights.get_shape().as_list())
hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
# return tensor (batch, num_labels)
return tf.matmul(hidden, layer4_weights) + layer4_biases
# Training computation.
logits = model(tf_train_dataset)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))
# Optimizer.
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(loss)
# Predictions for the training, validation, and test data.
train_prediction = tf.nn.softmax(logits)
valid_prediction = tf.nn.softmax(model(tf_valid_dataset))
test_prediction = tf.nn.softmax(model(tf_test_dataset))
num_steps = 1001
with tf.Session(graph=graph) as session:
tf.initialize_all_variables().run()
print('Initialized')
for step in range(num_steps):
offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
batch_data = train_dataset[offset:(offset + batch_size), :, :, :]
batch_labels = train_labels[offset:(offset + batch_size), :]
feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
_, l, predictions = session.run(
[optimizer, loss, train_prediction], feed_dict=feed_dict)
if (step % 50 == 0):
print('Minibatch loss at step %d: %f' % (step, l))
print('Minibatch accuracy: %.1f%%' % accuracy(predictions, batch_labels))
print('Validation accuracy: %.1f%%' % accuracy(valid_prediction.eval(), valid_labels))
print('Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels))這還沒有普通的神經網路好。
所以神經網路的架構很重要。
如果沒有好的架構，再深的神經網路也許沒有簡單的邏輯迴歸的效果好，還浪費了大量資源，真是吃力不討好。
這裡可以用學習速率衰減來提高一下測試集的準確度。準確度在，92.6%左右。
# only add learning rate decay
# 92.6%
batch_size = 128
patch_size = 5 # padding image pixels by 5*5
depth = 16 # depth
num_hidden = 64 # num of node in hidden layer
graph = tf.Graph()
with graph.as_default():
# Input data.
tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size, image_size, num_channels)) # num_channels=1 grayscale
tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)
# Variables.
global_step = tf.Variable(0)
learning_rate = tf.train.exponential_decay(0.1, global_step, 300, 0.7)
layer1_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, num_channels, depth], stddev=0.1))
layer1_biases = tf.Variable(tf.zeros([depth]))
layer2_weights = tf.Variable(tf.truncated_normal( [patch_size, patch_size, depth, depth], stddev=0.1))
layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))
layer3_weights = tf.Variable(tf.truncated_normal([28//7 * 28//7 * depth, num_hidden], stddev=0.1))
layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
layer4_weights = tf.Variable(tf.truncated_normal([num_hidden, num_labels], stddev=0.1))
layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))
# Model.
def model(data):
# data (batch, 28, 28, 1)
# weights reshaped to (patch_size*patch_size*num_channels, depth)
# data reshaped to (batch, 14, 14, patch_size*patch_size*num_channels)
# conv shape (batch, 14, 14, depth)
conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME') # convolution
hidden = tf.nn.relu(conv + layer1_biases)
# weights shape (patch_size, patch_size, depth, depth)
# weights reshaped into (patch_size*patch_size* depth, depth)
# hidden reshaped into (batch, 7, 7, patch_size*patch_size* depth)
# conv shape (batch, 7, 7, depth)
conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME') # convolution
# conv shape (batch, 7, 7, depth)
#print('conv1 shape', conv.get_shape().as_list())
conv = tf.nn.max_pool(conv, [1,2,2,1], [1,2,2,1], padding='SAME') # strides change dimensions
#print('conv2 shape', conv.get_shape().as_list())
hidden = tf.nn.relu(conv + layer2_biases)
# hidden shape (batch, 4, 4, depth)
shape = hidden.get_shape().as_list()
reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]])
# reshape (batch,4*4*depth)
# weights shape( 4 * 4*depth, num_hidden)
# hidden shape(batch, num_hidden)
# print('reshape shape', reshape.get_shape().as_list())
# print('layer3_weights', layer3_weights.get_shape().as_list())
hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
# return tensor (batch, num_labels)
return tf.matmul(hidden, layer4_weights) + layer4_biases
# Training computation.
logits = model(tf_train_dataset)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))
# Optimizer.
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)
# Predictions for the training, validation, and test data.
train_prediction = tf.nn.softmax(logits)
valid_prediction = tf.nn.softmax(model(tf_valid_dataset))
test_prediction = tf.nn.softmax(model(tf_test_dataset))
num_steps = 20001
with tf.Session(graph=graph) as session:
tf.initialize_all_variables().run()
print('Initialized')
for step in range(num_steps):
offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
batch_data = train_dataset[offset:(offset + batch_size), :, :, :]
batch_labels = train_labels[offset:(offset + batch_size), :]
feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
_, l, predictions = session.run(
[optimizer, loss, train_prediction], feed_dict=feed_dict)
if (step % 300 == 0):
print('current Learning rate', learning_rate.eval())
print('Minibatch loss at step %d: %f' % (step, l))
print('Minibatch accuracy: %.1f%%' % accuracy(predictions, batch_labels))
print('Validation accuracy: %.1f%%' % accuracy(valid_prediction.eval(), valid_labels))
print('Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels))相關文章
- TensorFlow實戰卷積神經網路之LeNet卷積神經網路
- TensorFlow上實現卷積神經網路CNN卷積神經網路CNN
- 神經網路 | 基於MATLAB 深度學習工具實現簡單的數字分類問題（卷積神經網路）神經網路Matlab深度學習卷積
- 深度學習三：卷積神經網路深度學習卷積神經網路
- 【Tensorflow_DL_Note6】Tensorflow實現卷積神經網路(1)卷積神經網路
- 【Tensorflow_DL_Note7】Tensorflow實現卷積神經網路(2)卷積神經網路
- 【深度學習篇】--神經網路中的卷積神經網路深度學習神經網路卷積
- 深度學習——LeNet卷積神經網路初探深度學習卷積神經網路
- 深度學習筆記------卷積神經網路深度學習筆記卷積神經網路
- 深度學習卷積神經網路筆記深度學習卷積神經網路筆記
- 深度學習基礎-基於Numpy的卷積神經網路（CNN）實現深度學習卷積神經網路CNN
- 深度學習經典卷積神經網路之AlexNet深度學習卷積神經網路
- 【卷積神經網路學習】(4)機器學習卷積神經網路機器學習
- TensorFlow 一步一步實現卷積神經網路卷積神經網路
- 【深度學習原理第4篇】卷積神經網路詳解（CNN）深度學習卷積神經網路CNN
- Tensorflow-卷積神經網路CNN卷積神經網路CNN
- 基於卷積神經網路和tensorflow實現的人臉識別卷積神經網路
- 深度學習革命的開端：卷積神經網路深度學習卷積神經網路
- 深度剖析卷積神經網路卷積神經網路
- Keras上實現卷積神經網路CNNKeras卷積神經網路CNN
- 吳恩達深度學習：簡單卷積網路吳恩達深度學習卷積
- 圖卷積神經網路(GCN)理解與tensorflow2.0程式碼實現卷積神經網路GC
- 深度學習-卷積神經網路-演算法比較深度學習卷積神經網路演算法
- [譯] 淺析深度學習神經網路的卷積層深度學習神經網路卷積
- 卷積神經網路的原理及Python實現卷積神經網路Python
- 《卷積神經網路的Python實現》筆記卷積神經網路Python筆記
- 深度學習之卷積神經網路(Convolutional Neural Networks, CNN)(二)深度學習卷積神經網路CNN
- 深度學習入門筆記（十八）：卷積神經網路（一）深度學習筆記卷積神經網路
- 卷積神經網路CNN-學習1卷積神經網路CNN
- 卷積神經網路學習筆記——SENet卷積神經網路筆記SENet
- [譯] 使用 Python 和 Keras 實現卷積神經網路PythonKeras卷積神經網路
- 【python實現卷積神經網路】開始訓練Python卷積神經網路
- 圖卷積神經網路分類的pytorch實現卷積神經網路PyTorch
- 卷積神經網路卷積神經網路
- TensorFlow 卷積神經網路之貓狗識別卷積神經網路
- Ng深度學習筆記——卷積神經網路基礎深度學習筆記卷積神經網路
- CNN-卷積神經網路簡單入門（2）CNN卷積神經網路
- 卷積神經網路學習筆記——Siamese networks（孿生神經網路）卷積神經網路筆記
- 8-深度學習之神經網路核心原理與演算法-卷積神經網路深度學習神經網路演算法卷積