C231n-KNN-assignment1-完全程式碼及註釋

IAMoldpan發表於2017-10-26

以下內容為C231n-assignment1-KNN的程式碼
作業網址：http://cs231n.github.io/assignments2017/assignment1/

Q1: k-Nearest Neighbor classifier (20 points)
The IPython Notebook knn.ipynb will walk you through implementing the kNN classifier.

以下是notobook中的程式碼以及結果（其中KNN演算法程式碼在最下方）

import sys 
print(sys.version)

# k-Nearest Neighbor (kNN) exercise

*Complete and hand in this completed worksheet (including its outputs and any supporting code outside of the worksheet) with your assignment submission. For more details see the [assignments page](http://vision.stanford.edu/teaching/cs231n/assignments.html) on the course website.*

The kNN classifier consists of two stages:

- During training, the classifier takes the training data and simply remembers it
- 在訓練中，分類器分析訓練資料並簡單記住這些資料
- During testing, kNN classifies every test image by comparing to all training images and transfering the labels of the k most similar training examples
- 在訓練中，KNN分類器將每個測試影像與訓練影像進行比較然後傳遞出k個最近距離的訓練集的標記
- The value of k is cross-validated
- K值通過交叉驗證得到


In this exercise you will implement these steps and understand the basic Image Classification pipeline, cross-validation, and gain proficiency in writing efficient, vectorized code.

# 初始化

import random
import numpy as np
from cs231n.data_utils import load_CIFAR10
import matplotlib.pyplot as plt

from __future__ import print_function

# 下面這個%得作用時使影像直接產生在這個頁面中而不是產生到一個新的視窗
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# Some more magic so that the notebook will reload external python modules;
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

# Load the raw CIFAR-10 data.
cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

# As a sanity check, we print out the size of the training and test data.
print('Training data shape: ', X_train.shape)
print('Training labels shape: ', y_train.shape)
print('Test data shape: ', X_test.shape)
print('Test labels shape: ', y_test.shape)

Training data shape: (50000, 32, 32, 3)
Training labels shape: (50000,)
Test data shape: (10000, 32, 32, 3)
Test labels shape: (10000,)

# Visualize some examples from the dataset.
# We show a few examples of training images from each class.
classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
num_classes = len(classes)
samples_per_class = 7
for y, cls in enumerate(classes):
    idxs = np.flatnonzero(y_train == y)
    idxs = np.random.choice(idxs, samples_per_class, replace=False)
    for i, idx in enumerate(idxs):
        plt_idx = i * num_classes + y + 1
        plt.subplot(samples_per_class, num_classes, plt_idx)
        plt.imshow(X_train[idx].astype('uint8'))
        plt.axis('off')
        if i == 0:
            plt.title(cls)
plt.show()

這裡寫圖片描述

# 取子樣本（從50000中去5000個出來），使程式執行的更快些
num_training = 5000
mask = list(range(num_training))
X_train = X_train[mask]
y_train = y_train[mask]

num_test = 500
mask = list(range(num_test))
X_test = X_test[mask]
y_test = y_test[mask]

# 將影像中的畫素變成一行資料
X_train = np.reshape(X_train, (X_train.shape[0], -1))
X_test = np.reshape(X_test, (X_test.shape[0], -1))
print(X_train.shape, X_test.shape)

(5000, 3072) (500, 3072)

from cs231n.classifiers import KNearestNeighbor  
#該程式碼在最下方

# Create a kNN classifier instance. 
# Remember that training a kNN classifier is a noop: 
# 執行這段程式碼需要較長的時間
# the Classifier simply remembers the data and does no further processing 
classifier = KNearestNeighbor()
classifier.train(X_train, y_train)

# Open cs231n/classifiers/k_nearest_neighbor.py and implement
# compute_distances_two_loops.

# Test your implementation:
dists = classifier.compute_distances_two_loops(X_test)
print(dists.shape)

(500, 5000)

# We can visualize the distance matrix: each row is a single test example and
# its distances to training examples
plt.imshow(dists, interpolation='none')
plt.show()

這裡寫圖片描述

# Now implement the function predict_labels and run the code below:
# We use k = 1 (which is Nearest Neighbor).
y_test_pred = classifier.predict_labels(dists, k=1)

# Compute and print the fraction of correctly predicted examples
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

Got 137 / 500 correct => accuracy: 0.274000

You should expect to see approximately 27% accuracy. Now lets try out a larger k, say k = 5:

y_test_pred = classifier.predict_labels(dists, k=5)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

Got 144 / 500 correct => accuracy: 0.288000

You should expect to see a slightly better performance than with k = 1.

# Now lets speed up distance matrix computation by using partial vectorization
# with one loop. Implement the function compute_distances_one_loop and run the
# code below:
dists_one = classifier.compute_distances_one_loop(X_test)

# To ensure that our vectorized implementation is correct, we make sure that it
# agrees with the naive implementation. There are many ways to decide whether
# two matrices are similar; one of the simplest is the Frobenius norm. In case
# you haven't seen it before, the Frobenius norm of two matrices is the square
# root of the squared sum of differences of all elements; in other words, reshape
# the matrices into vectors and compute the Euclidean distance between them.
difference = np.linalg.norm(dists - dists_one, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
    print('Good! The distance matrices are the same')
else:
    print('Uh-oh! The distance matrices are different')

Difference was: 0.000000
Good! The distance matrices are the same

# Now implement the fully vectorized version inside compute_distances_no_loops
# and run the code
dists_two = classifier.compute_distances_no_loops(X_test)

# check that the distance matrix agrees with the one we computed before:
difference = np.linalg.norm(dists - dists_two, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
    print('Good! The distance matrices are the same')
else:
    print('Uh-oh! The distance matrices are different')

Difference was: 0.000000
Good! The distance matrices are the same

# Let's compare how fast the implementations are
def time_function(f, *args):
    """
    Call a function f with args and return the time (in seconds) that it took to execute.
    """
    import time
    tic = time.time()
    f(*args)
    toc = time.time()
    return toc - tic

two_loop_time = time_function(classifier.compute_distances_two_loops, X_test)
print('Two loop version took %f seconds' % two_loop_time)

one_loop_time = time_function(classifier.compute_distances_one_loop, X_test)
print('One loop version took %f seconds' % one_loop_time)

no_loop_time = time_function(classifier.compute_distances_no_loops, X_test)
print('No loop version took %f seconds' % no_loop_time)

# you should see significantly faster performance with the fully vectorized implementation

Two loop version took 56.785069 seconds
One loop version took 136.449761 seconds
No loop version took 0.591535 seconds
(矩陣乘法真的很快！)

### Cross-validation
### 交叉驗證

We have implemented the k-Nearest Neighbor classifier but we set the value k = 5 arbitrarily. We will now determine the best value of this hyperparameter with cross-validation.
我們已經完成了k近鄰分類但是我們只測試了k=5的情況，我們將要通過交叉驗證來得出最好的k係數

num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]

X_train_folds = []
y_train_folds = []
################################################################################
# TODO:
# Split up the training data into folds. After splitting, X_train_folds and    #
# y_train_folds should each be lists of length num_folds, where                #
# y_train_folds[i] is the label vector for the points in X_train_folds[i].     #
# Hint: Look up the numpy array_split function.  
# 將訓練資料分成幾份，在分完後X_train_folds和y_train_folds應該為長度為num_folds
# 的lists，其中y_train_folds[i]是X_train_folds[i]的標誌向量
# 提示：看一下numpy中的array_split函式
################################################################################
X_train_folds = np.array_split(X_train, num_folds, axis=0) # list
y_train_folds = np.array_split(y_train, num_folds, axis=0) # list
################################################################################
#                                 END OF YOUR CODE                             #
################################################################################

# A dictionary holding the accuracies for different values of k that we find
# when running cross-validation. After running cross-validation,
# k_to_accuracies[k] should be a list of length num_folds giving the different
# accuracy values that we found when using that value of k.
# 一個包含不同k值對應我們進行交叉驗證時的不同準確度的字典。在進行交叉驗證後，k_to_accuracies[k] 
# 應該為一個長度為num_folds的list，其中每個k在我們測試中得到不同的準確度
k_to_accuracies = {}


################################################################################
# TODO:                                                                        #
# Perform k-fold cross validation to find the best value of k. For each        #
# possible value of k, run the k-nearest-neighbor algorithm num_folds times,   #
# where in each case you use all but one of the folds as training data and the #
# last fold as a validation set. Store the accuracies for all fold and all     #
# values of k in the k_to_accuracies dictionary.  #
# 進行k折交叉驗證，找到最好結果的k值。對於每個可能的k值執行k近鄰演算法num_folds
# 次。其中，一共num_folds疊中，用除了其中一疊的所有疊當做訓練集進行訓練，然後用
# 剩餘的一疊當做驗證集。把不用k的結果存在k_to_accuracies dictionary裡面
################################################################################
for i in range(num_folds):
    # 訓練 / 驗證集 比例 (80% 20%)
    X_train_batch = np.concatenate(X_train_folds[1:num_folds])   
    y_train_batch = np.concatenate(y_train_folds[1:num_folds])
    X_valid_batch = X_train_folds[0]   
    y_valid_batch = y_train_folds[0]

    # 交換資料 (for next iteration)  
    if i < num_folds - 1:
        tmp = X_train_folds[0]                   
        X_train_folds[0] = X_train_folds[i+1]
        X_train_folds[i+1] = tmp
        tmp = y_train_folds[0]
        y_train_folds[0] = y_train_folds[i+1]
        y_train_folds[i+1] = tmp


    # 訓練模型
    model = KNearestNeighbor()
    model.train(X_train_batch, y_train_batch)
    dists = model.compute_distances_no_loops(X_valid_batch)

    # 計算每個k的準確度
    for k in k_choices:
        y_valid_pred = model.predict_labels(dists, k=k)

        # 計算驗證準確度
        num_correct = np.sum(y_valid_pred == y_valid_batch)
        accuracy = float(num_correct) / y_valid_batch.shape[0]

        # 將準確度值放入字典中
        if i == 0:
            k_to_accuracies[k] = [] 
        k_to_accuracies[k].append(accuracy)
################################################################################
#                                 END OF YOUR CODE                             #
################################################################################

# 列印出計算好的準確度
for k in sorted(k_to_accuracies):
    for accuracy in k_to_accuracies[k]:
        print('k = %d, accuracy = %f' % (k, accuracy))

296000
k = 8, accuracy = 0.278000
k = 8, accuracy = 0.289000
k = 8, accuracy = 0.285000
k = 10, accuracy = 0.273000
k = 10, accuracy = 0.296000
k = 10, accuracy = 0.277000
k = 10, accuracy = 0.293000
k = 10, accuracy = 0.285000
k = 12, accuracy = 0.269000
k = 12, accuracy = 0.304000
k = 12, accuracy = 0.286000
k = 12, accuracy = 0.283000
k = 12, accuracy = 0.278000
k = 15, accuracy = 0.259000
k = 15, accuracy = 0.308000
k = 15, accuracy = 0.287000
k = 15, accuracy = 0.287000
k = 15, accuracy = 0.276000
k = 20, accuracy = 0.265000
k = 20, accuracy = 0.287000
k = 20, accuracy = 0.288000
k = 20, accuracy = 0.286000
k = 20, accuracy = 0.283000
k = 50, accuracy = 0.273000
k = 50, accuracy = 0.287000
k = 50, accuracy = 0.279000
k = 50, accuracy = 0.269000
k = 50, accuracy = 0.272000
k = 100, accuracy = 0.258000
k = 100, accuracy = 0.273000
k = 100, accuracy = 0.264000
k = 100, accuracy = 0.260000
k = 100, accuracy = 0.266000

# plot the raw observations
for k in k_choices:
    accuracies = k_to_accuracies[k]
    plt.scatter([k] * len(accuracies), accuracies)

# plot the trend line with error bars that correspond to standard deviation
accuracies_mean = np.array([np.mean(v) for k,v in sorted(k_to_accuracies.items())])
accuracies_std = np.array([np.std(v) for k,v in sorted(k_to_accuracies.items())])
plt.errorbar(k_choices, accuracies_mean, yerr=accuracies_std)
plt.title('Cross-validation on k')
plt.xlabel('k')
plt.ylabel('Cross-validation accuracy')
plt.show()

這裡寫圖片描述

# Based on the cross-validation results above, choose the best value for k,   
# retrain the classifier using all the training data, and test it on the test
# data. You should be able to get above 28% accuracy on the test data.
# 在交叉驗證的結果中，選擇最合適的k，重新操作一遍，你能得到大概28%的準確率
best_k = 10

classifier = KNearestNeighbor()
classifier.train(X_train, y_train)
y_test_pred = classifier.predict(X_test, k=best_k)

# 計算並展示準確率
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

Got 144 / 500 correct => accuracy: 0.288000

以下是KNN程式碼部分（其中KNN-no-loop程式碼講解在這裡）

import numpy as np

# 在python3.3後 將xrange和range合併，呼叫range相當於呼叫xrange
xrange = range

class KNearestNeighbor(object):
    """ a kNN classifier with L2 distance """

    def __init__(self):
        pass

    def train(self, X, y):
        """
        Train the classifier. For k-nearest neighbors this is just
        memorizing the training data.

        Inputs:
        - X: A numpy array of shape (num_train, D) containing the training data
        consisting of num_train samples each of dimension D.
        - y: A numpy array of shape (N,) containing the training labels, where
           y[i] is the label for X[i].
           :type self: object
        """

        self.X_train = X
        self.y_train = y

    def predict(self, X, k=1, num_loops=0):
        """
        Predict labels for test data using this classifier.

        Inputs:
        - X: A numpy array of shape (num_test, D) containing test data consisting
             of num_test samples each of dimension D.
        - k: The number of nearest neighbors that vote for the predicted labels.
        - num_loops: Determines which implementation to use to compute distances
          between training points and testing points.

        Returns:
        - y: A numpy array of shape (num_test,) containing predicted labels for the
          test data, where y[i] is the predicted label for the test point X[i].
        """
        if num_loops == 0:
            dists = self.compute_distances_no_loops(X)
        elif num_loops == 1:
            dists = self.compute_distances_one_loop(X)
        elif num_loops == 2:
            dists = self.compute_distances_two_loops(X)
        else:
            raise ValueError('Invalid value %d for num_loops' % num_loops)

        return self.predict_labels(dists, k=k)

    def compute_distances_two_loops(self, X):
        """
        Compute the distance between each test point in X and each training point
        in self.X_train using a nested loop over both the training data and the
        test data.

        Inputs:
        - X: A numpy array of shape (num_test, D) containing test data.

        Returns:
        - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
          is the Euclidean distance between the ith test point and the jth training
          point.
        """
        num_test = X.shape[0]  # 測試集數量
        num_train = self.X_train.shape[0]  # 訓練集數量
        sum = 0
        dists = np.zeros((num_test, num_train))  # 創造零矩陣，測試集數量為行數，訓練集數量為列數 500*5000
        for i in xrange(num_test):  # num_test 數量為500 num_train 數量為5000
            for j in xrange(num_train):
                #####################################################################
                # TODO:                                                             #
                # Compute the l2 distance between the ith test point and the jth    #
                # training point, and store the result in dists[i, j]. You should   #
                # not use a loop over dimension.                                    #
                #####################################################################
                # np.dot()對兩個一維向量進行運算，會自動轉換成1×n 與 n×1的形式從而得到一個數
                dists[i, j] = np.sqrt(np.dot(X[i] - self.X_train[j], X[i] - self.X_train[j]))
            #####################################################################
            #                       END OF YOUR CODE                            #
            #####################################################################
        return dists

    def compute_distances_one_loop(self, X):
        """
    Compute the distance between each test point in X and each training point
    in self.X_train using a single loop over the test data.

    Input / Output: Same as compute_distances_two_loops
    """
        num_test = X.shape[0]
        num_train = self.X_train.shape[0]
        dists = np.zeros((num_test, num_train))
        for i in xrange(num_test):
            dists[i, :] = np.sqrt(np.sum((self.X_train - X[i, :]) ** 2, axis=1))

        #######################################################################
        # TODO:                                                               #
        # Compute the l2 distance between the ith test point and all training #
        # points, and store the result in dists[i, :].                        #
        #######################################################################

        #######################################################################
        #                         END OF YOUR CODE                            #
        #######################################################################
        return dists

    def compute_distances_no_loops(self, X):
        """
        Compute the distance between each test point in X and each training point
        in self.X_train using no explicit loops.
        不適用任何迴圈去計算測試集和訓練集中的距離

        Input / Output: Same as compute_distances_two_loops
        """
        num_test = X.shape[0]
        num_train = self.X_train.shape[0]
        dists = np.zeros((num_test, num_train))

        #########################################################################
        # TODO:                                                                 #
        # Compute the l2 distance between all test points and all training      #
        # points without using any explicit loops, and store the result in      #
        # dists.                                                                #
        #                                                                       #
        # You should implement this function using only basic array operations; #
        # in particular you should not use functions from scipy.                #
        #                                                                       #
        # HINT: Try to formulate the l2 distance using matrix multiplication    #
        #       and two broadcast sums.                                         #
        #########################################################################
        test_sum = np.sum(np.square(X), axis=1)  # num_test x 1
        train_sum = np.sum(np.square(self.X_train), axis=1)  # num_train x 1
        inner_product = np.dot(X, self.X_train.T)  # num_test x num_train
        dists = np.sqrt(-2 * inner_product + test_sum.reshape(-1, 1) + train_sum)  # broadcast
         (http://blog.csdn.net/iamoldpan/article/details/78359195)
        # dists = np.sqrt((-2 * np.dot(X, self.X_train.T)) + np.sum(X ** 2, axis=1, keepdims=True)
        # + np.sum(self.X_train ** 2,axis=1))
        #########################################################################
        #                         END OF YOUR CODE                              #
        #########################################################################
        return dists

    def predict_labels(self, dists, k=1):

        """
        Given a matrix of distances between test points and training points,
        predict a label for each test point.

        Inputs:
        - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
          gives the distance betwen the ith test point and the jth training point.

        Returns:
        - y: A numpy array of shape (num_test,) containing predicted labels for the
          test data, where y[i] is the predicted label for the test point X[i].
        """

        num_test = dists.shape[0]  # 500為測試數量
        y_pred = np.zeros(num_test)  #
        for i in xrange(num_test):
            # A list of length k storing the labels of the k nearest neighbors to
            # the ith test point.
            closest_y = []
            #########################################################################
            # TODO:                                                                 #
            # Use the distance matrix to find the k nearest neighbors of the ith    #
            # testing point, and use self.y_train to find the labels of these       #
            # neighbors. Store these labels in closest_y.                           #
            # Hint: Look up the function numpy.argsort.                             #
            #########################################################################
            sorted_dic = np.argsort(dists[i])
            closest_y = self.y_train[sorted_dic[:k]]
            #########################################################################
            # TODO:                                                                 #
            # Now that you have found the labels of the k nearest neighbors, you    #
            # need to find the most common label in the list closest_y of labels.   #
            # Store this label in y_pred[i]. Break ties by choosing the smaller     #
            # label.                                                                #
            #########################################################################
            count = {}
            label = 0
            num = 0
            for j in range(k):
                count[closest_y[j]] = count.get(closest_y[j], 0) + 1
                if num < count[closest_y[j]]:
                    label = closest_y[j]
                    num = count[closest_y[j]]
            y_pred[i] = label

        #########################################################################
        #                           END OF YOUR CODE                            #
        #########################################################################

        return y_pred

C231n-SVM-assignment1-完全程式碼及註釋
2017-10-31
原始碼完全註釋：socket select
2018-10-20
原始碼
Nginx原始碼完全註釋（6）core/murmurhash
2012-09-09
Nginx原始碼
CSS程式碼註釋
2018-07-28
CSS
php程式碼註釋
2017-06-04
PHP
Nginx原始碼完全註釋（8）ngx_errno.c
2012-09-29
Nginx原始碼
Nginx原始碼完全註釋（5）core/ngx_cpuinfo.c
2012-09-09
Nginx原始碼UI
有趣的程式碼註釋
2019-05-27
請停止程式碼註釋
2019-06-04
javascript如何註釋程式碼
2017-04-14
JavaScript
程式碼才是最好的註釋
2021-09-09
HTML 程式碼註釋規範
2019-01-07
HTML
註釋程式碼的13技巧
2014-06-16
Java程式碼註釋規範
2016-03-02
Java
字串及註釋
2016-07-16
字串
Nginx原始碼完全註釋（9）nginx.c: ngx_get_options
2012-09-29
Nginx原始碼
iOS 註釋方法大全程式碼塊加快捷鍵註釋
2019-03-07
iOS
程式設計師是否有義務做好程式碼的註釋？你做好程式碼註釋了嗎？
2019-01-21
程式設計師
Pycharm 程式碼註釋風格模板
2018-01-21
PyCharm
jvm執行程式碼註釋部分
2024-07-26
JVM行程
Nginx原始碼完全註釋（1）ngx_alloc.h / ngx_alloc.c
2012-07-01
Nginx原始碼
Nginx原始碼完全註釋（7）ngx_palloc.h/ngx_palloc.c
2012-09-28
Nginx原始碼
Nginx原始碼完全註釋（4）ngx_queue.h / ngx_queue.c
2012-09-09
Nginx原始碼
Nginx原始碼完全註釋（3）ngx_list.h / ngx_list.c
2012-09-09
Nginx原始碼
Nginx原始碼完全註釋（2）ngx_array.h / ngx_array.c
2012-09-09
Nginx原始碼
iOS 註釋方法大全程式碼塊加快捷鍵自定義註釋
2017-12-14
iOS
註釋 · 佛祖保佑程式碼永無BUG
2019-03-25
PHP搞笑註釋程式碼-佛祖配美女
2018-01-26
PHP
Oracle PL/SQL程式碼中的註釋
2018-07-10
OracleSQL
谷歌輸入法PinyinIme 程式碼註釋
2013-12-04
谷歌
9個最有趣的程式碼註釋
2014-11-11
使用GhostDoc為程式碼生成註釋文件
2008-06-24
體面編碼之程式碼註釋評論
2018-12-31
Flutter 註解處理及程式碼生成
2019-07-02
Flutter
如何優雅地寫註釋：找到程式碼註釋的黃金平衡點
2024-07-23
vs快速註釋程式碼，vs程式碼行數調出來
2016-06-02
sublime text 多行程式碼註釋快捷鍵
2020-04-04
行程
竟有如此沙雕的程式碼註釋！
2020-06-08

C231n-KNN-assignment1-完全程式碼及註釋

相關文章