ELEC5307 Deep Learning

标签：layers ELEC5307 network Deep should will Learning need your

ELEC5307 Deep Learning

Project #1: Parameters in Neural Networks Due: 23 Oct 2020 11:59PM

1 Objectives

This laboratory aims to introduce the basic techniques in deep neural networks. In this laboratory you will:

• Learn to use PyTorch to load images and train a neural network for classification.
• Understand the functions of convolutional layers, pooling layers, fully connected layersand softmax layer, etc.

• Become familiar with the activation methods, pooling and initialization methods.
• Learn to select proper hyperparameters for better performance.
• Visualize your results and the objective to learn how different parameters contribute tothe final performance.

2 Instructions

2.1 Data description: CIFAR-10

You need to use the CIFAR-10 image dataset. The CIFAR-10 dataset consists of 60000 imagesin 10 classes, with 6000 images per class. There are 50000 training images and 10000 testimages, which are split by the publisher. The details and the downloads of the dataset is inhttps://www.cs.toronto.edu/~kriz/cifar.htmlThe classes include ‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’,and ‘truck’. All the images are manually labelled, and each image only contains one label. Theimages in CIFAR-10 are of size 3x32x32, i.e. 3-channel colour images of 32x32 pixels in size. Insome applications, each image has already been reshaped into a vector with dimension of 3072

(= 3 × 32 × 32).In PyTorch, you can use the function torchvision.datasets.CIFAR10 to automaticallydownload and read the dataset and the function torch.utils.data.DataLoader to load thedata (training and test) into your network.

2.2 Hyperparameters

Hyperparameters are crucial to your success in training a neural network. The weights of neuralnetworks will be modified during learning, and the hyperparameters will contribute to themodification of input images, the number of weights, and the way to update the weights. Hereare some groups of hyperparameters. For more hyperparameters and their usage in PyTorch,please refer to the official documentation website (https://pytorch.org/docs/stable/index.

html) and PyTorch Forum (https://discuss.pytorch.org).Deep Learning1Project #1: Parameters in Neural Networks

2.2.1 Transformation

This will influence the input images. The basic idea to conduct transformation is to add data

(called data augmentation in deep learning). The neural network contains many weights to

modify, which requires many images. However, the provided data are always not enough. By

making transformations, one image could be fed into the networks using different patches or

izes, which could increase the size of training set.These operations are included in torchvision.transforms. Some of the options are:

• resize: The images can be resized to square image or larger/smaller. Almost all thetransformation will need the resize operation.

• crop: You can crop the center (CenterCrop), or center plus the four corners (FiveCrop),or crop at a random location((RandomCrop)).flip: The images may occur from different view, so flip horizontally (RandomHorizontalFlip)or vertically (RandomVerticalFlip) could provide more possibilities of theimages.

• affine: This will modify the image by rotation or translation. It is not suitable to prede-fine some affine function because of the diversity of images, so torchvision only contains(RandomAffine).

• normalization The pixel values of the images are normalized to [0, 1] using this operation.You need to use the global mean and standard deviation of the whole dataset if youtrain the model from scratch. If you would want to fine-tune a model from some modelpretrained on ImageNet, you need to use the values from ImageNet.Please note that the normalization operation is not for data augmentation but for fasterconvergence. The rest four operations mentioned above are used for data augmentation.

2.2.2 Network Structure

These parameters will influence the structure of the neural networks. The most important

indicator is the capacity of the network, which is roughly equal to the number of parameters.

In that case, the deeper and wider the network is, the better potential performance it could

provide. However, you need to make sure the parameters are constrained carefully.

The operations are included in (torch.nn). Some of the options are:

• Depth: Roughly speaking, the deeper network will have the potential to provide betteresults. However, when the network becomes deeper, it will be more difficult to train asthe gradients are more possible to vanish or explode.

• Activation function: The most commonly used are ReLU, Tanh and Sigmoid. Thesefunctions provide non-linearity to the neural network.

• Pooling method: The common choices are max pooling (MaxPool2d) and average pool

ing (AvgPool2d). The size and stride of the pooling layers will change the sizes of feature

aps (i.e. width and height).

• Channel size: The input channel is 3 (for three colour channels R/G/B), and the output

channel should be the number of classes (10 for CIFAR-10). From shallow layers to deep

layers, the channel number always gradually increase (with the width and height numbers

decreasing). The larger the channel number, the more time you will need for both forward

and backward process. In PyTorch, fully connected layers can be basically defined as

Linear(in channel, out channel), 2-D convolutional layers can be basically defined

as Conv2d(in channel, out channel, kernel).

ELEC5307 Deep Learning 2Project #1: Parameters in Neural Networks

• Convolutional parameters: The kernel size, zero padding number and stride will influ

ence your output width and height by Woutput = (Winput−Kernel+2×P adding)/Stride+

1. You can try to use kernel size as 3 × 3, 5 × 5 and 7 × 7. By default, we do not want the

convolutional layers change the width/height of the feature maps, so you can select stride

and zero padding accordingly. For example, if you have kernel size 5 × 5, we always need

the zero padding as 2 and stride as 1.

• Dropout: This layer (Dropout) is a method of regularization, which will randomly set

zeros to some weights in the according layer.

2.2.3 Training Process

The training process is affected by how the data are fed into the network and how the weights are

initialized and updated. The data are fed into the network using torch.utils.data.DataLoader.

Some of the options are:

• shuffle: By default, we usually shuffle the input data in the training and validation partand do not shuffle the input data in test part.

• batch size: The larger batch size can often help you get better results, but it is limited byyour memory size (computer memory or GPU memory). There are also some exceptionthat larger batch sizes will make the performance worse, so you need to be careful inelecting this value.The weights can be initialized from pretrained model or by using torch.nn.init, wh代写 eELEC5307 Deep Learning rethe options includes Xavier, Nomal, Uniform, and Constant, etc.he weights will be updated in backward process according to the objective function, optimization function and learning rate. The objective functions are treated as special layers inyTorch in torch.nn, and the optimization operations can be found in torch.optim, andsome options are listed as below:

• Epochs: One epoch means a period that the network has been trained by seeing everytraining image. After several epochs, your training loss and validation accuracy will notchange much. You need to set a good number of epoch in order to get the best accuracyand avoid overfitting.

• Objective function: The cross-entropy loss (torch.nn.CrossEntropyLoss) is alwaysused in the classification problems. The soft-margin loss (torch.nn.SoftMarginLoss)and least square error (torch.nn.MSELoss) are also included mainly for binaryclassifi-cation problems.

• Update methods: The commonly used methods are Adam(torch.optim.Adam) andSGD(torch.optim.SGD). The parameters that are to be determined include the baselearning rate(lr), momentum rate((momentum)) and regularizer weight((weight decay)).You can also set different learning rate on different layers.

• Base learning rate: Larger learning rate (around 0.1) will change the weights dramat

ically, but will be useful when you train the model from scratch. Smaller learning rate

(0.01 0.0001) will be useful in fine-tuning from the pretrained models.

• Learning rate scheduler: The learning rate need to be cut down as the training isproceeding. The commonly used are step, multiple-step and exponential. You can writethe update policy by yourself or use the methods in torch.optim.lr scheduler.ELEC5307 Deep Learning 3Project #1: Parameters in Neural Networks

2.3 Result analysis In order to train a deep learning model successfully, the results of the network should be carefully

analyzed. The best way to analyze results is to visualize the values.

• The loss curve for both training and validation. Ideally, both losses should decrease andconverge after several epochs. If the training loss is still going down but the validationloss is increasing, then the model is overfitting. If the losses are still going down whenyou finish, then the model is underfitting. You should try to avoid both situations in areasonable number of epochs.

• The accuracy changes in the training set and validation set. The values should be increas

ing as the epoch is increasing. The pattern should be similar to the changes of losses but

in an opposite direction.

3 Experiments In the experiments, your job is to build a neural network for the classification in CIFAR10 andanalyze the results generated from different hyperparameters. Your taskincludes three parts.

The first part is about running a baseline model. You need to run a baseline model andprovide your visualization results.

• The second part is about finding suitable parameters from the given options. You willbe given several options of batch size, base learning rate and number of epochs,try to find the optimal combination of all these three hyperparameters to get the bestperformance.The third part is about other options. You need to firstly train a new baseline model andhen according to Appendix: Tasks for Part3, analyze the effect ofeachhyperparametersto your result.The detailed descriptions are as follows.

3.1 Part1: Baseline structure

You need to go through the part one of the 'project1_2024.ipynb' file first and run the baseline code.

This file is modified based on the official tutorial from PyTorch: https://pytorch.org/

tutorials/_downloads/cifar10_tutorial.ipynb. However, please note that their settings are

far from ideal.

For this reference and other references, if you used their codes, please point out in comments

and in the end of your submission in the ‘Reference’ part. You will be punished if you use all

others’ codes without changing anything by yourself. You are also be punished if you used

others’ codes but did not indicate in your submission.

You need to split your dataset into three parts: training, validation and test. The test dataset

is ready by default, and you need to separate several images (always smaller than the number of

test set) as validation set in your training. The validation set will help you avoid overfitting

problems.

The average time for running one epoch of such network is around 1-5 minutes depending

on the type of your cpu/gpu. The speed is low because the dataset is quite large. In that case,

you can try to randomly select a bit data from the original dataset to check the performance.

This technique is very useful when you face with large datasets so that you can quickly see the

results instead of waiting for a long time.

ELEC5307 Deep Learning 4Project #1: Parameters in Neural Networks

3.2 Part2: Select hyperparamaters

To successfully train this network, you need to select the proper batch size, base learning

rate and number of epochs. The options are as below:

• batch size: 2, 4, 8
• base learning rate: 0.005, 0.001, 0.0005, 0.0001, 0.00005, 0.00001
• number of epochs: 1, 2, 4, 8, 16

Your job is to select the hyperparameters that will help to train the network to get the best

performance in the test set. Meanwhile, the training time should be as short as possible, which

means you should not leave the network training for a super long time even it has converged

judging from the loss curve.

You are supposed to run the codes for several times, plot the corresponding loss curves

for training and validation, compute the accuracy for validation and test, and finally make a

decision. Please do NOT change the other hyperparameters in the given network for this part.

In the writing part, please provide your output images and your analysis. Your analysis should

include but are not limited to:

• What are the choices that could be empirically ignored without doing any experiments?

Were you correct after you conduct the experiments?

• How many epochs are passed when the network is converged?
• Why too large or too small learning rates are not good choices?
• What are the specification of the computer you are using? E.g. the cpu/gpu type andthe corresponding memory.

• How long do you run an epoch? Did you use samples of the original dataset to speed upyour progress, and how did it work?

• Are there any overfitting problems?

3.3 Part3: Other hyperparameters

For now you are supposed to have the ability to train a neural network. The rest of theexperiment is to build a brand new neural network as a baseline and play with some otherhyperparameters.Although you have learned the structures of different predefined neural networks, you arenot allowed to use them in Project 1. Instead, you should only build anetwork that contains:

• 3 convolutional layers, with the activation functions and pooling layers after each convolutional layers.

• 3 fully connected layers right after the last pooling layer. Please remember to makechanges of the data shapes (using function view) to make the feature maps flow into the

fully connected layer smoothly.

• 1 output layer, which is also a fully connected layer, but the output channel should be 10

(the number of classes).For the channel sizes and convolutional parameters, you are free to select your own hyper

parameters. However, since you are supposed to train the network using cpus, please do notELEC5307 Deep Learning 5Project #1: Parameters in Neural Networksmake layers with very large number of channels. The channel numbers for convolutional layer

should be no larger than 256, and the channel numbers for fully connect layers should be nolarger than 1024.

After you build your baseline model, you need to do the analysis on THREE kinds of

hyperparameters. The hyperparameters you are going to play with are defined by your SID as

indicated in Appendix: Tasks for Part3. For each of the three subtasks, you can play with all

the parameters in the methods. For example, you can select any number for weight decay and

momentum if you are playing with SGD.Please analyze the three tasks and select the best choice for your own network. You canalso change the other hyperparameters in 2.2 to better suit each choice, e.g. batch size, learningrate, etc., but please remember to control variables while you make the analysis. Please note

that your modifications of the hyperparameters based on your baseline network may or may notimprove the accuracy. You need to figure out how the hyperparameters influence the resultsand explain why. The analysis should be included in the written part in the .ipynb file.

You need to write ONE single python file (not a .ipynb file) that includes your baseline

network and your modified network with your trained files, and the output of this python file

should be the accuracy of the test set based on your own models. Your accuracy on test dataset

is also marked based on your baseline or modified network whichever is higher.

4 Submission and Grades You are supposed to finish this project on your own. Your submission should include the JupyterNotebook (‘project1.ipynb’) with your modification and written analyses, and the Python file

(‘project1.py’) with your trained model (named ‘baseline.pth’ and ‘modified.pth’ respectively).The files should be in a .zip file named as ‘project1 firstname lastname yourSID.zip’ withno spaces in the file name and submitted through Canvas. The .zip file can contain some of

he images if necessary. Your codes need to be well commented and the written part in the

notebook file need to have clear sections. The final grades are given based on the followingcriteria. For detailed marking scheme, please refer to Appendix: Marking Scheme.

• Your submissions should strictly follow the instructions.
• Your accuracy need to be no smaller than 70% either using your new baseline network oryour modified network.

• Your codes are correct and well organized. Your codes are well commented and thereferences are clear.

• The written part are well organized and has few typos. The report should contain thecorrect formulas when they are necessary.

• You have covered all the three hyperparameters you are assigned. If you did the wrongtask, the corresponding analysis will not be marked and you will also be punished.

• The visualization results are clear and well defined.
• You have shown your insights into the parameters and drawn some reasonable conclusions.No copy from your classmates. If you and some of your classmates copied codes from the

same online resource, you will also be penalized if you do not make any modifications orprovide the reference.

• Give references on all the codes and papers you referred to.

ELEC5307 Deep Learning 6Project #1: Parameters in Neural Networks

Appendix: Tasks for Part3 The task you are going to do depend on the last three digits from your SID (the 9-digit onyour student card).

 

标签：layers,ELEC5307,network,Deep,should,will,Learning,need,your
From： https://www.cnblogs.com/WX-codinghelp/p/18426850