AI6012: Machine Learning Methodologie Applications

AI6012: Machine Learning Methodologies & pplications Assignment (25 points)

Important notes: to finish this assignment, you are allowed to look up textbooks orsearch materials via Google for reference. NO plagiarism from classmates is allowed.The submission deadline is by 11:59 pm, Sept. 30, 2022. The file to be submittedis a single PDF (no source codes are required to be submitted). Multiple submissionattempts are allowed, and the last one will be graded. A submission link is availableunder “Assignments” of the course website in NTULearn.

Question 1 (10 marks):

Consider a multi-class classification problem of C classes.

Based on the parametric forms of the conditional probabilities of each class introducedon the 39th Page (“Extension to Multiple Classes”) of the lecture notes of L4, derivethe learning procedure of regularized logistic regression for multi-class classificationproblems.Hint: define a loss function by borrowing an idea from binary classification, and

derive the gradient descent rules to update {w(c)}’s.

Question 2 (5 marks):

This is a hands-on exercise to use the SVC API of scikitlearn1 to train a SVM with the linear kernel and the rbf kernel, respectively, on a binaryclassification dataset. The details of instructions are described as follows.

1. Download the a9a dataset from the LIBSVM Dataset page.This is a preprocessed dataset of the Adult dataset in the UCI Irvine MachineLearning Repository2 , which consists of a training set (available here) and a testset (available here)Each file (the train set or the test set) is a text format in which each line representsa labeled data instance as follows:label index1:value1 index2:value2 ...where “label” denotes the class label of each instance, “indexT” denotes theT-th feature, and valueT denotes the value of the T-th feature of the instance.1Read Pages 63-64 of the lecture notes of L5 for reference2The details of the original Adult dataset can be found here.1This is a sparse format, where only non-zero feature values are stored for eachinstance. For example, suppose given a data set, where each data instance has 5dimensions (features). If a data instance whose label is “+1” and the input datainstance vector is [2 0 2.5 4.3 0], then it is presented in a line as+1 1:2 3:2.5 4:4.3Hint: sciki-learn provides an API (“sklearn.datasets.load svmlight file”) to loadsuch a sparse data format. Detailed information is available here.

2. Regarding the linear kernel, show 3-fold cross-validation results in terms of classification accuracy on the training set with different values of the parameter C in{0.01, 0.05, 0.1, 0.5, 1}, respectively, in the followin代 写AI6012: Machine Learning Methodologie Applicationsg table. Note that for all thether parameters, you can simply use the default values or specify the specificvalues you used in your submitted PDF file.Table 1: The 3-fold cross-validation results of varying values of C in SVC with linearkernel on the a9a training set (in accuracy).

3. Regarding the rbf kernel, show 3-fold cross-validation results in terms of classification accuracy on the training set with different values of the parameter gamma (i.e., σ 2 on the lecture notes) in {0.01, 0.05, 0.1, 0.5, 1} and different values ofthe parameter C in {0.01, 0.05, 0.1, 0.5, 1}, respectively, in the following table.Note that for all the other parameters, you can simply use the default values orspecify the specific values you used in your submitted PDF file.Table 2: The 3-fold cross-validation results of varying values of gamma and C in SVCwith rbf kernel on the a9a training set (in accuracy).Hint: there are no specific APIs that integrates cross-validation into SVMs insciki-learn. However, you can use some APIs under the category “Model Selection → Model validation” to implement it. Some examples can be found here.

4. Based on the results shown in Tables 1-2, determine the best kernel and the bestparameter setting. Use the best kernel with the best parameter setting to train aSVM using the whole training set and make predictions on test set to generatethe following table:2Table 3: Test results of SVC on the a9a test set (in accuracy).Specify which kernel with what parameter settingAccuracy of SVMs?

Question 3 (5 marks): The optimization problem of linear soft-margin SVMs canbe re-formulated as an instance of empirical structural risk minimization (refer to Page37 on L5 notes). Show how to reformulate it. Hint: search reference about the hingeloss.

Question 4 (5 marks):

Using the kernel trick introduced in L5 to extend the regularized linear regression model (L3) to solve nonlinear regression problems. Derive aclosed-form solution (i.e., to derive a kernelized version of the closed-form solution onPage 50 of L3).