IMSE7140 Cracking CAPTCHAs

IMSE7140 Assignment 2 Cracking CAPTCHAs

(20 points)

2.1 Brief Introduction CAPTCHA or captcha is the acronym for “Completely Automated Public Turing test

to tell Computers and Humans Apart.” You must have been already familiar with itbecause of its popularity in preventing bot attacks or spam everywhere. This assignment, however, will guide you in implementing a deep learning model that can crack acommercial-level captcha!You deliverables for this assignment shouldincludeA single PDF file answers.pdf with answers to all the questions explicitly markedby “Q” with a serial number in this document, and

2. A train.py file to fulfill the programming task requirements marked by “PT.”Of course, GPUs can facilitate your experiments—Don’t worry if you don’t have any,the training requirement is deliberately simplified.

2.2 Training your model The captchas we will crack is the multicolorcaptcha. Please pip install the exact version1.2.0 (the current latest one) in case there might be any incompatibility for other releases.We use the following codes to generate captchas.

1from multicolorcaptcha import CaptchaGenerator23generator = CaptchaGenerator (0)4aptcha = generator . gen_captcha_image ( difficult_level =0)5mage = captcha . image6haracters = captcha . characters7

image . save ( f"{ characters }. png", "PNG")n this snippet, CaptchaGenerator(0) configures the image size to 256 × 144 pixels,nd the difficult level is set to 0 so that the only contains four 0–9 digits.Please run the code snippet on your computer. If the captcha is successfully generated,it should look like Figure 2.1.12.2. Training your model

1. QinFigure 2.1: Sample captcha with digits 0570The training and the validation datasets are generated and attached in folderscapts train and capts val. For any machine learning problem, before you start todeviseasolution, it is always a good idea to observe the data and gain some intuitionfirst. You may immediately recognize some difficulties in this task:

• The digits have a set of random fonts and colors;
• Some certain range of random rotations are applied to the digits;
• Some line segments are randomly added to the image.Such a task is considered impossible for traditional pattern recognition methods,which may tackle the problem in a process like this: image thresholding, segmentation, handcrafted filter design, and pattern matching. We can conjecture that “filterdesign” may fail in capturing useful features and “pattern matching” may havea poorperformance.Fortunately, in the deep learning era, we can delegate the pattern or feature extraction job to deep neural networks. As introduced in the previous lecture “Deep Learningfor Computer Vision,” the slide “Understand feature maps: CAPTCHA recognition”shows that a typical architecture for the task consists oftwo parts:

1. A backbone model to extract a feature map from the captcha image, and
2. A certain amount of prediction heads to interpret the feature map to readableforms.

We will follow this architecture in this assignment. I encourage you to search opensource solutions and learn from their experience. Here we follow this Kaggle post byAshadullah Shawon.

PT| Use capts train as the training dataset, capts val as the validation dataset, and Kerasas the deep learning framework, referring to Shawon’s solution, provide the training codetrain.py that fulfills the following requirements. “Copy and paste” the codes from theoriginal post is allowed, as well as other AI-generated codes.22.3. Example: A practical model

1. Qin
2. The maximal number for epochs should be 10. Considering some studentswill train the model by CPU, it is fair to limit the number of epochs, so the trainingime for the model should be less than half an hour.

2. The accuracy for one digit should be no less than 30% after training for

10 epochs. The training outputs contain four accuracies respective to the fourdigits. Since they are similar, you will only need to examine one of them. Keep inmind that 30% for one digit indicates that the overall 代写IMSE7140 Cracking CAPTCHAs accuracy for the recognitionis only 0.3 4 = 0.81%. Such a low accuracy is not useful for cracking thecaptcha.However, on the one hand, you may need a GPU to experiment on a practicalsolution; on the other hand, a wild guess for a 0–9 digit has an accuracy of 10%,so if your model’s accuracy can reach 30% after 10 epochs, it already indicatesthe model learns from the training set. Hint: if the accuracy for one digit keepswandering around 0.1 but not increasing in the first two or three epochs, it is thignal that you should modify somewhere in your code and try again.

3. The trained model should be saved as a file my model.keras after training.Though, this model file my model.keras doesn’t need to be uploaded.Q1| Can we convert the captcha images to grayscale at the preprocessing stage before training? What is the possible advantage by doing that? If any, can you point out thepossible disadvantage?

Q2| After the 10-epoch training, what are your accuracies of one digit, for the training andthe validation datasets respectively?Q3| Is the accuracy for the validation dataset lower than that for the training dataset? Whatare the possible reasons?

Q4| How can we improve the model’s performance on the validation dataset? List at least

three different measures.

2.3 Example: A practical model

To demonstrate that the backbone–heads architecture can actually solve the real-world

captcha, I trained a relatively large model by an Nvidia GeForce RTX 3090 GPU.

You may find in attached the model file 0991-0.9956.keras and the inference codeinference.py. The accuracies versus training epochs are shown in Figure 2.2. Theinference code reads a randomly generated captcha, inferences the model, and comparesthe predicted results with the targets. You can press “n” for the next captcha or “q” toquit the program. You may need to pip install keras cv to run the code.

Q5| What kind of backbone did I use in the model 0991-0.9956.keras?Q6| The backbone’s pre-trained weights on the ImageNet 2012 dataset were loaded beforetraining. What is the possible advantage by doing that?Q7| Why didn’t Iuse any dropout in the model? Guess the reason.

Q8| In Figure 2.2, you may have noticed that the accuracies rise very fast from 0 to 0.9, butsignificantly slow from 0.95 to 0.99. Explain the phenomenon.

Q9| Using the same hardware (which means you can’t upgrade the GPU, for example), hocan we speed up the learning process of the model, i.e. the rate of convergence?32.3. Example: A practical model

Figure 2.2: Accuracies through 1000 epochs in trainingQ10| Since the accuracy for one digit is about 99%, the overall accuracy for a captcha is

0.994 ≈ 96%. This performance would be better than humans. Can you propose somethat can even further improve the performance?Please note that, not all the questions above have a definite answer. You may alsoneed to do some research as the course doesn’t cover all the details in class. The sourcecode for training this model and the reference answers will be available on Moodle orsent by email after all the students completing the submission.