INFS3208 – Cloud Computing Programming

School of Information Technology and Electrical Engineering

INFS3208 – Cloud Computing Programming Assignment Task III (10 Marks)

Task description: In this assignment, you are asked to write a piece of Spark code to count occurrences of verbs in the

UN debates and find the most similar debate contents. The returned result should be the top 10 verbs that are most frequently used in all debates and the debate that is most similar to the one we provide. This assignment is to test your ability to use transformation and action operations in Spark

RDD programming and your understanding of Vector Database. You will be given three files,including a UN General Debates dataset (un-general-debates.csv), a verb list (all_verbs.txt)and a verb dictionary file (verb_dict.txt). These source files are expected to be stored in a HDFS.You can choose either Scala or Python to complete this assignment in the Jupyter Notebook. There are

ome technical requirements in your code submission as follows:

Objectives:

1. Read Source Files from HDFS and Create RDDs (1.5 marks):

• Read the UN General Debates dataset (un-general-debates.csv) from HDFS andconvert only the “text” column into an RDD. Details of un-general-debates.csv areprovided in the Preparation section below (1 mark).

• Read the verb list file (all_verbs.txt) and verb dictionary file (verb_dict.txt) fromHDFS and load them into separate RDDs (0.5 marks).

• Note: If you failed to read files from HDFS, you can still read them from the local filesystem in work/nbs/ and complete the following tasks.

2. Use Learned RDD Operations to Preprocess the Debate Texts (3 marks):

• Remove empty lines (0.5 marks).
• Remove punctuations that could attach to the verbs (0.5 marks).

o E.g., “work,” and “work” will be counted differently, if you DO NOT remove thepunctuation.

• Change the capitalization or case of text (0.5 marks).

o E.g., “WORK”, “Work” and “work” will be counted as three different verbs, if you

DO NOT make all of them in lower-case.

• Find all verbs in the RDD by matching the words in the given verb list (all_verbs.txt)0.5 mark).

• Convert all verbs in different tenses into the simple present tense by looking up theverbs in the verb dictionary list (verb_dict.txt) (1 mark).

o E.g., regular verb: “work” - works”, “worked”, and “working”.

o E.g., irregular verb: “begin” - “begins”, “began”, and “begun”. o E.g., linking verb “be” and its various forms, including “is”, “am”, “are”, “was”,“were”, “being” and “been”.

o E.g., (work, 100), (works,50), (working,150) should be counted as (work, 300).

3. Use learned RDD Operations to Count Verb Frequency (3 marks):

• Count the top 10 frequently used verbs in UN debates (2 marks).
• Display the results in the format (“verb1”, count1), (“verb2”, count2), … and in adescending order of the counts (1 marks).

4. Use Vector Database (Faiss) to Find the Most Similar Debate (2.5 marks):

• Convert the original debates into 代 写INFS3208 – Cloud Computing Programming vectors and store them in a proper Index (1.5 mark).
• Search the debate content that has the most similar idea to “Global climate change is

both a serious threat to our planet and survival.” (1 mark)

Preparation: In this individual coding assignment, you will apply your knowledge of Vector Database, Spark, SparkRDD Programming and HDFS (in Lectures 7-10). Firstly, you should read Task Description tounderstand what the task is and what the technical requirements include. Secondly, you should reviewthe creation and usage of Faiss, transformations and actions in Spark, and usage of HDFS in Lecturesand Practicals 7-10. In the Appendix, there are some transformation and action operations you coulduse in this assignment. Lastly, you need to write the code (Scala or Python) in the Jupyter Notebook.

All technical requirements need to be fully met to achieve full marks. You can either practise on

the GCP’s VM or your local machine with Oracle Virtualbox if you are unable to access GCP. Please

read the Example of writing Spark code below to have more details.

Assignment Submission:

 You need to compress only the Jupyter Notebook (.ipynb) file.

 The name of the compressed file should be named “FirstName_LastName_StudentNo.zip”.

 You must make an online submission to Blackboard before 3:00 PM on Friday, 11/10/2024

 Only one extension application could be approved due to medical conditions.

Main Steps: Step 1:Log in your VM instance and change to your home directory. We recommend using a VM instancewith at least 4 vCPUs, 8G memory and 20GB free disk space.

Step 2:

git clone https://github.com/csenw/cca3.git && cd cca3Run these commands to download the required docker-compose.yml file and configuration files. Step 3:sudo chmod -R 777 nbs/docker-compose up -dRun all the containers using docker-compose

Step 4:Open the Jupyter Notebook (http://external_IP:8888) and you can find all the files under thework/nbs/ folder. This is also the folder where you should write the notebook (.ipynb) file. Step 5:docker psdocker exec <container_id> hdfs dfs -put /home/nbs/all_verbs.txt /all_verbs.txt

docker exec <container_id> hdfs dfs -put /home/nbs/verb_dict.txt /verb_dict.txt

docker exec <container_id> hdfs dfs -put /home/nbs/un-general-debates.csv /un

general-debates.csv

Run the above commands to put the three source files into HDFS. Substitute <container_id> with

your namenode container ID. After that, you should see the three files from HDFS web interface athttp://external_IP/explorer.html

Step 6:The un-general-debates.csv is a dataset that includes the text of each country’s statement fromthe general debate, separated by “country”, “session”, “year” and “text”. This dataset includes overforty years of data from different countries, which allows for the exploration of differences betweencountries and over time [1,2]. It is organized in the following format:In this assignment, we only consider the “text” column.The verb_dict.txt file contains different tenses of each verb, separated by commas. The first word

is the simple present tense of the verb. The all_verbs.txt file contains all the verbs.tep 7:Create a Jupyter Notebook to complete the programming objectives.We provide some intermediate output samples below. Please note that these outputs are NOT answersand may vary from your outputs due to different implementations and different Spark behaviours.

• Intermediate output sample 1, take only verbs:
• Intermediate output sample 2, top 10 verb counts (without converting verb tenses):• Intermediate output sample 3, most similar debate:You are free to use your own implementation. However, your result should reasonably reflect the top10 verbs that are most frequently used in UN debates, and the most similar debate contents to thesentence “Global climate change is both a serious threat to our planet and survival.”

Reference: [1] UN General Debates, https://www.kaggle.com/datasets/unitednations/un-general-debates.[2] Alexander Baturo, Niheer Dasandi, and Slava Mikhaylov, "Understanding State Preferences WithText As Data: Introducing the UN General Debate Corpus". Research & Politics, 2017. Appendix:

Transformations:

Transformation

Meaning

map(func)Return a new distributed dataset formed by passing each element of the

source through a function func.filter(func)Return a new dataset formed by selecting those elements of the source onwhich funcreturns true.

flatMap(func)Similar to map, but each input item can be mapped to 0 or more outputtems (so funcshould return a Seq rather than a single item).

union(otherDataset)Return a new dataset that contains the union of the elements in the sourcedataset and the argument.

intersection(otherDataset)Return a new RDD that contains the intersection of elements in the sourcedataset and the argument.

distinct([numPartitions]))Return a new dataset that contains the distinct elements of the source

dataset.groupByKey([numPartitions])

When called on a dataset of (K, V) pairs, returns a dataset of (K,Iterable<V>) pairs.Note: If you are grouping in order to perform an aggregation (such as asum or average) over each key, using reduceByKey or aggregateByKey willyield much better performance.

Note: By default, the level of parallelism in the output depends on thenumber of partitions of the parent RDD. You can pass anoptional numPartitions argument to set a different number of tasks.reduceByKey(func,

numPartitions])When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs

where the values for each key are aggregated using the given reducefunction func, which must be of type (V,V) => V. Like in groupByKey, thenumber of reduce tasks is configurable through an optional secondargument.

sortByKey([ascending],[numPartitions]) When called on a dataset of (K, V) pairs where K implements Ordered,returns a dataset of (K, V) pairs sorted by keys in ascending or descendingorder, as specified in the boolean ascending argument.join(otherDataset,[numPartitions])

When called on datasets of type (K, V) and (K, W), returns a dataset of (K,

(V, W)) pairs with all pairs of elements for each key. Outer joins aresupported through leftOuterJoin, rightOuterJoin, and fullOuterJoin.Actions:

Action Meaning

reduce(func)Aggregate the elements of the dataset using a function func (which takes

two arguments and returns one). The function should be commutativeand associative so that it can be computed correctly in parallel.

collect()Return all the elements of the dataset as an array at the driver program.This is usually useful after a filter or other operation that returns asufficiently small subset of the data.count()Return the number of elements in the dataset.first()Return the first element of the dataset (similar to take(1)).take(n)Return an array with the first n elements of the dataset.

countByKey() Only available on RDDs of type (K, V). Returns a hashmap of (K, Int) pairswith the count of each key.

foreach(func) Run a function func on each element of the dataset. This is usually donefor side effects such as updating an Accumulator or interacting withexternal storage systems.Note: modifying variables other than Accumulators outside ofthe foreach() may result in undefined behavior. See Understandingclosures for more details.