Content classifier presentation

Transcript

1. Content classifier
2. Algorithm - Take the page as an input - Remove common words - Count occurrences of every word - Sort and calculate time - Classify it
3. Input with open("page.txt") as page: pageContent = page.read().lower().split()
4. Remove common words with open("common-english-words.txt") as com: commonWords = com.read().lower().split(",") for word in commonWords: while(word in pageContent): pageContent.remove(word)
5. Adjustment #1 - Improve your removing procedure for word in commonWords: i = 0 while i < len(pageContent): if pageContent[i] == word: pageContent.pop(i) else: i += 1
6. jack Anna Rose Mike Jill Next available position jack Anna Rose Jill Low-level details of lists implementation Removal of element “Mike” jack Anna Rose Jill
7. count occurrences of every word words_occurrences = {} for word in pageContent: if word in words_occurrences: words_occurrences[word] += 1 else: words_occurrences[word] = 1
8. Dictionaries (hash tables) overview A hash table is a data structure that is used to store ( keys : value ) pairs Let’s store this pair ( Ahmed : 420 ) Key Hash index Ahmed 479 79 … Ahmed : 420 … 99 98 97 79 3 2 1 0 now we can store and search for data in constant time -- NOT EXACTLY --
9. Sort and calculate time words_occurrences = list(words_occurrences.items()) #Bubble sort beginning = time.time() sorted_list = ourAlgo.BubbleSort(words_occurrences[:]) t = (time.time()-beginning)*1000 . . .
10. Classification Categories Sports Science Food Politics WE WILL BUILD A
11. Adjustment #2 - Remove repeated words in your wordlists repeated = {} for word in sports: repeated[word] = 1 for word in politics: if word in repeated: repeated[word] += 1 else: repeated[word] = 1 for word in food: if word in repeated: repeated[word] += 1 else: repeated[word] = 1 for word in science: if word in repeated: repeated[word] += 1 else: repeated[word] = 1 for key,value in repeated.items(): if value > 1: if key in sports: sports.remove(key) if key in science: science.remove(key) if key in politics: politics.remove(key) if key in food: food.remove(key)
12. Adjustment #3 - Add a new category categories = [["sports", 0], ["politics", 0], ["food", 0], ["science", 0],["Art", 0]]
13. Web scraping overview Web scraping is extracting data from a website. HTML Web scraping Data
14. from bs4 import BeautifulSoup from urllib import request pageContent = str(request.urlopen("https://www.euronews.com/2019/10/17/bu lgarian-police-arrest-six-after-racist-abuse-hurled-at-england-players- during-euro-2020").read()) pageContent = BeautifulSoup(pageContent,"html.parser") for item in pageContent.select("p"): print(item.get_text()) no more copypasta
16. Team 7 Ahmed Elghrbawy Mahmoud Saad Ahmed Raslan Ahmed Eissa Mohamed Jamal Mahmoud Afify Mohamed Mobarek Mohamed Elsayed Mahmoud Abd Elhaleem Ahmed Ibrahim