Finding Unavailable Data

This talk is about data - where to get it and how to create it if it doesn’t exist. I’ll take the audience through the process of creating the dataset for my most recent project and show how to view unavailable data as an opportunity rather than an obstacle to answering questions. I’ll cover how to get and read data as well as popular libraries for data analysis and processing in Python — NLTK (Natural Language Toolkit), Panda, Gensim and techniques like regular expressions.

Transcript

1. Yeli @YellzHeard omayeli.com
2. Finding Unavailable Data
4. What is the male equivalent of a nun?
5. google.com
6. google.com
7. quora.com
8. quora.com
9. english.stackexchange.com
10. 1. You can find all the gendered words. 2. You can find the equivalent of a gendered word.
11. → lady / gentleman → prince / princess → king / queen → father / mother → seamstress / seamster → ministress / minister → iron man → cougar
14. Where and how to get data
15. APIs Static Data Web Scraping
16. ['woman', 'female', 'girl', 'lady', 'women', 'mother', 'daughter', 'wife'] ['man', 'male', 'boy', 'men', 'son', 'father', 'husband'] A gendered word is a word with one of these terms (above ) in its definition.
19. APIs: Application Programming Interface
20. programmableweb.com
21. wordnik.com
22. wordnik.com
23. ['woman', 'female', 'girl', 'lady', 'women', 'mother', 'daughter', 'wife'] ['man', 'male', 'boy', 'men', 'son', 'father', 'husband']
25. 400 words
26. Static Data .json .txt .csv ...
29. → lady / gentleman → prince / princess → king / queen → father / mother → seamstress → ministress → iron man
30. ['woman', 'female', 'girl', 'lady', 'women', 'mother', 'daughter', 'wife'] ['man', 'male', 'boy', 'men', 'son', 'father', 'husband']
32. Regular Expressions -> a sequence of characters that define a search pattern
33. regextester.com
35. ['woman', 'female', 'girl', 'lady', 'women', 'mother', 'daughter', 'wife'] ['man', 'male', 'boy', 'men', 'son', 'father', 'husband']
37. regextester.com
39. regextester.com
51. ~ 8000 words
53. Patterns
54. Patterns -> object of a preposition
59. nltk -> natural language toolkit -> for processing the english language
60. text-processing.com/demo/tokenize Tokenization -> chopping up a string into pieces (called tokens) -> throwing away certain characters, such as punctuation
64. Patterns -> object of a preposition -> clothing items
66. collinsdictionary.com/us/word-list
68. Web Scraping Icons made by Smashicons from www.flaticon.com/authors/smashicons
69. urllib.request -> opening URLs BeautifulSoup -> parsing HTML documents
74. ~ 4000 words
76. → lady / gentleman → prince / princess → king / queen → father / mother → actor / actress
77. bionlp-www.utu.fi/wv_demo/
78. Word2Vec -> words to vectors
79. suriyadeepan.github.io
80. My meal wasn’t very tasty so I put some maggi on it. My meal wasn’t very tasty so I put some salt on it. My meal wasn’t very tasty so I put some seasoning on it. I sat on the chair to eat my meal.
83. Gensim -> Google trained word2vec model
85. Yeli @YellzHeard omayeli.com