Annotate.io – automated data cleaning Ian Ozsvald @IanOzsvald ModelInsight.io

[email protected] @IanOzsvald PyDataLondon February 2015 Automated data cleaning ● Cleaning raw text is boring, hinders new projects ● Help non-programmers investigate dirty data ● Keynote “The real unsolved problems in data science” ● What tooling exists? ● Can't the machine do this for us?

[email protected] @IanOzsvald PyDataLondon February 2015 CV Cleaning What I have Cleaning operation? Accenture PLC Remove PLC? ACCENTURE Lowercase? Lancôme Remove 'foreign characters'? Lancôme Strip pre/post whitespace? Société Générale Deal with badly encoded data? Huge variey of company names, all hand-written

[email protected] @IanOzsvald PyDataLondon February 2015 Declarative CV Cleaning What I have What I want Accenture PLC accenture ACCENTURE accenture Lancôme lancome Lancôme lancome Société Générale societe generale

[email protected] @IanOzsvald PyDataLondon February 2015 Salary extraction on adzuna

[email protected] @IanOzsvald PyDataLondon February 2015 Salary extraction What I have What I want To 53K w/benefits 53000 30000 OTE plus bonus 30000 £55000 salary 55000 Excellent “” Forty two thousand GBP 42000

[email protected] @IanOzsvald PyDataLondon February 2015 Salary extraction No regular expressions! No fiddling!

[email protected] @IanOzsvald PyDataLondon February 2015 Your data cleaning problems? ● Write-ups: http://ianozsvald.com/ ● http://annotate.io/ announce email list & working demo (Python 2.7&3.4) ● Tell me ● what data do you want to clean? ● where have you lost time cleaning before?