Data doesn't grow in tables

Transcript

1. Data doesn’t grow in tables Dealing with large sets of documents
2. –An investigative reporter “We're working with 40 GB of XXX and would like to search within the documents for certain keywords (like XXX) so we can identify XXX. Ideally we should be able to tag the docs..”
3. Some lingo • OCR (Optical Character Recognition) • NLP (Natural Language Processing) • NER (Named
 Entity
 Recognition) • Regular
 Expressions
4. Cases
5. Exhibit A
6. Exhibit B
7. Exhibit C
8. Exhibit D
9. Tools
10. Tables in disguise http://tabula.nerdpower.org
11. Docs in a cloud http://documentcloud.org
12. Clustering, tagging, mining http://overview.ap.org
13. Let them eat PDF https://github.com/CrowData
14. All the visuals Jigsaw
15. Spoken word magic http://sayit.mysociety.org/
16. Whats missing? Easy-to-use ElasticSearch Commercial-grade OCR Configurable pipelines
17. Stefan Wehrmeyer, correctiv.org, @stefanwehrmeyer ! ! ! ! ! ! ! Friedrich Lindenberg, codeforafrica.org, @pudo