Annif: Automated subject indexing tool using a library, archive and museum metadata corpus

This is the first public presentation about the (initial prototype of) Annif automated subject indexing tool. It was prepared for the LODLAM Challenge 2017 competition. The full video pitch that was based on these slides (but with some extras!) can be found on Vimeo: https://vimeo.com/212577974

Google Slides: https://tinyurl.com/annif-lodlam

Transcript

1. Annif Automated subject indexing tool using a library, archive and museum metadata corpus Osma Suominen National Library of Finland LODLAM Challenge 2017
6. YSA YSO Allärs KOKO €£$
8. Hot tub by a lake Andrei Niemimäki CC BY-SA Metadata about 13M documents Many of them tagged with subjects
9. Hot tub by a lake Andrei Niemimäki CC BY-SA
10. Hot tub by a lake Andrei Niemimäki CC BY-SA
11. Finna API All Finna metadata is !
12. almost 28 000 concepts that can be used for subject indexing
13. Building Annif
15. Finna API subject searches: - renewable natural resources type=Subject - “renewable natural resources” type=Subject - topic_facet:”renewable natural resources” From each result, collect - title - description/abstract/summary, if available
16. Finna API subject searches: - renewable natural resources type=Subject - “renewable natural resources” type=Subject - topic_facet:”renewable natural resources” From each result, collect - title - description/abstract/summary, if available Renewable energy in power systems Luonnonvaratilinpito. Puuainestilinpito Local politics of renewable energy : Project planning, siting conflicts and citizen participation Sustainable biotechnology : sources of renewable energy Native people and renewable resource management : the 1986 symposium of the Alberta Society of Professional Biologists co-sponsored by Alberta Native Affairs and Indian and Northern Affairs Canada Tiivistelmä: Pienimuotoisten biokaasulaitosten ympäristövaikutus : paikallinen ja globaali näkökulma. Renewable hydrogen and fuel cells in vehicles Community action plan for renewable energies : summary = Aktionsplan der Gemeinschaft für Erneuerbare Energie : zusammenfassung = Plan d'action communautaire dans le domaine des energies renouvelables : resume Environmental impact of household biogas plants in India : local and global perspective Renewable natural resources : a management handbook for the 1980s Bioenergy 2009 : 31.8.-4.9.2009 : book of proceedings part 2 The existence of steady states in growth models with renewable resources and pollution Redox reactions and water quality in cultivated boreal acid sulphate soils in relation to water management Tiivistelmä: Energiantuotanto ja päästöt. Biotechnology and renewable energy Perspectives of renewable energy resources utilization (regional aspects) : proceedings of the third international seminar, 11.-13. September 1995, Petrozavodsk, Russia Renewable energy sources statistics in the European Union : 1989-1997 Traditional knowledge and renewable resource management in northern regions Mechanical, microstructural and barrier properties of agricultural biopolymer films and foams : a literature review Environmental assessment of green chemicals : LCA of bio-based chemicals
17. Renewable energy in power systems Luonnonvaratilinpito. Puuainestilinpito Local politics of renewable energy : Project planning, siting conflicts and citizen participation Sustainable biotechnology : sources of renewable energy Native people and renewable resource management : the 1986 symposium of the Alberta Society of Professional Biologists co-sponsored by Alberta Native Affairs and Indian and Northern Affairs Canada Tiivistelmä: Pienimuotoisten biokaasulaitosten ympäristövaikutus : paikallinen ja globaali näkökulma. Renewable hydrogen and fuel cells in vehicles Community action plan for renewable energies : summary = Aktionsplan der Gemeinschaft für Erneuerbare Energie : zusammenfassung = Plan d'action communautaire dans le domaine des energies renouvelables : resume Environmental impact of household biogas plants in India : local and global perspective Renewable natural resources : a management handbook for the 1980s Bioenergy 2009 : 31.8.-4.9.2009 : book of proceedings part 2 The existence of steady states in growth models with renewable resources and pollution Redox reactions and water quality in cultivated boreal acid sulphate soils in relation to water management Tiivistelmä: Energiantuotanto ja päästöt. Biotechnology and renewable energy Perspectives of renewable energy resources utilization (regional aspects) : proceedings of the third international seminar, 11.-13. September 1995, Petrozavodsk, Russia Renewable energy sources statistics in the European Union : 1989-1997 Traditional knowledge and renewable resource management in northern regions Mechanical, microstructural and barrier properties of agricultural biopolymer films and foams : a literature review Environmental assessment of green chemicals : LCA of bio-based chemicals
18. Renewable energy in power systems Luonnonvaratilinpito. Puuainestilinpito Local politics of renewable energy : Project planning, siting conflicts and citizen participation Sustainable biotechnology : sources of renewable energy Native people and renewable resource management : the 1986 symposium of the Alberta Society of Professional Biologists co-sponsored by Alberta Native Affairs and Indian and Northern Affairs Canada Tiivistelmä: Pienimuotoisten biokaasulaitosten ympäristövaikutus : paikallinen ja globaali näkökulma. Renewable hydrogen and fuel cells in vehicles Community action plan for renewable energies : summary = Aktionsplan der Gemeinschaft für Erneuerbare Energie : zusammenfassung = Plan d'action communautaire dans le domaine des energies renouvelables : resume Environmental impact of household biogas plants in India : local and global perspective Renewable natural resources : a management handbook for the 1980s Bioenergy 2009 : 31.8.-4.9.2009 : book of proceedings part 2 The existence of steady states in growth models with renewable resources and pollution Redox reactions and water quality in cultivated boreal acid sulphate soils in relation to water management Tiivistelmä: Energiantuotanto ja päästöt. Biotechnology and renewable energy Perspectives of renewable energy resources utilization (regional aspects) : proceedings of the third international seminar, 11.-13. September 1995, Petrozavodsk, Russia Renewable energy sources statistics in the European Union : 1989-1997 Traditional knowledge and renewable resource management in northern regions Mechanical, microstructural and barrier properties of agricultural biopolymer films and foams : a literature review Environmental assessment of green chemicals : LCA of bio-based chemicals Elasticsearch text index
19. Renewable energy in power systems Luonnonvaratilinpito. Puuainestilinpito Local politics of renewable energy : Project planning, siting conflicts and citizen participation Sustainable biotechnology : sources of renewable energy Native people and renewable resource management : the 1986 symposium of the Alberta Society of Professional Biologists co-sponsored by Alberta Native Affairs and Indian and Northern Affairs Canada Tiivistelmä: Pienimuotoisten biokaasulaitosten ympäristövaikutus : paikallinen ja globaali näkökulma. Renewable hydrogen and fuel cells in vehicles Community action plan for renewable energies : summary = Aktionsplan der Gemeinschaft für Erneuerbare Energie : zusammenfassung = Plan d'action communautaire dans le domaine des energies renouvelables : resume Environmental impact of household biogas plants in India : local and global perspective Renewable natural resources : a management handbook for the 1980s Bioenergy 2009 : 31.8.-4.9.2009 : book of proceedings part 2 The existence of steady states in growth models with renewable resources and pollution Redox reactions and water quality in cultivated boreal acid sulphate soils in relation to water management Tiivistelmä: Energiantuotanto ja päästöt. Biotechnology and renewable energy Perspectives of renewable energy resources utilization (regional aspects) : proceedings of the third international seminar, 11.-13. September 1995, Petrozavodsk, Russia Renewable energy sources statistics in the European Union : 1989-1997 Traditional knowledge and renewable resource management in northern regions Mechanical, microstructural and barrier properties of agricultural biopolymer films and foams : a literature review Environmental assessment of green chemicals : LCA of bio-based chemicals Elasticsearch text index
20. Using Annif to index new documents
23. transport planning: 100% energy efficiency: 99% transport: 96% bicycle and pedestrian traffic: 100% city bikes (bicycle sharing): 80% naturally occurring substances: 80% hay (green fodder): 100% vegetarian food: 96% wings: 92% material flows: 100% MIPS: 92% eco-efficiency: 94% MIPS: 100% material input: 70% eco-efficiency: 70% traffic policy: 100% goods traffic: 96% service transport: 93%
24. transport planning: 100% energy efficiency: 99% transport: 96% bicycle and pedestrian traffic: 100% city bikes (bicycle sharing): 80% naturally occurring substances: 80% hay (green fodder): 100% vegetarian food: 96% wings: 92% material flows: 100% MIPS: 92% eco-efficiency: 94% MIPS: 100% material input: 70% eco-efficiency: 70% traffic policy: 100% goods traffic: 96% service transport: 93% Overall highest scoring topics: • material flows • eco-efficiency • MIPS • material input • Works • goods • bicycle and pedestrian traffic • transport planning • city bikes (bicycle sharing) • greenhouses
25. Indexing large document collections
26. Indexing Wikipedia by topics Finnish Wikipedia has 410 000 articles (620 MB as raw text) Automated subject indexing took 6-7 hours 1-3 topics per article (average ~2)
27. Indexing Wikipedia by topics Finnish Wikipedia has 410 000 articles (620 MB as raw text) Automated subject indexing took 6-7 hours 1-3 topics per article (average ~2) Examples: (random sample) Wikipedia article YSO topics Ahvenuslammi (Urjala) shores Brasilian Grand Prix 2016 race drivers, formula racing, karting Guy Topelius folk poetry researcher, saccharin HMS Laforey warships Liigacup football, football players Pää Kii ensembles (groups), pop music RT-21M Pioneer missiles Runoja pop music, recording (music recordings), compositions (music) Sjur Røthe skiers, skiing, Nordic combined Veikko Lavi lyricists, comic songs
28. Most common topics in Finnish Wikipedia
29. Most common topics in Finnish Wikipedia Image credits: Petteri Lehtonen [CC BY-SA 3.0] Hockeybroad/Cheryl Adams [CC BY-SA 3.0] Tomisti [CC BY-SA 3.0] Tuomas Vitikainen [CC BY-SA 3.0]
30. Try it yourself
31. Demo web form at annif.org
33. Apply it on your own data! Choose an indexing vocabulary Load the corpus into a text index Prepare a corpus from your existing metadata Use it to index new documents
35. Finna API Finto SPARQL API ocr.space API ResponsiveVoice API Kerttu Suominen Elasticsearch python-elasticsearch langdetect nltk requests Pillow WikiExtractor jQuery jQuery Mobile Code (CC0): https://github.com/osma/annif Demo: annif.org Mobile app: m.annif.org API: api.annif.org Wikipedia-YSO data: SPARQL, CSV These slides: http://tinyurl.com/annif-lodlam