OWL Reasoner Evaluation 2013 competition results #ore2013

46ee69b10ad91bdf47c4df1effeebb55?s=47 spbail
July 24, 2013

OWL Reasoner Evaluation 2013 competition results #ore2013

My slides announcing the results of the ORE 2013 OWL reasoner competition. Note that I announced Konclude as the winner of the RL satisfiability challenge, but that was an error with the graphs which is now corrected - the actual winner was MORe (although Konclude did almost as well and was significantly faster).

46ee69b10ad91bdf47c4df1effeebb55?s=128

spbail

July 24, 2013
Tweet

Transcript

  1. Competition: Rafael Gonçalves, Nicolas Matentzoglu, Bijan Parsia Organisers / PC

    chairs: Ernesto Jimenez Ruiz, Samantha Bail
  2. BRIEF OVERVIEW • Competition details • What we measure •

    The corpus • Offline competition results • Full corpus • User submitted ontologies • Online competition results • Winning reasoners • Winning bet
  3. COMPETITION DETAILS • 14 participating reasoners • 9 OWL 2

    DL, 4 EL,1 RL • Challenges (per profile) • Classification • Consistency • Satisfiability (of randomly selected classes) • Corpus • random sample from web crawl + known repositories • 204 DL, 200 EL, 197 RL • TrOWL • Konclude • TReasoner • HermiT • MORe • FaCT++ • Jfact • Chainsaw • WSClassifier • ELK • jcel • SnoRocket • ELepHant • BaseVISor
  4. SUBMITTED ONTOLOGIES • Data Mining OPtimization (DMOP) - C. Maria

    Keet, Agnieszka !awrynowicz, Claudia d’Amato, Melanie Hilario • Genomic CDS - Matthias Samwald • FMA and GALEN versions - Weihong Song, Bruce Spencer, Weichang Du • KB Bio 101- Vinay K. Chaudhri, Michael A. Wessel, Stijn Heymans
  5. COMPETITION DETAILS Rafael “Benchmark” Gonçalves

  6. Rafael “Sadface” Goncalves TEETHING ISSUES

  7. TEETHING ISSUES • Over a week to get systems to

    adhere to our I/O specs • OWL API parsers/serializers causing problems = painful • Memory/time management = painful (combining ulimit and Java’s Xmx results in many errors) • Running 14 different systems in different languages on one machine/OS = painful • Conclusion: running a reasoner competition is tough. • But: everything will be easier next year!
  8. WHAT WE MEASURE Reasoner OWL ontology (valid syntax, non-trivial, in

    profile) Fail • error (parser, Java...) • timeout (5 minutes) • incorrect result Reasoning task Success • processed without errors • completed without timeout • correct result
  9. THE CORPUS !""#$%&#'()(*$+,"!")'-#.$/01")*$2,3)-)4.$5-678(9'()$/:2$ ;(<"9,'")4=$ >2$ ?@-##$$ 5(*,A@$ 2-)B($ C2$ ?@-##$$

    5(*,A@$ 2-)B($ D2$ ?@-##$$ 5(*,A@$ 2-)B($ ;2$ ?@-##$$ 5(*,A@$ 2-)B($ EF$?@-##$GEFF$-0$$ HFF$5(*,A@$EFFIJKKK$ HFF$2-)B($LEFFF$ ;-6*"@$?(#(7M"6$
  10. THE CORPUS DL EL RL

  11. THE HARDWARE • We aimed for “standard” machines: • Cluster

    of identical PCs (1 reasoner per machine) • QuadCore Intel Xeon CPU @ 2.33GHz • 12GB RAM / 8GB RAM assigned to process • Running some rather old Fedora version (Fedora 12) • Java version1.6.0_18
  12. OFFLINE-COMPETITION

  13. RESULTS: CLASSIFICATION DL Winner: HermiT

  14. RESULTS: CONSISTENCY DL Winner: Konclude (also fastest!)

  15. RESULTS: SAT DL Winner: Konclude

  16. RESULTS: CLASSIFICATION EL Winner: ELK

  17. RESULTS: CONSISTENCY EL Winner: ELK (also fastest!)

  18. RESULTS: SAT EL Winner: Chainsaw

  19. RESULTS: CLASSIFICATION RL Winner: TReasoner

  20. RESULTS: CONSISTENCY RL Winner: Konclude (also fastest!)

  21. RESULTS: SAT RL Winner: MORe

  22. SPECIAL MENTION The jury says: “This candidate only entered one

    competition and did struggle with a high number of timeouts, but the ones that were classified were incredibly fast. We want to see more!” Thumbs up, ELepHant.
  23. BEST NEWCOMER The jury says: “This candidate hardly ever made

    it to the top, but consistently performed well in terms of time and robustness - a steady, reliable workhorse.” Well done, MORe!
  24. LIVE-COMPETITION

  25. LIVE COMPETITION - EL The winning EL reasoner correctly classified

    196 out of 200 OWL 2 EL ontologies. Well done, ELK!
  26. LIVE COMPETITION - DL The winning DL reasoner correctly classified

    153 out of 221 OWL 2 DL ontologies. Well done, WSClassifier!
  27. LIVE COMPETITION - BETS We have a clear winner who

    bet exactly the right number for their reasoner. The winner will be announced at the Social Dinner tonight!
  28. WHAT NEXT? • Data will be available online soon! •

    Benchmarking framework • Ontologies • Results • ORE 2014 in Vienna! • Ongoing activities on the W3C OWLED community group • More information: http://ore2013.cs.manchester.ac.uk
  29. ACKNOWLEDGEMENTS • ORE participants (reasoner and ontology submissions!) • PC

    members • Additional reviewers • DL organisers • in particular the local organisers Birte Glimm & Yevgeny Kazakov & their helpers • The reasoner tamers: Rafael Gonçalves & Nico Matentzoglu • Infrastructure provider • Our sponsor!