How Blackboard Curbs Cheating with Elasticsearch

Blackboard uses Elasticsearch to power SafeAssign, its plagiarism detection software. Learn about their internal web search engine, how they transitioned from a Solr deployment to a multi-terabyte, full-text search engine powered by Elasticsearch, and their experiences running their infrastructure on AWS.

Slavomir Tecl l Development Manager l Blackboard

Transcript

1. Blackboard March 8, 2017 This paper has been plagiarized Slavek Tecl, R&D development manager
2. education technology and services for schools, districts and institutions
3. Learning management system Blackboard’s flagship Blackboard Learn
4. Learn plugin Educational tool Plagiarism prevention service Blackboard SafeAssign
5. SafeAssign originality report
6. High level architecture SafeAssign Engine DB Services Logstash ES Logs ES Institutional & global database ES Internet User data
7. Web Institutional & global database Logging, ELK stack • ~3B docs • ~200ms query time • ~10M queries/day • Hundreds of millions docs • Ever growing • ~6M queries/day • 2 months of logging data • ~2B records • Kibana frontend Our clusters
8. Web Institutional & global database Logging, ELK stack • Spark jobs parsing big data • Storm continuous jobs • User data • Logstash agents • Redis queueing Feeding clusters
9. How we built the Web cluster Start small(ish) 1 2 3 4 5 Load with data Performance test Tune up and repeat previous step Enjoy
10. DATA LOADING TOOL? SPARK
11. MOST FREQUENT OPERATION? FULL TEXT QUERY AKA MATCH QUERY
12. PERFORMANCE TESTING? JMETER
13. NOTABLE TAKEAWAYS FROM PERFORMANCE TUNING
14. MERGE INDEXES = SEARCH FASTER* * https://wiki.apache.org/lucene-java/ImproveSearchingSpeed
15. USE MORE CLIENT NODES = OFFLOAD CPU, SEARCH FASTER* * https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-node
16. https://www.elastic.co/guide/en/elasticsearch/guide/current/common-terms.html { } Documents that match only the less important terms are probably of very little interest.
17. USE CUTOFF FREQUENCY = FILTER OUT MATCHES WITH COMMON WORDS
18. RESCORE MATCHES = RUN MORE EXPENSIVE SECONDARY QUERY LATER* * https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-rescore.html
19. SOURCE FIELD TO KEEP OR NOT TO KEEP?
20. KEEP J AT SOME POINT, YOU MAY NEED TO REINDEX THE DATA
21. { } Why doesn’t ES route requests based on search queue size? * https://github.com/elastic/elasticsearch/issues/12496
22. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. Please attribute Elastic with a link to elastic.co