Embedded Interactive Learning Analytics Dashboards with Elasticsearch and Kibana

My talk at the Swiss Elastic Meetup: https://www.meetup.com/elasticsearch-switzerland/events/233201724/

The React Group at EPFL conducts research and development in the field of interaction systems with particular focus on educational technology and learning analytics. Making sense of growing educational data requires deployment and usage of appropriate scalable, real-time processing tools supporting a flexible data schema, such as Elasticsearch. Until recently, the exploitation of Elasticsearch for learning analytics purposes by practitioners was hindered by a high entrance barrier due to the complexity of the query language and the query specificities. This is changing with the ongoing development of Kibana, an open-source tool that allows to conduct analysis, build visualisations of Elasticsearch data through a graphical user interface and combine those visualisations into dashboards. This feature is particularly important since learning dashboards are a popular mean of obtaining insight into the educational data by presenting an aggregated view of different learning indicators on a single display.
In this talk, I will first demonstrate how user-educational platform interactions can be captured by using the xAPI standard. Afterwards, we will build and play with a Kibana dashboard visualising these interactions in real-time. Finally, I will demonstrate how such a dashboard can be embedded into an existing web platform allowing its user to benefit from the provided analytics.

Transcript

1. EMBEDDED INTERACTIVE LEARNING DASHBOARDS WITH ELASTICSEARCH AND KIBANA
 The copyright of images belongs to their authors. Drop me a message at andrii@vozniuk.com to remove Talk description: http://www.meetup.com/elasticsearch-switzerland/events/233201724/ ANDRII VOZNIUK REACT-EPFL Elastic Meetup Switzerland Lausanne, September 2016
2. GOALS
 OF THE TALK • Present the case of using Elasticsearch & Kibana to build learning dashboards • Demonstrate the interactive analytics tools in action • Motivate thinking on how the tools can be useful in your particular cases
3. HAVE YOU EVER USED ELASTICSEARCH OR KIBANA?
4. ANDRII VOZNIUK https://about.me/vozniuk RESEARCHER CS PhD Candidate SOFTWARE ENGINEER Web, Mobile, Data ENTREPRENEUR Knowledge Sharing Systems TODAY Data Engineer
5. BACKGROUND
6. REACT - EPFL L AU S A N N E
7. REACT = COORDINATION & INTERACTION SYSTEMS GROUP We Research Computer-Mediated Interaction And Build Interaction Systems Focus On • Educational Technology • Humanitarian Knowledge Sharing
8. SPEAKUP is a co-located social media to improve audience interaction speakup.info
9. GRAASP a social media as a personal and collaborative learning environment graasp.net graspeo.org
10. GRAASPBOX facilitating
 knowledge delivery into underconnected environments
11. GRAASP IS A MEAN WEB APP M MongoDB E Express.js A AngularJs N Node.js Front-end mongoose express Server Database
12. WE USE ELASTIC TOOLS A LOT SEARCH RECOMMENDATIONS NLP CONTENT ANALYTICS
 INTERACTION ANALYTICS MONITORING
13. GRAASP SHORT DEMO
14. LEARNING ANALYTICS & DASHBOARDS
15. Allow platform end-users to gain insights by exploring the Interactions TEACHERS
 STUDENTS KNOWLEDGE MANAGERS CONTENT CREATORS FOR AWARENESS & REFLECTION THE GOAL OF EMBEDDED INTERACTIVE DASHBOARDS
16. APPROACH
17. ELASTICSEARCH MAPS INDEXES KIBANA VISUALISES INTERACTIONS ANALYTICS PIPELINE PLATFORM CAPTURES INTERACTIONS
18. ANALYTICS PIPELINE PLATFORM CAPTURES INTERACTIONS
19. LOGS ARE NOT THE BEST FORMAT FOR ANALYSIS
20. ESPECIALLY, WHEN COMING FROM MULTIPLE SYSTEMS
21. GETTING TIMESTAMP SHOULD BE EASY, RIGHT? [19/FEB/2015:19:00:00 +0000] 1424372400 [2015-02-19 19:00:00,000] FEB 19 19:00:00 2015-02-19T19:00:00+02:00 APACHE UNIX TIMESTAMP LOG4J POSTFIX.LOG ISO 8601 OVER 40 FOMATS
22. CAPTURING INTERACTIONS WITH xAPI
23. WHAT IS THE PURPOSE OF xAPI? xAPI = ExPERIENCE API It is a standard to capture in a unified way experiences of the user, in our case user-tool interactions
24. HOW DOES xAPI WORK? Users interact with tools These interactions are observed and recorded by the tools as xAPI statements The tools [store] and send the statements to a central system (Learning Record Store or LRS) for further usage, for instance, Analysis
25. XAPI Source: http://www.slideshare.net/aaronesilvers/going-with-xapi
26. TIMESTAMP ACTOR VERB OBJECT RESULT CONTEXT xAPI FORMAT SPECIFICATION WHEN? WHO? DID WHAT? WITH WHAT? WITH WHAT RESULT? IN WHICH CONTEXT? Today at 10:15 Andrii answered the question five with the grade four while it was raining outside
27. INTERACTION TYPES (VERBS) ABANDONED ANSWERED ASKED ATTEMPTED ATTENDED COMMENTED COMPLETED EXITED EXPERIENCED FAILED IMPORTED INITIALIZED INTERACTED LAUNCHED MASTERED PASSED PREFERRED PROGRESSED REGISTERED RESPONDED RESUMED SATISFIED SCORED SHARED SUSPENDED TERMINATED VOIDED WAIVED MESSAGED INVITED MENTIONED
28. SAMPLE XAPI STATEMENT TIMESTAMP ACTOR id name mbox countryname VERB id display OBJECT id definition name description objectType CONTEXT ipAddress location city countryCode countryName
29. ACTIVITY STREAM
30. ELASTICSEARCH MAPS INDEXES ANALYTICS PIPELINE PLATFORM CAPTURES INTERACTIONS
31. YOU KNOW, FOR SEARCH BUT THERE IS MORE THAN THAT elastic.co/products/elasticsearch
32. BRINGING DATA TO ELASTICSEARCH Front-end mongoose express Server Database mongoosastic mongoosastic is a mongoose plugin updating ES on mongoose events
33. ELASTICSEARCH PROCESSING PIPELINE MAPPING ANALYSIS QUERY DATA FLEXIBLE SCHEMA INTERPRETING FIELDS DATA INTO TOKENS TOKENISERS, FILTERS SEARCH ANALYTICS
34. MAPPING TELLS ELASTICSEARCH HOW TO TREAT THE DATA DIFFERENTLY FROM RELATIONAL DATABASES, WHERE SCHEMA IS STATIC IN ELASTICSEARCH MAPPING IS FLEXIBLE DEFAULT MAPPING IS CREATED AUTOMATICALLY Core types: string, integer/long, float/double, boolean, and null Other types: Array, Object, Nested, IP, GeoPoint, GeoShape, Attachment
35. ELASTICSEARCH MAPPING FOR xAPI STATEMENTS
36. ELASTICSEARCH MAPS INDEXES KIBANA VISUALISES INTERACTIONS ANALYTICS PIPELINE PLATFORM CAPTURES INTERACTIONS
37. elastic.co/products/kibana
38. KIBANA IS FOR INTERACTIVE DASHBOARDS
39. KIBANA Enables near-real-time visualisation and analysis of streaming data Allows interactive data exploration and supports cross-filtering Multiple chart types: bar charts, line and scatter plots, histograms, pie charts, maps No need to know programming or query language in most of the cases It’s open-source, there are extensions
40. ANOTHER EXAMPLE WITH LIGHT COLOURS SCHEME
41. ANOTHER EXAMPLE MONITORING MONGODB
42. ELASTICSEARCH MAPS INDEXES KIBANA VISUALISES INTERACTIONS ANALYTICS PIPELINE PLATFORM CAPTURES INTERACTIONS
43. EMBEDDED &
 INTERACTIVE ANALYTICS
44. EMBEDDING DASHBOARD INTO THE PLATFORM Front-end mongoose express Server Database Kibana
 iframe
45. ANALYTICS EMBEDDED INTO THE INTERACTION CONTEXT TRY YOURSELF AT graasp.net
46. DASHBOARD IN GRASP
47. ELK =
 ELASTICSEARCH+ LOGSTASH+
 KIBANA This is an elk as well SUMMARY ELK is a powerful analytics tool that can be integrated into your platform to create value for your users
48. LEARNING RESOURCES
49. ELASTICSEARCH REFERENCE https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html ELK GUIDES https://www.elastic.co/guide/index.html MULTIPLE TUTORIALS http://www.elasticsearchtutorial.com/ YOUTUBE VIDEOS https://www.youtube.com/watch?v=Kqs7UcCJquM
 https://www.youtube.com/watch?v=wHWb1d_VGp8 https://www.youtube.com/watch?v=1gnpzL9jBqY https://www.youtube.com/watch?v=SH5hLM2asB8 GO THROUGH IT. IT IS REALLY GOOD.
50. BOOKS
51. BOOKS
52. QUESTIONS? FEEDBACK? about.me/vozniuk andrii@vozniuk.com DESIGNED BY LINA BERZHANER