Adding Context to Queries: The Story Behind Adobe’s API and UI

Adobe uses Elasticsearch to power its API & UI with realtime, high-traffic business queries involving user-specific values. Marko will share the past, current, and future Elasticsearch implementation plans at Adobe – which includes millions of indexes with user-specific documents of prepared responses.

Transcript

1. ‹#› Marko Iskander, Senior Computer Scientist 01/25/2016
 @markoiskander Adding Context to Queries
2. 2 Elasticsearch Use Case • Small(er) dataset • Complex business queries • Need for high throughput
3. 3 {api} Browse UI Admin Operations
5. Let’s talk demo 5
6. 6 Typekit Browse
7. 7 Typekit Use
8. 8 Photoshop
13. 13 Muse
14. Before Elasticsearch 14
15. 15 MySQL
16. 16 MySQL
17. 17 Before Elasticsearch • Query for all families for a given subscription • Find all it’s metadata from 3+ tables • Merge in the per font data • Add the user specific context on top • Return dataset Queries on Highly Normalized Tables
18. In Comes Elasticsearch 18 Let’s talk about the data
19. name: "Acumin Pro",
 fonts: [{
 name: "Bold",
 licenses: {
 full: {
 available_for_web: true, available_for_desktop: true
 },
 trial: {
 available_for_web: true, available_for_desktop: false
 }
 }
 },
 {
 name: "Light",
 licenses: {
 full: {
 available_for_web: true, available_for_desktop: false
 },
 trial: {
 available_for_web: true, available_for_desktop: false
 }
 }
 }]
20. {
 term: {
 "licensing.<subscription>.available_for_web" => true
 }
 }
21. Bring Your Own License
22. So how do we consider the user’s context when querying our data? 22
23. Custom scripting to the rescue! 23
24. {
 _script: {
 script: “id_web_availability[doc['family_id'].value] ?: false",
 params: {
 id_web_availability: @id_web_availability},
 }
 }
25. How does the data get into Elasticsearch? 25
26. Indexing Elasticsearch Data 26 Data is written to MySQL 1 2 3 4 5 Job is kicked off in an after save hook to sync Elasticsearch Prepare the whole document for the given Record Bulk operation delete/ insert Change takes effect for users
27. 27 Indexing Elasticsearch Data • MySQL is still the source of truth • Indexed data is 100% rebuildable • Rollout of new Fonts and changes to a font are delayed only by seconds • Migrations are easier since restructuring the entire doc means we simply reindex with a different index name
28. API Performance
29. 3 node m4.xlarge 5600 QPS 15 consumer nodes 60 Concurrent Processes Cluster Stats
30. What’s Next For Typekit? 30
31. 31 Summary • Elastic search powers Typekit’s API, Browse UI and various other admin operations • Use a combination of term filters, queries and custom scripting to achieve fast queries for API and UI • Data is written to Elasticsearch Asynchronously via an after save pipeline Complex business queries, small data, high throughput Questions? markoiskander@gmail.com @markoiskander