MarkLogic World 2012 - BBC DSP [Jem Rayfield]

Dynamic Semantic Publishing at the BBC
Jem Rayfield
Lead Tecnical Architect
BBC Future Media

Transcript

1. Future Media © BBC MMXII BBC Dynamic Semantic Publishing [DSP] MarkLogic World 2012 •  Jem Rayfield : Lead Technical Architect •  BBC Future Media
2. Future Media © BBC MMXII Outline BBC News Online BBC World Cup 2010 BBC Sport 2012 BBC News Mobile
3. Future Media © BBC MMXII Radio since 1922 TV Since 1930 Web since 1994
4. Future Media © BBC MMXII http://bbc.co.uk/news online
5. Future Media © BBC MMXII BBC News [Static Publishing]
6. Future Media © BBC MMXII Static News Architecture
7. Future Media © BBC MMXII BBC CPS/CMS Asset Authoring
8. Future Media © BBC MMXII BBC CPS/CMS Index Authoring
9. Future Media © BBC MMXII Static News The Good 1) Simple 2) Scales cheaply 3) Difficult to break [bad rendering logic etc..] 4) Handles high load
10. Future Media © BBC MMXII Static News The BAD 1)  Relational taxonomic meta model 2) Static! Inflexible! SSI! 3) Document publishing 4) Content non re-usable 5) Content non repurpose-able 6) Difficult to personalize 7) Publication per output
11. Future Media © BBC MMXII BBC World Cup 2010 http://bbc.co.uk/worldcup
12. Future Media © BBC MMXII 1.  32 teams, 8 groups, 736 players  776 pages 2.  Fixtures & Results, Groups & Teams pages 3.  To many web pages for too few journalists 4.  Improve the publishing system to help achieve all of this World Cup 2010
13. Future Media © BBC MMXII Page Per Player http://news.bbc.co.uk/sport/football/world_cup_2010/groups_and_teams/team/england/wayne_rooney
14. Future Media © BBC MMXII Page Per Team
15. Future Media © BBC MMXII Page Per Group
16. Future Media © BBC MMXII BBC Sport: http://www.bbc.co.uk/ontologies/sport Open Sport Ontology
17. Future Media © BBC MMXII Semantic publishing USER EXPERIENCE ONTOLOGY TRIPLE STORE
18. Future Media © BBC MMXII Graffiti: Suggest -> Tag [Player]
19. Future Media © BBC MMXII Graffiti: Suggest -> Tag [Location] (Geonames)
20. Future Media © BBC MMXII Rationale •  Automated content publishing •  Huge increase in content breadth (number of manageable pages) •  Content re-use and re-purposing, increasing reach •  Simplified content management •  Journalist headcount reduction •  Multi-dimensional entry points and semantic navigation •  Improved user experience with high levels of user engagement •  Dynamic, state (time|event) and semantic driven page layout •  Personalized content aggregations •  Open data and API’s
21. Future Media © BBC MMXII World Cup DSP Architecture
22. Future Media © BBC MMXII API Stack
23. Future Media © BBC MMXII •  750+ Dynamic aggregations/pages (Player, Squad, Group, etc..) •  Average unique page requests a day : 2 million + •  Average OWLIM SPARQL queries a day : 1 million •  100s RDF statement updates/inserts per minute with full OWL reasoning and associated inference. •  Multi data center fully resilient, clustered 6 node triple store •  RDF graph model ideally suited to model domain representations such as sport World Cup statistics the GOOD
24. Future Media © BBC MMXII •  Sports stories and indices static •  Sport content not responsive or personalized •  RDF Store unable to handle thousands of statistic updates a second •  RDF Store forward-chained closures expensive increase write latency •  RDF graph model and SPARQL not ideally suited to the BBC’s News and Sport document publication model World Cup statistics the BAD
25. Future Media © BBC MMXII BBC Sport 2012; Online Refresh http://bbc.co.uk/sport
26. Future Media © BBC MMXII Sport Refresh 2012 •  Page per Athlete [10,000+], Page per country [200+], Page per Discipline [400-500], Page per venue, Page per team  A lot of output… •  Almost real time statistics and live event pages •  Time coded, metadata annotated, on demand video, 58,000 hours of content •  Far too many web pages for far too few journalists •  DSP annotation architecture to automate content aggregation
27. Future Media © BBC MMXII Sport Refresh 2012; Red -> Bright Yellow
28. Future Media © BBC MMXII 10000+ Dynamic Aggregations
29. Future Media © BBC MMXII Lots of Dynamic (Live) sports stats
30. Future Media © BBC MMXII Dynamic Navigation
31. Future Media © BBC MMXII Static Stories + Dynamic includes
32. Future Media © BBC MMXII Augment architecture with a Content Store 1.  Atomic content assets stored in MarkLogic XML store 2.  XML content queryable via Xquery 3.  Content Assets searchable 4.  Sports statistics searchable/queryable via XQuery 5.  Ontological SPARQL via BigOWLIM, assets Xquery via MarkLogic
33. Future Media © BBC MMXII Static Sports stats (OLD)
34. Future Media © BBC MMXII Dynamic Sports stats
35. Future Media © BBC MMXII API Stack MarkLogic!!
36. Future Media © BBC MMXII “You could run Nasa on that” Lee Pollington
37. Future Media © BBC MMXII Xquery master<->master rep
38. Future Media © BBC MMXII Plenty of Caching
39. Future Media © BBC MMXII Sport Stats REST API
40. Future Media © BBC MMXII Sport Stats REST API SSL Accessible API GET https://api.live.bbc.co.uk/sportsdata/statsapi/football/table/ais/competition/118996114 GET https://api.live.bbc.co.uk/sportsdata/statsapi/football/table/ais/competition/118996114 Accept: application/json GET https://api.live.bbc.co.uk/sportsdata/statsapi/football/videprinter GET https://api.int.bbc.co.uk/sportsdata/statsapi/formula1/year/2012/calendar Accept: application/json etc……etc…..etc….
41. Future Media © BBC MMXII Sport Stats REST API Demo….
42. Future Media © BBC MMXII 2012  http://m.bbc.co.uk/news online
43. Future Media © BBC MMXII Dynamic News mobile •  Multi device capability •  Responsive Web design •  Built on a dynamic service API •  New re-usable content model •  Dynamic assets
44. Future Media © BBC MMXII Responsive Dynamic News mobile (iPad)
45. Future Media © BBC MMXII Responsive Dynamic News mobile (iPhone)
46. Future Media © BBC MMXII Recap  Static News Architecture
47. Future Media © BBC MMXII Dynamic News Architecture
48. Future Media © BBC MMXII New Content Model (Re-usable XML/RDF )
49. Future Media © BBC MMXII MarkLogics handy Xinclude resolution Including story data on news index XML <item> <xi:include href="http://www.bbc.co.uk/asset/13447877" xpointer="xmlns(bbc=http://www.bbc.co.uk/content/asset) xpointer(/bbc:story/bbc:itemMeta)"> <xi:fallback> <!-- Unable to find href="http://www.bbc.co.uk/asset/13447877" xpointer="xmlns(bbc=http://www.bbc.co.uk/content/asset) xpointer(/bbc:story/bbc:itemMeta)" --> </xi:fallback> </xi:include> ...
50. Future Media © BBC MMXII News Index API Including story data on news index XML HTTP GET https://api.live.bbc.co.uk/content/asset/news/technology/ HTTP Headers X-Candy-Audience: Domestic X-Candy-Platform: EnhancedMobile Accept: application/json Or HTTP Headers X-Candy-Audience: Domestic X-Candy-Platform: EnhancedMobile Accept: application/xml Contextualised output •  Audience •  Platform •  Response type
51. Future Media © BBC MMXII News Story API Including story data on news index XML HTTP GET https://api.live.bbc.co.uk/content/asset/news/uk-17829360 HTTP Headers X-Candy-Audience: Domestic X-Candy-Platform: EnhancedMobile Accept: application/json Or HTTP Headers X-Candy-Audience: Domestic X-Candy-Platform: EnhancedMobile Accept: application/xml
52. Future Media © BBC MMXII BBC sport site re-engineered to use fully dynamic approach (News Mobile style) BBC news high web site re-engineered to use fully dynamic approach (News Mobile style) Real-time Olympics 2012 stats and video overlay Upgrade to MarkLogic 5 MarkLogic XA Transactions (Removing handcrafted Xquery for master/master replication) MarkLogic Binary storage R&D Etc…. Platform future…..
53. Future Media © BBC MMXII Questions? Slides: jem.rayfield@bbc.co.uk Twitter: @jemrayfield