Building the DesignSpark Notification Feed

My talk on how we built the notification feed on DesignSpark for Unified Diff 22/02/2018

Transcript

1. Building the DesignSpark Notification Feed
2. Activity Streams are syndicated activities commonly used in social web apps and services, similar to those in Facebook, Instagram and Twitter. Activity Streams and Feeds are typically the core experience of social networks, they are also used for notification feeds and audit logs. What is an activity stream?
3. Everybody wants to go to Japan has one
4. ian rush m8
8. Flat feeds Types of feeds Aggregated feeds Notification feeds Activity log
9. • Notification feed ✅ • Flat feed ✅ • Activity log (in the future) What did our client want?
10. • DesignSpark is a community website for engineers and makers • A community of ~1m users sharing content on engineering related activities • Portal of a number of popular tools to help engineers complete their projects • Steady stream of visitors, spikes during newsletter send out and big events • Really interested in reporting on site usage and analytics A little about DesignSpark
11. So how did we build this thing?
12. 3 main bits Events Queue Activity
13. 3 main bits Events Queue Activity
14. Event Sourcing 101 • Storing all the changes (events) to the system, rather than just its current state. • When using Event Sourcing, we describe our system’s actions by associating them with Events • Events are a point in the application’s timeline and should be immutable • Use the history of events to derive the current state • Classic example: Bank Account • Store the transactions that give you your balance, not just update your balance
15. Why Event Sourcing? • Needed to store an entire history of all activity that occurred on the site • Wanted to consider our ES activity store as ephemeral • Wanted to be able to rebuild the activity store and consumer network easily • Wanted the handling of activity to be done asynchronously • Wanted deep insights into what activity users were performing on the site • I saw a cool talk from Netflix on how they used it for their downloads feature
16. What are projections? • Projection is about deriving current state from the stream of events • Given a stream of events, we can project them to any structural representation • Could be SQL tables, NoSQL tables, Elastic Search index, whatevs • They work through listeners on the events which is used to build up these read models • The great thing about these read models is they can be considered ephemeral - you can always rebuild them through events
17. What are our projections? • user_followers - to keep track of which users are following who • user_statistics - to keep track of the stats for a user, e.g. no. of followers, no. of likes etc. • user_statistics_content - like the above but related to content, e.g. no. of articles created etc. • user_likes - to keep track of what content a user likes • user_watches - to keep track of what content a user is watching • activity_stream - yep, our ES activity stream is just another projection • consumer_network - same as above, just another projection
18. Event Sourcing lessons learnt • Simple concept, hard to get right • Aim to not need migrations, make schema complete & flexible • Rebuild !== Reconstitute • Aggregate ID from projection to retrieve events • i.e. to get all events for aggregate for reconstitution • Still learning
19. 3 main bits Events Queue Activity
20. FIFO Queue 2 Queues (we use SQS currently) Standard Queue • Used for updates to the consumer network • Needs to guarantee ordering to ensure we don’t end up in situation where someone is following someone they have unfollowed or vice versa • Used for activity • Messages can arrive in any order
21. 3 main bits Events Queue Activity
22. Creating a consumer network { Alice knows Bob, Steve and Christine { { Consumer Producers { • A consumer subscribes to 1 or more producers • A producer can be a user, item of content, a page; anything that can produce a notification • Need to keep track of these relationships
23. Storage Strategies Fan-out on write Fan-out on read v
24. Fan-out on write Store a pre-computed news feed for each consumer
25. Fan-out on read Store atomic activities and compose a notification feed at the query time
26. http://jeffterrace.com/docs/feeding-frenzy-sigmod10-web.pdf
27. Storage technology • Already part of our infrastructure • Documented oriented storage • Experience within the team • Powerful querying and filtering capabilities • Fast • Easily scalable
28. Dealing with Activity { Alice follow Bob { Actor Object • Key concepts: Actor, Verb, Object and sometimes Target • Turns out we’re not the first to do this?! • Hello activitystrea.ms! { Verb { Alice added Why you should buy a Raspberry Pi? in Articles { Actor Object { Verb { Target
29. activitystrea.ms • A specification for working with activity streams, from the same people who are behind oAuth • Really, really useful • Helped massively with some of the decision making around how and what to store
30. activitystrea.ms
31. Alice followed Bob 9 days ago Anatomy of an activity
32. Querying for activity • Following query will get all activity created (produced) by people/ things a user (consumer) is subscribed to • Problem: doesn’t deal with notifying implicit subscribers
33. Audience Targeting Alice comment ‘Top Article!’ on Why you should buy a Raspberry Pi? Who should be notified (see in their notification feed) from the above? • Alice should be notified in her activity log • Author of ‘Why you should buy a Raspberry Pi?’ should be notified that someone has commented on their article • Watchers of ‘Why you should buy a Raspberry Pi?’ should be notified Previous query would only notify followers of Alice, which wasn’t actually a requirement!
35. Solving Audience Targeting • activitystrea.ms to the rescue! Property Value Description to JSON Array of Activity Stream Objects Specifies the public primary audience of an activity. cc JSON Array of Activity Stream Objects Specifies the public secondary audience of an activity. bto JSON Array of Activity Stream Objects Specifies the private primary audience of an activity. bcc JSON Array of Activity Stream Objects Specifies the private secondary audience of an activity. Activity Log Notification Feed where bto is me where cc is things inside my network OR where I am in to Put simply for our use case:
36. Solving Audience Targeting Alice comment ‘Top Article!’ Why you should buy a Raspberry Pi?
37. Current query to get a users notification feed Simples!
38. Other things to note when displaying • Each notification needs to be translatable, we do this on the front end using our standard translator • Need to know when a notification is unread • Dealt with by storing the time the user last viewed their notification feed • Comparing this with notifications since that date to get the number unread • Highlighting any after that date • Dealing with inflection, e.g. You followed Bob, Bob replied to your comment etc.
40. What does it all look like?
41. Thanks! Follow me on MySpace: https://myspace.com/jenkoian Useful links: https://getstream.io/ http://jeffterrace.com/docs/feeding-frenzy-sigmod10-web.pdf http://www.quentinsuire.com/presentations/a-news-feed-with-elasticsearch/index.html#/ http://activitystrea.ms/