Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building the DesignSpark Notification Feed

Ian Jenkins
February 22, 2018

Building the DesignSpark Notification Feed

My talk on how we built the notification feed on DesignSpark for Unified Diff 22/02/2018

Ian Jenkins

February 22, 2018
Tweet

More Decks by Ian Jenkins

Other Decks in Technology

Transcript

  1. Activity Streams are syndicated activities commonly used in social web

    apps and services, similar to those in Facebook, Instagram and Twitter. Activity Streams and Feeds are typically the core experience of social networks, they are also used for notification feeds and audit logs. What is an activity stream?
  2. • Notification feed ✅ • Flat feed ✅ • Activity

    log (in the future) What did our client want?
  3. • DesignSpark is a community website for engineers and makers

    • A community of ~1m users sharing content on engineering related activities • Portal of a number of popular tools to help engineers complete their projects • Steady stream of visitors, spikes during newsletter send out and big events • Really interested in reporting on site usage and analytics A little about DesignSpark
  4. Event Sourcing 101 • Storing all the changes (events) to

    the system, rather than just its current state. • When using Event Sourcing, we describe our system’s actions by associating them with Events • Events are a point in the application’s timeline and should be immutable • Use the history of events to derive the current state • Classic example: Bank Account • Store the transactions that give you your balance, not just update your balance
  5. Why Event Sourcing? • Needed to store an entire history

    of all activity that occurred on the site • Wanted to consider our ES activity store as ephemeral • Wanted to be able to rebuild the activity store and consumer network easily • Wanted the handling of activity to be done asynchronously • Wanted deep insights into what activity users were performing on the site • I saw a cool talk from Netflix on how they used it for their downloads feature
  6. What are projections? • Projection is about deriving current state

    from the stream of events • Given a stream of events, we can project them to any structural representation • Could be SQL tables, NoSQL tables, Elastic Search index, whatevs • They work through listeners on the events which is used to build up these read models • The great thing about these read models is they can be considered ephemeral - you can always rebuild them through events
  7. What are our projections? • user_followers - to keep track

    of which users are following who • user_statistics - to keep track of the stats for a user, e.g. no. of followers, no. of likes etc. • user_statistics_content - like the above but related to content, e.g. no. of articles created etc. • user_likes - to keep track of what content a user likes • user_watches - to keep track of what content a user is watching • activity_stream - yep, our ES activity stream is just another projection • consumer_network - same as above, just another projection
  8. Event Sourcing lessons learnt • Simple concept, hard to get

    right • Aim to not need migrations, make schema complete & flexible • Rebuild !== Reconstitute • Aggregate ID from projection to retrieve events • i.e. to get all events for aggregate for reconstitution • Still learning
  9. FIFO Queue 2 Queues (we use SQS currently) Standard Queue

    • Used for updates to the consumer network • Needs to guarantee ordering to ensure we don’t end up in situation where someone is following someone they have unfollowed or vice versa • Used for activity • Messages can arrive in any order
  10. Creating a consumer network { Alice knows Bob, Steve and

    Christine { { Consumer Producers { • A consumer subscribes to 1 or more producers • A producer can be a user, item of content, a page; anything that can produce a notification • Need to keep track of these relationships
  11. Storage technology • Already part of our infrastructure • Documented

    oriented storage • Experience within the team • Powerful querying and filtering capabilities • Fast • Easily scalable
  12. Dealing with Activity { Alice follow Bob { Actor Object

    • Key concepts: Actor, Verb, Object and sometimes Target • Turns out we’re not the first to do this?! • Hello activitystrea.ms! { Verb { Alice added Why you should buy a Raspberry Pi? in Articles { Actor Object { Verb { Target
  13. activitystrea.ms • A specification for working with activity streams, from

    the same people who are behind oAuth • Really, really useful • Helped massively with some of the decision making around how and what to store
  14. Querying for activity • Following query will get all activity

    created (produced) by people/ things a user (consumer) is subscribed to • Problem: doesn’t deal with notifying implicit subscribers
  15. Audience Targeting Alice comment ‘Top Article!’ on Why you should

    buy a Raspberry Pi? Who should be notified (see in their notification feed) from the above? • Alice should be notified in her activity log • Author of ‘Why you should buy a Raspberry Pi?’ should be notified that someone has commented on their article • Watchers of ‘Why you should buy a Raspberry Pi?’ should be notified Previous query would only notify followers of Alice, which wasn’t actually a requirement!
  16. Solving Audience Targeting • activitystrea.ms to the rescue! Property Value

    Description to JSON Array of Activity Stream Objects Specifies the public primary audience of an activity. cc JSON Array of Activity Stream Objects Specifies the public secondary audience of an activity. bto JSON Array of Activity Stream Objects Specifies the private primary audience of an activity. bcc JSON Array of Activity Stream Objects Specifies the private secondary audience of an activity. Activity Log Notification Feed where bto is me where cc is things inside my network OR where I am in to Put simply for our use case:
  17. Other things to note when displaying • Each notification needs

    to be translatable, we do this on the front end using our standard translator • Need to know when a notification is unread • Dealt with by storing the time the user last viewed their notification feed • Comparing this with notifications since that date to get the number unread • Highlighting any after that date • Dealing with inflection, e.g. You followed Bob, Bob replied to your comment etc.
  18. Thanks! Follow me on MySpace: https://myspace.com/jenkoian Useful links: https://getstream.io/ http://jeffterrace.com/docs/feeding-frenzy-sigmod10-web.pdf

    http://www.quentinsuire.com/presentations/a-news-feed-with-elasticsearch/index.html#/ http://activitystrea.ms/