Reactive SoundCloud - Tackling complexity in large applications

In this talk I give some insights into how we rethought our app architecture from the ground up. I show where the complexity lies in our application and how we use the Reactive Extensions (Rx) to gain the upper hand back on dealing with the representation and composition of asynchronous data flows. I also introduce propeller, our storage library which fits snuggly into our reactive architecture.

Transcript

1. View Slide

2. Reactive SoundCloud
   October 2014
   Matthias Käppler
   Tackling complexity in large
   applications
   

   View Slide

3. Complexity
   title, date, 01 of 10
   

   View Slide

4. Essential Complexity
   vs.
   Accidental Complexity
   title, date, 01 of 10
   No Silver Bullet — Essence and Accidents of Software Engineering,
   Fred Brooks, 1986
   

   View Slide

5. Essential Complexity
   title, date, 01 of 10
   The complexity inherent to your
   problem domain.
   

   View Slide

6. Essential Complexity
   title, date, 01 of 10
   Dealing with multiple asynchronous
   data sources in a fault tolerant manner
   → Propagation of change
   → Propagation of errors
   

   View Slide

7. Accidental Complexity
   title, date, 01 of 10
   Self-inflicted complexity, e.g. due to lack of,
   misuse of, or overuse of abstractions.
   

   View Slide

8. Accidental Complexity
   title, date, 01 of 10
   Unsuitable database abstraction
   Little support for handling async
   No standard event/error model
   Optimizing for the 1%
   

   View Slide

9. title, date, 01 of 10
   “It’s the duty of the architect to solve
   the problems inherent in essential
   complexity without introducing
   accidental complexity.”
   -- Neil Ford
   

   View Slide

10. Architecture
    title, date, 01 of 10
    

    View Slide

11. The Ball of Mud
    title, date, 01 of 10
    

    View Slide

12. Layered Architecture
    title, date, 01 of 10
    

    View Slide

13. Featurized Architecture
    title, date, 01 of 10
    

    View Slide

14. Problem Statement
    title, date, 01 of 10
    Introduce a uniform model for
    concurrency, compositional events
    and error handling.
    

    View Slide

15. Proposition
    title, date, 01 of 10
    This can be accomplished using
    functional reactive programming.
    

    View Slide

16. Proposition
    title, date, 01 of 10
    Sorry, I meant
    functional-ish reactive-ish
    programming.
    “For purity we are trying to
    avoid using the term FRP
    in the context of Rx”
    https://twitter.
    com/headinthebox/status/4925832274090721
    28
    

    View Slide

17. Technologies
    title, date, 01 of 10
    

    View Slide

18. RxJava
    github.com/ReactiveX/RxJava
    title, date, 01 of 10
    

    View Slide

19. Messaging Protocol
    title, date, 01 of 10
    Observer
    Observer
    Observable
    Observer
    Observable
    onNext(T) onError(Throwable)
    React
    Deliver /
    Transform
    Emit
    

    View Slide

20. Transformable Streams
    title, date, 01 of 10
    Observer
    Observable
    Observer
    onNext(T)
    Operator
    onNext(R)
    

    View Slide

21. Transformable Streams
    title, date, 01 of 10
    https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables
    

    View Slide

22. Example: Loading data
    Observable track(Urn trackUrn) {
    return trackFromStorage(trackUrn)
    .toList()
    .flatMap(syncIfEmpty(trackUrn));
    }
    Observable fullTrack(Urn trackUrn) {
    return trackFromStorage(trackUrn)
    .zipWith(trackStorage.trackDetails(trackUrn),
    mergeFunction);
    }
    title, date, 01 of 10
    

    View Slide

23. Schedulers
    track(trackUrn)
    .subscribeOn(Schedulers.newThread())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(observer);
    title, date, 01 of 10
    Parameterized concurrency via schedulers
    

    View Slide

24. Message Passing
    title, date, 01 of 10
    interface EventBus {
    Subscription subscribe(Queue queue,
    Observer observer);
    void publish(Queue queue, T event);
    Subject queue(Queue queue);
    }
    

    View Slide

25. Message Passing
    title, date, 01 of 10
    Receiver
    @Inject EventBus eventBus;
    eventBus.subscribe(EventQueue.MY_QUEUE,
    new Subscriber() {
    public void onNext(MyEvent event) { … }
    });
    Sender
    @Inject EventBus eventBus;
    eventBus.publish(EventQueue.MY_QUEUE, new MyEvent());
    

    View Slide

26. propeller
    title, date, 01 of 10
    

    View Slide

27. propeller
    title, date, 01 of 10
    

    View Slide

28. Traditional Storage
    title, date, 01 of 10
    Storage
    Logic
    Presentation
    Network
    class Track {
    String title
    User user
    }
    class TrackStorage {
    void insert(Track track)
    Track load(long trackId)
    }
    Track
    Track
    

    View Slide

29. CQRS
    title, date, 01 of 10
    ReadStorage
    Logic
    Presentation
    Network
    class ApiTrack {
    String title
    User user
    }
    class TrackWriteStorage {
    void insert(ApiTrack track)
    }
    class TrackReadStorage {
    PropertySet load(long id)
    }
    WriteStorage
    ApiTrack
    PropertySet
    

    View Slide

30. Traditional Model
    title, date, 01 of 10
    class Track {
    private Urn urn;
    private String title;
    private User user;
    }
    

    View Slide

31. Traditional Model
    title, date, 01 of 10
    class Track {
    private Urn urn;
    private String title;
    private User user;
    }
    Track track = new Track();
    track.setUrn(new Urn(“soundcloud:tracks:1”));
    track.setTitle(“title”);
    track.setUser(new User(...));
    

    View Slide

32. Properties
    title, date, 01 of 10
    interface Track {
    Property URN = Property.of(Urn.class);
    Property TITLE = Property.of(String.class);
    Property CREATOR = User.URN;
    }
    

    View Slide

33. Property Sets
    title, date, 01 of 10
    interface Track {
    Property URN = Property.of(Urn.class);
    Property TITLE = Property.of(String.class);
    Property CREATOR = User.URN;
    }
    PropertySet track = PropertySet.from(
    Track.URN.bind(new Urn(“soundcloud:tracks:1”)),
    Track.TITLE.bind(“track title”)
    Track.CREATOR.bind(new Urn(“soundcloud:users:2”))
    );
    

    View Slide

34. (Type) safe & composable
    title, date, 01 of 10
    // will throw in if TITLE is missing
    String title = track.get(Track.TITLE);
    // returns ‘alt’ if TITLE is missing
    String title = track.getOrElse(Track.TITLE,
    “alt”);
    // returns null if TITLE is missing
    String title = track.getOrElseNull(Track.TITLE);
    // combines two sets
    PropertySet ps3 = ps1.merge(ps2);
    

    View Slide

35. propeller
    title, date, 01 of 10
    Design goals
    → Minimal bias
    → Symmetric push and pull
    → Uniform error handling
    

    View Slide

36. Query
    title, date, 01 of 10
    PropellerDatabase propeller =
    new PropellerDatabase(sqliteDatabase);
    Query q = Query.from(“tracks”)
    .select(“title”)
    .whereEq(“plays”, 42)
    .limit(1);
    QueryResult result = propeller.query(q);
    

    View Slide

37. Result Iterators
    title, date, 01 of 10
    for (CursorReader row : result) {
    String title = row.getString(“title”);
    ...
    }
    

    View Slide

38. Row Mapping
    title, date, 01 of 10
    mapper = new ResultMapper {
    @Override
    public Track map(CursorReader row) {
    return new Track(
    row.getString(“title”)
    );
    }
    }
    List tracks = result.toList(mapper);
    

    View Slide

39. Result Observables
    title, date, 01 of 10
    DatabaseScheduler scheduler =
    new DatabaseScheduler(prop, sched);
    Observable tracks =
    scheduler.scheduleQuery(query).map(mapper);
    

    View Slide

40. What else?
    title, date, 01 of 10
    Read
    Joining, filtering, ordering, limiting, column functions
    Write
    (bulk)insert, (bulk)upsert, update, delete, truncate
    Transactions
    Integration test helpers
    

    View Slide

41. Summary
    title, date, 01 of 10
    Rx reduces complexity when composing data sources
    Rx only provides the cogs, YOU build the clockwork
    Don’t fall victim to MacGyver, always challenge your
    architecture!
    

    View Slide

42. Thank you
    @mttkay
    mk@soundcloud.com
    

    View Slide

43. Presentation: Fragments
    title, date, 01 of 10
    class CommentsFragment extends DefaultFragment
    implements ReactiveListComponent<...> {
    @Inject CommentsOperations operations;
    @Inject EndlessAdapter adapter
    @Inject ListViewController controller;
    private ConnectableObservable> comments;
    private Subscription sub = Subscriptions.empty()
    // life-cycle methods
    ...
    }
    

    View Slide

44. Presentation: Fragments
    title, date, 01 of 10
    public Observable> buildObservable() {
    TrackUrn trackUrn = getArguments().getParcelable(...);
    this.comments = operations.comments(trackUrn)
    .map(toCommentViewModel)
    .observeOn(mainThread())
    .replay(1);
    this.comments.subscribe(adapter);
    }
    public Subscription connectObservable() {
    this.sub = comments.connect();
    return sub;
    }
    

    View Slide

45. Presentation: Fragments
    title, date, 01 of 10
    public void onViewCreated(View view, Bundle bundle) {
    super.onViewCreated(view, bundle);
    controller.connect(this, comments);
    }
    public void onDestroy() {
    this.sub.unsubscribe();
    super.onDestroy();
    }
    

    View Slide

46. Practices
    title, date, 01 of 10
    

    View Slide

47. Branch by Abstraction
    title, date, 01 of 10
    <>
    branch
    <>
    master
    <>
    origin/master
    

    View Slide

48. master
    Branch by Abstraction
    title, date, 01 of 10
    abstraction
    layer
    http://martinfowler.com/bliki/BranchByAbstraction.html
    <> <>
    

    View Slide

49. master
    Branch by Abstraction
    title, date, 01 of 10
    abstraction
    layer
    http://martinfowler.com/bliki/BranchByAbstraction.html
    <>
    

    View Slide

50. master
    Branch by Abstraction
    title, date, 01 of 10
    http://martinfowler.com/bliki/BranchByAbstraction.html
    <>
    

    View Slide

51. master
    Trunk Based Development
    title, date, 01 of 10
    feature-B .isEnabled()
    .isEnabled()
    feature-A
    .isEnabled()
    feature-C
    

    View Slide

52. Confident Code
    title, date, 01 of 10
    http://en.wikipedia.org/wiki/Tony_Hoare
    

    View Slide

53. Confident Code
    title, date, 01 of 10
    http://remoteworker.files.wordpress.com/2010/11/rep3-100822-020.
    jpg?w=225&h=300
    

    View Slide

54. Confident Code
    title, date, 01 of 10
    Uses “null objects” or option types:
    Collections.emptyList()
    Optional.of()
    

    View Slide

55. Confident Code
    title, date, 01 of 10
    Follows a narrative structure that
    tells, not asks:
    1. Gather inputs
    2. Perform work
    3. Deliver result
    4. Handle failure
    

    View Slide