Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reactive SoundCloud - Tackling complexity in large applications

Reactive SoundCloud - Tackling complexity in large applications

In this talk I give some insights into how we rethought our app architecture from the ground up. I show where the complexity lies in our application and how we use the Reactive Extensions (Rx) to gain the upper hand back on dealing with the representation and composition of asynchronous data flows. I also introduce propeller, our storage library which fits snuggly into our reactive architecture.

Matthias Käppler

October 10, 2014
Tweet

More Decks by Matthias Käppler

Other Decks in Programming

Transcript

  1. Reactive SoundCloud
    October 2014
    Matthias Käppler
    Tackling complexity in large
    applications

    View full-size slide

  2. Complexity
    title, date, 01 of 10

    View full-size slide

  3. Essential Complexity
    vs.
    Accidental Complexity
    title, date, 01 of 10
    No Silver Bullet — Essence and Accidents of Software Engineering,
    Fred Brooks, 1986

    View full-size slide

  4. Essential Complexity
    title, date, 01 of 10
    The complexity inherent to your
    problem domain.

    View full-size slide

  5. Essential Complexity
    title, date, 01 of 10
    Dealing with multiple asynchronous
    data sources in a fault tolerant manner
    → Propagation of change
    → Propagation of errors

    View full-size slide

  6. Accidental Complexity
    title, date, 01 of 10
    Self-inflicted complexity, e.g. due to lack of,
    misuse of, or overuse of abstractions.

    View full-size slide

  7. Accidental Complexity
    title, date, 01 of 10
    Unsuitable database abstraction
    Little support for handling async
    No standard event/error model
    Optimizing for the 1%

    View full-size slide

  8. title, date, 01 of 10
    “It’s the duty of the architect to solve
    the problems inherent in essential
    complexity without introducing
    accidental complexity.”
    -- Neil Ford

    View full-size slide

  9. Architecture
    title, date, 01 of 10

    View full-size slide

  10. The Ball of Mud
    title, date, 01 of 10

    View full-size slide

  11. Layered Architecture
    title, date, 01 of 10

    View full-size slide

  12. Featurized Architecture
    title, date, 01 of 10

    View full-size slide

  13. Problem Statement
    title, date, 01 of 10
    Introduce a uniform model for
    concurrency, compositional events
    and error handling.

    View full-size slide

  14. Proposition
    title, date, 01 of 10
    This can be accomplished using
    functional reactive programming.

    View full-size slide

  15. Proposition
    title, date, 01 of 10
    Sorry, I meant
    functional-ish reactive-ish
    programming.
    “For purity we are trying to
    avoid using the term FRP
    in the context of Rx”
    https://twitter.
    com/headinthebox/status/4925832274090721
    28

    View full-size slide

  16. Technologies
    title, date, 01 of 10

    View full-size slide

  17. RxJava
    github.com/ReactiveX/RxJava
    title, date, 01 of 10

    View full-size slide

  18. Messaging Protocol
    title, date, 01 of 10
    Observer
    Observer
    Observable
    Observer
    Observable
    onNext(T) onError(Throwable)
    React
    Deliver /
    Transform
    Emit

    View full-size slide

  19. Transformable Streams
    title, date, 01 of 10
    Observer
    Observable
    Observer
    onNext(T)
    Operator
    onNext(R)

    View full-size slide

  20. Transformable Streams
    title, date, 01 of 10
    https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables

    View full-size slide

  21. Example: Loading data
    Observable track(Urn trackUrn) {
    return trackFromStorage(trackUrn)
    .toList()
    .flatMap(syncIfEmpty(trackUrn));
    }
    Observable fullTrack(Urn trackUrn) {
    return trackFromStorage(trackUrn)
    .zipWith(trackStorage.trackDetails(trackUrn),
    mergeFunction);
    }
    title, date, 01 of 10

    View full-size slide

  22. Schedulers
    track(trackUrn)
    .subscribeOn(Schedulers.newThread())
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(observer);
    title, date, 01 of 10
    Parameterized concurrency via schedulers

    View full-size slide

  23. Message Passing
    title, date, 01 of 10
    interface EventBus {
    Subscription subscribe(Queue queue,
    Observer observer);
    void publish(Queue queue, T event);
    Subject queue(Queue queue);
    }

    View full-size slide

  24. Message Passing
    title, date, 01 of 10
    Receiver
    @Inject EventBus eventBus;
    eventBus.subscribe(EventQueue.MY_QUEUE,
    new Subscriber() {
    public void onNext(MyEvent event) { … }
    });
    Sender
    @Inject EventBus eventBus;
    eventBus.publish(EventQueue.MY_QUEUE, new MyEvent());

    View full-size slide

  25. propeller
    title, date, 01 of 10

    View full-size slide

  26. propeller
    title, date, 01 of 10

    View full-size slide

  27. Traditional Storage
    title, date, 01 of 10
    Storage
    Logic
    Presentation
    Network
    class Track {
    String title
    User user
    }
    class TrackStorage {
    void insert(Track track)
    Track load(long trackId)
    }
    Track
    Track

    View full-size slide

  28. CQRS
    title, date, 01 of 10
    ReadStorage
    Logic
    Presentation
    Network
    class ApiTrack {
    String title
    User user
    }
    class TrackWriteStorage {
    void insert(ApiTrack track)
    }
    class TrackReadStorage {
    PropertySet load(long id)
    }
    WriteStorage
    ApiTrack
    PropertySet

    View full-size slide

  29. Traditional Model
    title, date, 01 of 10
    class Track {
    private Urn urn;
    private String title;
    private User user;
    }

    View full-size slide

  30. Traditional Model
    title, date, 01 of 10
    class Track {
    private Urn urn;
    private String title;
    private User user;
    }
    Track track = new Track();
    track.setUrn(new Urn(“soundcloud:tracks:1”));
    track.setTitle(“title”);
    track.setUser(new User(...));

    View full-size slide

  31. Properties
    title, date, 01 of 10
    interface Track {
    Property URN = Property.of(Urn.class);
    Property TITLE = Property.of(String.class);
    Property CREATOR = User.URN;
    }

    View full-size slide

  32. Property Sets
    title, date, 01 of 10
    interface Track {
    Property URN = Property.of(Urn.class);
    Property TITLE = Property.of(String.class);
    Property CREATOR = User.URN;
    }
    PropertySet track = PropertySet.from(
    Track.URN.bind(new Urn(“soundcloud:tracks:1”)),
    Track.TITLE.bind(“track title”)
    Track.CREATOR.bind(new Urn(“soundcloud:users:2”))
    );

    View full-size slide

  33. (Type) safe & composable
    title, date, 01 of 10
    // will throw in if TITLE is missing
    String title = track.get(Track.TITLE);
    // returns ‘alt’ if TITLE is missing
    String title = track.getOrElse(Track.TITLE,
    “alt”);
    // returns null if TITLE is missing
    String title = track.getOrElseNull(Track.TITLE);
    // combines two sets
    PropertySet ps3 = ps1.merge(ps2);

    View full-size slide

  34. propeller
    title, date, 01 of 10
    Design goals
    → Minimal bias
    → Symmetric push and pull
    → Uniform error handling

    View full-size slide

  35. Query
    title, date, 01 of 10
    PropellerDatabase propeller =
    new PropellerDatabase(sqliteDatabase);
    Query q = Query.from(“tracks”)
    .select(“title”)
    .whereEq(“plays”, 42)
    .limit(1);
    QueryResult result = propeller.query(q);

    View full-size slide

  36. Result Iterators
    title, date, 01 of 10
    for (CursorReader row : result) {
    String title = row.getString(“title”);
    ...
    }

    View full-size slide

  37. Row Mapping
    title, date, 01 of 10
    mapper = new ResultMapper {
    @Override
    public Track map(CursorReader row) {
    return new Track(
    row.getString(“title”)
    );
    }
    }
    List tracks = result.toList(mapper);

    View full-size slide

  38. Result Observables
    title, date, 01 of 10
    DatabaseScheduler scheduler =
    new DatabaseScheduler(prop, sched);
    Observable tracks =
    scheduler.scheduleQuery(query).map(mapper);

    View full-size slide

  39. What else?
    title, date, 01 of 10
    Read
    Joining, filtering, ordering, limiting, column functions
    Write
    (bulk)insert, (bulk)upsert, update, delete, truncate
    Transactions
    Integration test helpers

    View full-size slide

  40. Summary
    title, date, 01 of 10
    Rx reduces complexity when composing data sources
    Rx only provides the cogs, YOU build the clockwork
    Don’t fall victim to MacGyver, always challenge your
    architecture!

    View full-size slide

  41. Presentation: Fragments
    title, date, 01 of 10
    class CommentsFragment extends DefaultFragment
    implements ReactiveListComponent<...> {
    @Inject CommentsOperations operations;
    @Inject EndlessAdapter adapter
    @Inject ListViewController controller;
    private ConnectableObservable> comments;
    private Subscription sub = Subscriptions.empty()
    // life-cycle methods
    ...
    }

    View full-size slide

  42. Presentation: Fragments
    title, date, 01 of 10
    public Observable> buildObservable() {
    TrackUrn trackUrn = getArguments().getParcelable(...);
    this.comments = operations.comments(trackUrn)
    .map(toCommentViewModel)
    .observeOn(mainThread())
    .replay(1);
    this.comments.subscribe(adapter);
    }
    public Subscription connectObservable() {
    this.sub = comments.connect();
    return sub;
    }

    View full-size slide

  43. Presentation: Fragments
    title, date, 01 of 10
    public void onViewCreated(View view, Bundle bundle) {
    super.onViewCreated(view, bundle);
    controller.connect(this, comments);
    }
    public void onDestroy() {
    this.sub.unsubscribe();
    super.onDestroy();
    }

    View full-size slide

  44. Practices
    title, date, 01 of 10

    View full-size slide

  45. Branch by Abstraction
    title, date, 01 of 10
    <>
    branch
    <>
    master
    <>
    origin/master

    View full-size slide

  46. master
    Branch by Abstraction
    title, date, 01 of 10
    abstraction
    layer
    http://martinfowler.com/bliki/BranchByAbstraction.html
    <> <>

    View full-size slide

  47. master
    Branch by Abstraction
    title, date, 01 of 10
    abstraction
    layer
    http://martinfowler.com/bliki/BranchByAbstraction.html
    <>

    View full-size slide

  48. master
    Branch by Abstraction
    title, date, 01 of 10
    http://martinfowler.com/bliki/BranchByAbstraction.html
    <>

    View full-size slide

  49. master
    Trunk Based Development
    title, date, 01 of 10
    feature-B .isEnabled()
    .isEnabled()
    feature-A
    .isEnabled()
    feature-C

    View full-size slide

  50. Confident Code
    title, date, 01 of 10
    http://en.wikipedia.org/wiki/Tony_Hoare

    View full-size slide

  51. Confident Code
    title, date, 01 of 10
    http://remoteworker.files.wordpress.com/2010/11/rep3-100822-020.
    jpg?w=225&h=300

    View full-size slide

  52. Confident Code
    title, date, 01 of 10
    Uses “null objects” or option types:
    Collections.emptyList()
    Optional.of()

    View full-size slide

  53. Confident Code
    title, date, 01 of 10
    Follows a narrative structure that
    tells, not asks:
    1. Gather inputs
    2. Perform work
    3. Deliver result
    4. Handle failure

    View full-size slide