Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reactive SoundCloud - Tackling complexity in large applications

Reactive SoundCloud - Tackling complexity in large applications

In this talk I give some insights into how we rethought our app architecture from the ground up. I show where the complexity lies in our application and how we use the Reactive Extensions (Rx) to gain the upper hand back on dealing with the representation and composition of asynchronous data flows. I also introduce propeller, our storage library which fits snuggly into our reactive architecture.

5cfa38a7e54e6c6a850dc6169a699246?s=128

Matthias Käppler

October 10, 2014
Tweet

Transcript

  1. None
  2. Reactive SoundCloud October 2014 Matthias Käppler Tackling complexity in large

    applications
  3. Complexity title, date, 01 of 10

  4. Essential Complexity vs. Accidental Complexity title, date, 01 of 10

    No Silver Bullet — Essence and Accidents of Software Engineering, Fred Brooks, 1986
  5. Essential Complexity title, date, 01 of 10 The complexity inherent

    to your problem domain.
  6. Essential Complexity title, date, 01 of 10 Dealing with multiple

    asynchronous data sources in a fault tolerant manner → Propagation of change → Propagation of errors
  7. Accidental Complexity title, date, 01 of 10 Self-inflicted complexity, e.g.

    due to lack of, misuse of, or overuse of abstractions.
  8. Accidental Complexity title, date, 01 of 10 Unsuitable database abstraction

    Little support for handling async No standard event/error model Optimizing for the 1%
  9. title, date, 01 of 10 “It’s the duty of the

    architect to solve the problems inherent in essential complexity without introducing accidental complexity.” -- Neil Ford
  10. Architecture title, date, 01 of 10

  11. The Ball of Mud title, date, 01 of 10

  12. Layered Architecture title, date, 01 of 10

  13. Featurized Architecture title, date, 01 of 10

  14. Problem Statement title, date, 01 of 10 Introduce a uniform

    model for concurrency, compositional events and error handling.
  15. Proposition title, date, 01 of 10 This can be accomplished

    using functional reactive programming.
  16. Proposition title, date, 01 of 10 Sorry, I meant functional-ish

    reactive-ish programming. “For purity we are trying to avoid using the term FRP in the context of Rx” https://twitter. com/headinthebox/status/4925832274090721 28
  17. Technologies title, date, 01 of 10

  18. RxJava github.com/ReactiveX/RxJava title, date, 01 of 10

  19. Messaging Protocol title, date, 01 of 10 Observer<T> Observer<T> Observable<T>

    Observer<T> Observable<T> onNext(T) onError(Throwable) React Deliver / Transform Emit
  20. Transformable Streams title, date, 01 of 10 Observer<T> Observable<T> Observer<R>

    onNext(T) Operator<T, R> onNext(R)
  21. Transformable Streams title, date, 01 of 10 https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables

  22. Example: Loading data Observable<Track> track(Urn trackUrn) { return trackFromStorage(trackUrn) .toList()

    .flatMap(syncIfEmpty(trackUrn)); } Observable<Track> fullTrack(Urn trackUrn) { return trackFromStorage(trackUrn) .zipWith(trackStorage.trackDetails(trackUrn), mergeFunction); } title, date, 01 of 10
  23. Schedulers track(trackUrn) .subscribeOn(Schedulers.newThread()) .observeOn(AndroidSchedulers.mainThread()) .subscribe(observer); title, date, 01 of 10

    Parameterized concurrency via schedulers
  24. Message Passing title, date, 01 of 10 interface EventBus {

    <T> Subscription subscribe(Queue<T> queue, Observer<T> observer); <T> void publish(Queue<T> queue, T event); <T> Subject<T, T> queue(Queue<T> queue); }
  25. Message Passing title, date, 01 of 10 Receiver @Inject EventBus

    eventBus; eventBus.subscribe(EventQueue.MY_QUEUE, new Subscriber() { public void onNext(MyEvent event) { … } }); Sender @Inject EventBus eventBus; eventBus.publish(EventQueue.MY_QUEUE, new MyEvent());
  26. propeller title, date, 01 of 10

  27. propeller title, date, 01 of 10

  28. Traditional Storage title, date, 01 of 10 Storage Logic Presentation

    Network class Track { String title User user } class TrackStorage { void insert(Track track) Track load(long trackId) } Track Track
  29. CQRS title, date, 01 of 10 ReadStorage Logic Presentation Network

    class ApiTrack { String title User user } class TrackWriteStorage { void insert(ApiTrack track) } class TrackReadStorage { PropertySet load(long id) } WriteStorage ApiTrack PropertySet
  30. Traditional Model title, date, 01 of 10 class Track {

    private Urn urn; private String title; private User user; }
  31. Traditional Model title, date, 01 of 10 class Track {

    private Urn urn; private String title; private User user; } Track track = new Track(); track.setUrn(new Urn(“soundcloud:tracks:1”)); track.setTitle(“title”); track.setUser(new User(...));
  32. Properties title, date, 01 of 10 interface Track { Property<Urn>

    URN = Property.of(Urn.class); Property<String> TITLE = Property.of(String.class); Property<Urn> CREATOR = User.URN; }
  33. Property Sets title, date, 01 of 10 interface Track {

    Property<Urn> URN = Property.of(Urn.class); Property<String> TITLE = Property.of(String.class); Property<Urn> CREATOR = User.URN; } PropertySet track = PropertySet.from( Track.URN.bind(new Urn(“soundcloud:tracks:1”)), Track.TITLE.bind(“track title”) Track.CREATOR.bind(new Urn(“soundcloud:users:2”)) );
  34. (Type) safe & composable title, date, 01 of 10 //

    will throw in if TITLE is missing String title = track.get(Track.TITLE); // returns ‘alt’ if TITLE is missing String title = track.getOrElse(Track.TITLE, “alt”); // returns null if TITLE is missing String title = track.getOrElseNull(Track.TITLE); // combines two sets PropertySet ps3 = ps1.merge(ps2);
  35. propeller title, date, 01 of 10 Design goals → Minimal

    bias → Symmetric push and pull → Uniform error handling
  36. Query title, date, 01 of 10 PropellerDatabase propeller = new

    PropellerDatabase(sqliteDatabase); Query q = Query.from(“tracks”) .select(“title”) .whereEq(“plays”, 42) .limit(1); QueryResult result = propeller.query(q);
  37. Result Iterators title, date, 01 of 10 for (CursorReader row

    : result) { String title = row.getString(“title”); ... }
  38. Row Mapping title, date, 01 of 10 mapper = new

    ResultMapper<Track> { @Override public Track map(CursorReader row) { return new Track( row.getString(“title”) ); } } List<Track> tracks = result.toList(mapper);
  39. Result Observables title, date, 01 of 10 DatabaseScheduler scheduler =

    new DatabaseScheduler(prop, sched); Observable<Track> tracks = scheduler.scheduleQuery(query).map(mapper);
  40. What else? title, date, 01 of 10 Read Joining, filtering,

    ordering, limiting, column functions Write (bulk)insert, (bulk)upsert, update, delete, truncate Transactions Integration test helpers
  41. Summary title, date, 01 of 10 Rx reduces complexity when

    composing data sources Rx only provides the cogs, YOU build the clockwork Don’t fall victim to MacGyver, always challenge your architecture!
  42. Thank you @mttkay mk@soundcloud.com

  43. Presentation: Fragments title, date, 01 of 10 class CommentsFragment extends

    DefaultFragment implements ReactiveListComponent<...> { @Inject CommentsOperations operations; @Inject EndlessAdapter adapter @Inject ListViewController controller; private ConnectableObservable<List<Comment>> comments; private Subscription sub = Subscriptions.empty() // life-cycle methods ... }
  44. Presentation: Fragments title, date, 01 of 10 public Observable<List<Comment>> buildObservable()

    { TrackUrn trackUrn = getArguments().getParcelable(...); this.comments = operations.comments(trackUrn) .map(toCommentViewModel) .observeOn(mainThread()) .replay(1); this.comments.subscribe(adapter); } public Subscription connectObservable() { this.sub = comments.connect(); return sub; }
  45. Presentation: Fragments title, date, 01 of 10 public void onViewCreated(View

    view, Bundle bundle) { super.onViewCreated(view, bundle); controller.connect(this, comments); } public void onDestroy() { this.sub.unsubscribe(); super.onDestroy(); }
  46. Practices title, date, 01 of 10

  47. Branch by Abstraction title, date, 01 of 10 <<new component>>

    branch <<old component>> master <<old component>> origin/master
  48. master Branch by Abstraction title, date, 01 of 10 abstraction

    layer http://martinfowler.com/bliki/BranchByAbstraction.html <<new component>> <<old component>>
  49. master Branch by Abstraction title, date, 01 of 10 abstraction

    layer http://martinfowler.com/bliki/BranchByAbstraction.html <<new component>>
  50. master Branch by Abstraction title, date, 01 of 10 http://martinfowler.com/bliki/BranchByAbstraction.html

    <<new component>>
  51. master Trunk Based Development title, date, 01 of 10 feature-B

    .isEnabled() .isEnabled() feature-A .isEnabled() feature-C
  52. Confident Code title, date, 01 of 10 http://en.wikipedia.org/wiki/Tony_Hoare

  53. Confident Code title, date, 01 of 10 http://remoteworker.files.wordpress.com/2010/11/rep3-100822-020. jpg?w=225&h=300

  54. Confident Code title, date, 01 of 10 Uses “null objects”

    or option types: Collections.emptyList() Optional.of()
  55. Confident Code title, date, 01 of 10 Follows a narrative

    structure that tells, not asks: 1. Gather inputs 2. Perform work 3. Deliver result 4. Handle failure