Reactive SoundCloud - Tackling complexity in large applications

In this talk I give some insights into how we rethought our app architecture from the ground up. I show where the complexity lies in our application and how we use the Reactive Extensions (Rx) to gain the upper hand back on dealing with the representation and composition of asynchronous data flows. I also introduce propeller, our storage library which fits snuggly into our reactive architecture.

Transcript

2. Reactive SoundCloud October 2014 Matthias Käppler Tackling complexity in large applications
3. Complexity title, date, 01 of 10
4. Essential Complexity vs. Accidental Complexity title, date, 01 of 10 No Silver Bullet — Essence and Accidents of Software Engineering, Fred Brooks, 1986
5. Essential Complexity title, date, 01 of 10 The complexity inherent to your problem domain.
6. Essential Complexity title, date, 01 of 10 Dealing with multiple asynchronous data sources in a fault tolerant manner → Propagation of change → Propagation of errors
7. Accidental Complexity title, date, 01 of 10 Self-inflicted complexity, e.g. due to lack of, misuse of, or overuse of abstractions.
8. Accidental Complexity title, date, 01 of 10 Unsuitable database abstraction Little support for handling async No standard event/error model Optimizing for the 1%
9. title, date, 01 of 10 “It’s the duty of the architect to solve the problems inherent in essential complexity without introducing accidental complexity.” -- Neil Ford
10. Architecture title, date, 01 of 10
11. The Ball of Mud title, date, 01 of 10
12. Layered Architecture title, date, 01 of 10
13. Featurized Architecture title, date, 01 of 10
14. Problem Statement title, date, 01 of 10 Introduce a uniform model for concurrency, compositional events and error handling.
15. Proposition title, date, 01 of 10 This can be accomplished using functional reactive programming.
16. Proposition title, date, 01 of 10 Sorry, I meant functional-ish reactive-ish programming. “For purity we are trying to avoid using the term FRP in the context of Rx” https://twitter. com/headinthebox/status/4925832274090721 28
17. Technologies title, date, 01 of 10
18. RxJava github.com/ReactiveX/RxJava title, date, 01 of 10
19. Messaging Protocol title, date, 01 of 10 Observer<T> Observer<T> Observable<T> Observer<T> Observable<T> onNext(T) onError(Throwable) React Deliver / Transform Emit
20. Transformable Streams title, date, 01 of 10 Observer<T> Observable<T> Observer<R> onNext(T) Operator<T, R> onNext(R)
21. Transformable Streams title, date, 01 of 10 https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables
22. Example: Loading data Observable<Track> track(Urn trackUrn) { return trackFromStorage(trackUrn) .toList() .flatMap(syncIfEmpty(trackUrn)); } Observable<Track> fullTrack(Urn trackUrn) { return trackFromStorage(trackUrn) .zipWith(trackStorage.trackDetails(trackUrn), mergeFunction); } title, date, 01 of 10
23. Schedulers track(trackUrn) .subscribeOn(Schedulers.newThread()) .observeOn(AndroidSchedulers.mainThread()) .subscribe(observer); title, date, 01 of 10 Parameterized concurrency via schedulers
24. Message Passing title, date, 01 of 10 interface EventBus { <T> Subscription subscribe(Queue<T> queue, Observer<T> observer); <T> void publish(Queue<T> queue, T event); <T> Subject<T, T> queue(Queue<T> queue); }
25. Message Passing title, date, 01 of 10 Receiver @Inject EventBus eventBus; eventBus.subscribe(EventQueue.MY_QUEUE, new Subscriber() { public void onNext(MyEvent event) { … } }); Sender @Inject EventBus eventBus; eventBus.publish(EventQueue.MY_QUEUE, new MyEvent());
26. propeller title, date, 01 of 10
27. propeller title, date, 01 of 10
28. Traditional Storage title, date, 01 of 10 Storage Logic Presentation Network class Track { String title User user } class TrackStorage { void insert(Track track) Track load(long trackId) } Track Track
29. CQRS title, date, 01 of 10 ReadStorage Logic Presentation Network class ApiTrack { String title User user } class TrackWriteStorage { void insert(ApiTrack track) } class TrackReadStorage { PropertySet load(long id) } WriteStorage ApiTrack PropertySet
30. Traditional Model title, date, 01 of 10 class Track { private Urn urn; private String title; private User user; }
31. Traditional Model title, date, 01 of 10 class Track { private Urn urn; private String title; private User user; } Track track = new Track(); track.setUrn(new Urn(“soundcloud:tracks:1”)); track.setTitle(“title”); track.setUser(new User(...));
32. Properties title, date, 01 of 10 interface Track { Property<Urn> URN = Property.of(Urn.class); Property<String> TITLE = Property.of(String.class); Property<Urn> CREATOR = User.URN; }
33. Property Sets title, date, 01 of 10 interface Track { Property<Urn> URN = Property.of(Urn.class); Property<String> TITLE = Property.of(String.class); Property<Urn> CREATOR = User.URN; } PropertySet track = PropertySet.from( Track.URN.bind(new Urn(“soundcloud:tracks:1”)), Track.TITLE.bind(“track title”) Track.CREATOR.bind(new Urn(“soundcloud:users:2”)) );
34. (Type) safe & composable title, date, 01 of 10 // will throw in if TITLE is missing String title = track.get(Track.TITLE); // returns ‘alt’ if TITLE is missing String title = track.getOrElse(Track.TITLE, “alt”); // returns null if TITLE is missing String title = track.getOrElseNull(Track.TITLE); // combines two sets PropertySet ps3 = ps1.merge(ps2);
35. propeller title, date, 01 of 10 Design goals → Minimal bias → Symmetric push and pull → Uniform error handling
36. Query title, date, 01 of 10 PropellerDatabase propeller = new PropellerDatabase(sqliteDatabase); Query q = Query.from(“tracks”) .select(“title”) .whereEq(“plays”, 42) .limit(1); QueryResult result = propeller.query(q);
37. Result Iterators title, date, 01 of 10 for (CursorReader row : result) { String title = row.getString(“title”); ... }
38. Row Mapping title, date, 01 of 10 mapper = new ResultMapper<Track> { @Override public Track map(CursorReader row) { return new Track( row.getString(“title”) ); } } List<Track> tracks = result.toList(mapper);
39. Result Observables title, date, 01 of 10 DatabaseScheduler scheduler = new DatabaseScheduler(prop, sched); Observable<Track> tracks = scheduler.scheduleQuery(query).map(mapper);
40. What else? title, date, 01 of 10 Read Joining, filtering, ordering, limiting, column functions Write (bulk)insert, (bulk)upsert, update, delete, truncate Transactions Integration test helpers
41. Summary title, date, 01 of 10 Rx reduces complexity when composing data sources Rx only provides the cogs, YOU build the clockwork Don’t fall victim to MacGyver, always challenge your architecture!
42. Thank you @mttkay mk@soundcloud.com
43. Presentation: Fragments title, date, 01 of 10 class CommentsFragment extends DefaultFragment implements ReactiveListComponent<...> { @Inject CommentsOperations operations; @Inject EndlessAdapter adapter @Inject ListViewController controller; private ConnectableObservable<List<Comment>> comments; private Subscription sub = Subscriptions.empty() // life-cycle methods ... }
44. Presentation: Fragments title, date, 01 of 10 public Observable<List<Comment>> buildObservable() { TrackUrn trackUrn = getArguments().getParcelable(...); this.comments = operations.comments(trackUrn) .map(toCommentViewModel) .observeOn(mainThread()) .replay(1); this.comments.subscribe(adapter); } public Subscription connectObservable() { this.sub = comments.connect(); return sub; }
45. Presentation: Fragments title, date, 01 of 10 public void onViewCreated(View view, Bundle bundle) { super.onViewCreated(view, bundle); controller.connect(this, comments); } public void onDestroy() { this.sub.unsubscribe(); super.onDestroy(); }
46. Practices title, date, 01 of 10
47. Branch by Abstraction title, date, 01 of 10 <<new component>> branch <<old component>> master <<old component>> origin/master
48. master Branch by Abstraction title, date, 01 of 10 abstraction layer http://martinfowler.com/bliki/BranchByAbstraction.html <<new component>> <<old component>>
49. master Branch by Abstraction title, date, 01 of 10 abstraction layer http://martinfowler.com/bliki/BranchByAbstraction.html <<new component>>
50. master Branch by Abstraction title, date, 01 of 10 http://martinfowler.com/bliki/BranchByAbstraction.html <<new component>>
51. master Trunk Based Development title, date, 01 of 10 feature-B .isEnabled() .isEnabled() feature-A .isEnabled() feature-C
52. Confident Code title, date, 01 of 10 http://en.wikipedia.org/wiki/Tony_Hoare
53. Confident Code title, date, 01 of 10 http://remoteworker.files.wordpress.com/2010/11/rep3-100822-020. jpg?w=225&h=300
54. Confident Code title, date, 01 of 10 Uses “null objects” or option types: Collections.emptyList() Optional.of()
55. Confident Code title, date, 01 of 10 Follows a narrative structure that tells, not asks: 1. Gather inputs 2. Perform work 3. Deliver result 4. Handle failure