Evolving your software - The notion of design emergence

Instead of a big bang know-it-all upfront architectural design, which I have often come across and observed in a lot of software development companies, I propose to defer your architectural decision as late as possible and let those design choices "emerge" as you continue to evolve your product codebase.

Transcript

1. EVOLVING YOUR SOFTWARE THE NOTION OF DESIGN EMERGENCE
2. 2 Who Are we yeongsheng@odd-e.com 陈永胜 @yeongshen g yeongsheng-tan
3. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE “THE BEST ARCHITECTURES, REQUIREMENTS, AND DESIGNS EMERGE FROM SELF-ORGANIZING TEAMS.” – PRINCIPLES BEHIND THE AGILE MANIFESTO
4. AGENDA ‣ EMERGENT DESIGN ‣ CHOOSE YOUR POISON ‣ SWITCHING OUT AN ARCHITECTURAL LAYER ‣ WHEN RUBBER HITS THE ROAD ‣ SHOW ME THE CODE ‣ Q&A
5. EMERGENT DESIGN LISTEN, LEARN, UNDERSTAND, ADAPT
6. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE KNOWLEDGE CREATION & PRESERVATION KNOWLEDGE CREATION KNOWLEDGE PRESERVATION
7. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHEN DO YOU HAVE BEST KNOWLEDGE? KNOWLEDGE VS TIME Knowledge 0 25 50 75 100 Time AUG SEP OCT NOV DEC
8. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HYPOTHETICAL KNOWLEDGE VS TIME Knowledge 0 25 50 75 100 Time AUG SEP OCT NOV DEC
9. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE REALITY KNOWLEDGE VS TIME Knowledge 0 25 50 75 100 Time AUG SEP OCT NOV DEC
10. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT WE SHOULD STRIVE FOR KNOWLEDGE VS TIME Knowledge 0 25 50 75 100 Time AUG SEP OCT NOV DEC
11. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE Desig n (try to make all decisions) Implement the initial design Desig n (decide an initial direction) Design/Code Re f l ect an d Decide Direction Design/Code D esign/C ode Re f l ect an d Decide Direction TRADITIONAL EMERGENT TRADITIONAL VS EMERGENT
12. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MAKE IT WORK |> MAKE IT RIGHT |> MAKE IT FAST "BIG DESIGN UP FRONT IS DUMB, BUT DOING NO DESIGN UP FRONT IS EVEN DUMBER" THE PRIMITIVE WHOLE ISN'T JUST A BUNCH OF UNASSEMBLED PARTS, IT'S A "UNDER-DEVELOPED WHOLE" AND SOMETHING THAT DOES WORK
13. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE MOTIVATION -> SKILLS -> TIMING
14. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT UPFRONT DECISIONS ARE HARDEST TO CHANGE? ▸ Programming Language ▸ Operating System/Processor Architecture ▸ Database
15. CHOOSE YOUR POISON CHOICE - NOW OR LATER?
16. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HERE BE DRAGONS…
17. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE DEFER CHOICE, MAINTAIN FLEXIBILITY Technical Requirement Tech Stack A Tech Stack B HTTP Server Nginx/Apache2 + Phusion Passenger Erlang - Cowboy Request Processing Scala + Akka / Ruby on Rails Erlang - BEAM scheduler/Phoenix/raxx Long Running Requests Go Erlang - OTP/Task/ Agent Server-side State Aerospike/Redis Erlang - OTP/ets Data Persistence Aerospike + MongoDB/MySQL Erlang - Mnesia/dets Background Jobs Cron, Bash scripts, Java Quartz / Ruby + Resque/Sidekiq Erlang - Task/Agent Service crash recovery systemd/upstart Erlang - Supervisor
18. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE HOMOGENOUS -> SIMPLICITY ▸ Erlang/Elixir gives you options to implement initial solution exclusively homogenous Erlang stack. ▸ Switch to alternatives more full blown solution when enough knowledge has been acquired (e.g. through usage or after data characterisation has been conducted, etc). ▸ Homogeneous stack, single repo systems are easier to develop, test and maintain.
19. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE ▸ Phoenix.LiveView ▸ https://github.com/yeongsheng-tan/live_view_todos ▸ Branches ▸ master ▸ distributed_phx_pubsub RESPONSIVE INTERACTIVITY W/O JS
20. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT IS NEEDED? ▸ Production code -> ▸ Phoenix 1.4.9 and above ▸ hex package: phoenix.liveview ▸ Look Ma, No Javascript!!!
21. SWITCHING OUT AN ARCHITECTURAL LAYER D.A.S.H + NEWSQL
22. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE YOUR WEAKEST LINK ▸ Traditional RDBMS ▸ NoSQL ▸ NewSQL
23. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE D.A.S.H ▸ Beyond 12-Factor-App ▸ 4 properties of CloudNative Ready DB DISPOSABILITY API SYMMETRY SHARED NOTHING HORIZONTAL SCALING D : A : S : H :
24. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE
25. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE ENABLES GLOBAL, SCALABLE, RESILIENT SERVICES ▸ Low-touch and highly automated distributed DB for operators; Simple to reason about for developers. ▸ Industry-leading consistency, even on massively scaled deployments. Distributed transactions without the pain of eventual consistency issues. ▸ Always-on database that accepts reads and writes on all nodes without generating con f l icts. ▸ Flexible deployment in any environment, without tying you to any platform or vendor. ▸ Familiar tooling for working with relational data (i.e., SQL).
26. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE PERILS OF INCONSISTENCIES ▸ Emin Gün Sirer, an Associate Professor at Cornell University, wrote a blog post blaming eventually consistent data stores for the lost bitcoins. He mentions MongoDB, Cassandra and Riak among the NoSQL solutions that are vulnerable to banking thefts because:
27. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE CONSENSUS REPLICATION ▸ Raft ▸ Commit when quorum has written data (nodes >= 3) ▸ 1 consensus group per data range ibution s replication protocol (Raft) d transactions ent, repair, rebalance Gossip / Raft SQL API Distributed, transactional KV Node 1 Node 2 Node 3
28. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE @cockroachdb Put “cherry” Leader apricot banana blueberry cherry grape Follower apricot banana blueberry grape Follower apricot banana blueberry grape Consensus Replication Put “cherry” Leader apricot banana blueberry cherry grape Follower apricot banana blueberry grape Follower apricot banana blueberry grape Put “cherry” Consensus Replication RAFT CONSENSUS PROTOCOL
29. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE @cockroach Write committed when 2 out of 3 nodes have written data Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape Consensus Replication Put “cherry” Leader apricot banana blueberry cherry grape Put “cherry”
30. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE @cockroachdb Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape Consensus Replication Put “cherry” Leader apricot banana blueberry cherry grape Put “cherry” Ack Ack Follower apricot banana blueberry cherry grape Follower apricot banana blueberry grape Consensus Replication Put “cherry” Leader apricot banana blueberry cherry grape Put “cherry” cherry Ack
31. WHEN RUBBER HITS THE ROAD CAN IT BE TRUSTED?
32. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE COCKROACHDB PASSES JEPSEN TESTING ▸ CockroachDB Beta passes Jepsen testing ▸ Jepsen tests lessons 2+ years on... ▸ Register, Bank, Monotonic & Sets ▸ G2, Sequential, Comments & Nemesis modes (clock skews, network failures)
33. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE
34. DEMO
35. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE ▸ Elixir Ecto ▸ https://github.com/yeongsheng-tan/lunchbox_api.git ▸ Branches ▸ master ▸ cockroachdb_postgrex_cdb_api_jwt_auth_e2e_tests MONOLITHIC TO MULTI-MASTER DB
36. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE WHAT IS NEEDED? ▸ Production code -> hex package: postgrex_cdb ▸ Test code -> hex package: ecto_replay_sandbox ▸ No prod code changes!!! ▸ Test code changes.
37. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE “FROM A PRACTICAL POINT OF VIEW, CHANGEABILITY IS THE ONLY DESIGN METRIC THAT MATTERS; CODE THAT’S EASY TO CHANGE IS WELL-DESIGNED.” – SANDI METZ
38. APPLYING CAP TO BUILD DISTRIBUTED ROBUST PHOENIX API APP
39. EVOLVING YOUR SOFTWARE - THE NOTION OF DESIGN EMERGENCE CREDITS ▸ Nate Stewart ▸ Ben Darnell ▸ Aphyr Jepsen ▸ Diana Hsieh ▸ Emin Gün Sirer ▸ Saša Jurić