Scalable Distributed Design - Let me show you my architectures.

Architects, those that design buildings, study architecture from the past to the present to best understand their craft. As software developers we aren't easily afforded this opportunity, this has always frustrated me, and has thus lead me to always want to share with my fellow craftsmen.

In this talk I lift the veil on a two of my enterprise projects implemented using a message based architecture. I walk through the architecture of the projects highlighting the SOA patterns used. The first architecture focuses on patterns employed for performance and scalability and the second focuses on big data principles for data visualisation using projections and the lambda architecture. This presentation also focuses on the do's and don'ts when designing a scalable message based architecture, as well as potential technology you may consider such as NServiceBus.

Transcript

1. 08-Mar-16 1 Chris Tite Scalable Distributed Design – Let me show you my architectures Crafting Code
2. 08-Mar-16 2 The Interlace
3. 08-Mar-16 3 Fallingwater
4. 08-Mar-16 4 Beer!
5. 08-Mar-16 5 The Problem
6. 08-Mar-16 6 Looking inside 25 individuals in 15 minutes -> 3 Months I have a hammer! (SSIS) • Falls over / Memory hungry / Slow to process (ws) One technology provider to rule them all! • Consider open source -> dockerize it Consider the data • Multiple sources / Highly connected Unit Testing / Performance Testing ?
7. 08-Mar-16 7 Introducing a bus for robustness Commands vs Events Request / Reaction not Request / Response Host Service Pattern Transactional Service Pattern! What about Testing!
8. 08-Mar-16 8 Parallel Pipes Pattern
9. 08-Mar-16 9 Applying a solution Apply a Reactive Pattern Dispatch on a shard point Separate response paths Coordinate responses Multiple Data Sources Distributor Pattern for Load Balancing Applied using NServiceBus Consider Akka Rolling your own is not advised!
10. 08-Mar-16 10 Thinking about the data • Sometimes relational isn’t the way • Always consider a shard point • Verification Type • Entity Type • Customer • Consider how data ages • Break data into small raw data points • Bring the data closer • Apply Active Service Pattern
11. 08-Mar-16 11 Active Service Pattern • Process all data locally • Pull data from remote point • Individually or by batch
12. 08-Mar-16 12 Burning Point!
13. 08-Mar-16 13 Aggregated Reporting Pattern • Break data in small raw datasets (events) • Log all raw data • Create projections off raw data • Batch / Fast / Serving • Single projection per report / view • Query of CQRS
14. 08-Mar-16 14 Final Solution
15. 08-Mar-16 15 Points to remember • Reactive systems help to scale • Request / Response pattern (ws) creates coupling • Transactions are your friends • Think about your data • Find suitable shard points • Denormalised projected data saves processing time • Active services reduce wait time • Always consider alternative technology • Open sources projects • Polyglot database solutions can work • Simplify endpoints • Easier to scale • Provides testable points • Simplifies performance testing
16. 08-Mar-16 16 /* THANK YOU*/ Chris Tite Kaleidocode Chris.tite@kaleidocode.co.za +27 83 256 2069 http://www.devconf.co.za/
17. 08-Mar-16 17 Consider Durban 7th on New York Times 52 places to visit in 2015