Heterogeneous backend architecture at Ninchat

Presented at http://reaktordevday.fi/2013/

Transcript

1. Timo Savola Backend Lead, Ninchat Reaktor Dev Day 2013-10-04 Heterogeneous backend architecture at Ninchat
2. Copyright © Somia Reality Oy 2013. All Rights Reserved. • IRC • Connecting things • Need a solid platform Origins
3. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Generic messaging ◦ Text-based chat for humans ◦ Arbitrary message types for robots • Public and private channels • One-on-one conversations Features
4. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Guest users ◦ Temporary, anonymous chatters • Permanent users ◦ Verified user identity ◦ Create channels etc. Features
5. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Open API ◦ WebSocket or HTTP long-polling ◦ JSON headers ◦ Binary payloads ◦ https://ninchat.com/api • Client libraries ◦ https://github.com/ninchat/ninchat-python ◦ https://github.com/ninchat/ninchat-java Features
6. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Official clients ◦ https://ninchat.com ◦ Android app (beta) • Embedding for third-party sites ◦ Customer service ◦ Shoutbox Features
7. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Flexibility ◦ Business needs • Low maintenance overhead ◦ Automatic configuration ◦ Automatic failover Design goals
8. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Long client connections • Asynchronous event delivery ◦ One-to-many • Data model ◦ Object graph consistency ◦ Immutable content Implications
9. Copyright © Somia Reality Oy 2013. All Rights Reserved. Flow Actor Listeners
10. Copyright © Somia Reality Oy 2013. All Rights Reserved. Main components Logic workers Session nodes Connection servers
11. Copyright © Somia Reality Oy 2013. All Rights Reserved. Connection server • Manage client connections • Hide the complexity of HTTP long- polling • High concurrency • Stateless
12. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Implements all Ninchat features • Request/broadcast programming model • Database access • Fast development pace • Stateless Logic worker
13. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Live session registry • Buffer events in memory ◦ To be consumed by clients... ▪ ...with bad connections… • ...so keep them until acknowledged • Resembles Redis a bit • Minimal responsibilities • Stateful Session node
14. Copyright © Somia Reality Oy 2013. All Rights Reserved. Logic workers Connection servers Session nodes Codebase volatility
15. Copyright © Somia Reality Oy 2013. All Rights Reserved. Logic workers Connection servers Session nodes Centrality
16. Copyright © Somia Reality Oy 2013. All Rights Reserved. Logic workers Connection servers Session nodes Scaling needs
17. Copyright © Somia Reality Oy 2013. All Rights Reserved. Use the right tool... Python 70% C++ 8% Go 19%
18. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Goroutines for concurrency ◦ Segmented stacks • Communicate, don’t synchronize • Fresh take on interfaces ◦ Implicit satisfaction • Good WebSocket support early on • 2009, Google • http://golang.org Go for servers
19. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Rapid application development • Massive library support • Flexible language comes with high execution overhead Python for workers
20. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Predictable execution ◦ Performance ◦ Memory allocation • Single-threaded implementation ◦ Less bugs ◦ Concurrent updates would just contend • Less agile language C++ for sessions
21. Copyright © Somia Reality Oy 2013. All Rights Reserved. Type system spectrum Python • Dynamic typing Go • Static type inference • Compile time duck interfacing C++ • Static typing
22. Copyright © Somia Reality Oy 2013. All Rights Reserved. Language love Python • Context manager protocol Go • Deferred function calls C++ • RAII
23. Copyright © Somia Reality Oy 2013. All Rights Reserved. Communication patterns Untargeted, Immediate Targeted, Reliable Transactional Logic workers Session nodes Connection servers
24. Copyright © Somia Reality Oy 2013. All Rights Reserved. Communication solutions Offhand ZeroMQ ZeroMQ Logic workers Session nodes Connection servers
25. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Message queue library • No central broker required • High-level I/O patterns, including: ◦ Multi-sender pipeline ◦ Request/response multiplexing • Asynchronous API ◦ Simplifies application code • http://zeromq.org ZeroMQ
26. Copyright © Somia Reality Oy 2013. All Rights Reserved. • In-house protocol • One-way message passing • Load-balancing ◦ Untargeted • Aggressive failover • Inline health check ◦ Did the first database read work? • https://github.com/ninchat/offhand Offhand
27. Copyright © Somia Reality Oy 2013. All Rights Reserved. Reverse binding Clients Servers Connect Send requests
28. Copyright © Somia Reality Oy 2013. All Rights Reserved. Glue Offhand ZeroMQ ZeroMQ Python C++ Protocol Buffers 3% Go
29. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Object serialization format • Generates code from a “schema” • Same data object model in all languages • Extend with backward-compatibility • Google • https://code.google.com/p/protobuf Protocol Buffers
30. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Protocol Buffers definitions ◦ Compact ◦ Convenient • ZeroMQ pipeline ◦ No blocking Real-time metrics
31. Copyright © Somia Reality Oy 2013. All Rights Reserved. • Plug into DNS via dnsmasq • ZeroMQ broadcasting ◦ Edge-triggered state changes • Amazon S3 ◦ Highly available configuration storage ◦ Level-triggered state changes • https://github.com/ninchat/nameq Service discovery
32. Copyright © Somia Reality Oy 2013. All Rights Reserved. • AWS • PostgreSQL • Python 2.7 • Ubuntu LTS Trivia
33. timo@ninchat.com