How to build & support high load REST API

I would like to tell the story of our experience of building the HighLoad REST API that has the following parameters:
* Intuitive
* Scalable
* Fault tolerant
* Protection from attack
* High bandwidth

Also I would like to describe:
* how we did LoadTesting
* how we survived a lots of DDoS attacks
* how we did deployment without downtime

The following list of technologies were used: .NET Core 2, ASP.NET Core 2, Consul, Fabio, Orleans.NET, Kafka, RabbitMQ, Serilog, xMetrics, Grafana, ELK

Transcript

1. How to build & support high load REST API Andrey Vinda
2. Betting revenue
3. Load statistics Active users 30 000 Passive users 5000 Logins 5000 (min)
4. Highload
5. Highload
6. About myself 15+ years of experience Many different products Like new technologies Launched several APIs
7. Business
8. Business
9. Users
10. Oops, something went wrong
11. Go to microservices
12. Business needs APIs
13. Login API Authenticate Fast response High throughput Resistance to DDoS and Brute Force attacks High Availability (99.99 %)
14. What we need Anti-robot defense Resistance to DDoS attack Quick response High throughput Scalable High availability
15. What next Defense High throughput High availability Observability
16. Defense Maintenance Robot detection Fraud detection Blocking
17. Protection & Security HOW TO IDENTIFY POSSIBLE FRAUD? HOW TO PROTECT FROM ROBOTS? WHAT TO ANALYZE? WHERE TO STORE DATA FOR ANALYZING?
18. Incapsula Cookie challenge JS cookie challenge
19. Defense. What to choose
20. Protection CAPTCHA BLOCKS
21. Captcha Send a CAPTCHA challenge, expecting a human response to the challenge LoginAPI generates captcha value for current session and stores it inside Orleans Use a non-white background with interspersed or roughened
22. Fraud detection Analyze IP addresses User-Agent Amount of failed login attempts per minute Amount of success login attempts per minute
23. Blocking Block players from the login base on the analysis of failed and success attempts per period
24. What next Defense High throughput High availability Observability
25. Quick response & High throughput Is captcha required? Whether login request is blocked? Quick search player by its login Login reports: success & failed Collect metrics
26. Standard building blocks
27. Application with stateless services
28. Data shipping paradigm
29. Stateless architecture: Pros Services are easy to scale No state in services When service server is dead – a new one could be up without any crucial affect
30. Stateless architecture: Cons Add latency Non efficient access to data Limits scalability • Master / slave replication • Sharding Concurrency
31. Stateless architecture: Cons Add latency Non efficient access to data Limits scalability • Master / slave replication • Sharding Concurrency Cache invalidation
32. What we want FAST ANALYZE OF INPUT REQUESTS IN-MEMORY STATISTICS OF SUCCESS / FAILED REQUESTS FAST AND SERIALIZABLE ACCESS TO THE CACHE DATA INVALIDATION EASY SCALING
33. Nice to have Distributed In-Memory cache Concurrency access Logic inside of the cache
34. Solutions
35. Actor model Actor can: • Receive messages • Make local decisions • Create more actors • Send more messages • Determine how to respond to the next message received
36. Orleans .NET Framework used to create Scalable, Distributed, .NET Applications Focused on low response latency and high concurrency Usable in any .NET application (but frequently used with WebAPI Applications) Based on a system of VirtualActors
37. Virtual actors Actor instances always exist, virtually • Application neither creates nor deletes them. They never fail. • Code can always call methods on an actor Activations are created on-demand • If there is no existing activation, a message sent to it triggers instantiation • Transparent recovery from server failures • Lifecycle is managed by the runtime • Runtime can create multiple activations of stateless actors (for performance) Location transparency • Actors can pass around references to one another or persist them • These are logical (virtual) references, always valid, not tied to a specific activation
38. Orleans: Grain = Virtual actor • Grains are C# classes Derived from GrainBase Implement an Interface (e.g. IPlayer, ISession) • Messages passing = Calling Interface methods Ex: AddLoginSession (int playerId, Guid sessionId) • Can be Stateless or Stateful
39. Orleans: Grains Grains: Individually isolated objects that are messaging
40. Orleans: Concurrency model for Grains Orleans uses a cooperative multithreading scheduler Scheduler schedules only one message at a time for a grain A message is processed completely before another message is scheduled A message is processed as a sequence of one or more turns (continuations)
41. Orleans cluster Transparent scalability by default New Silos can be added at any time Location transparency Multiplexed communication Efficient scheduling
42. Orleans grains Session Grain Player Grain Online Grain Operator Grain IP Grain Login Grain Cache layer Statistics layer Online layer Grain Cache Service Operator Cache Grain Player Cache Grain
43. Orleans usage Cache pre-population on startup Update cache on a regular basis (Kafka loader) In-memory statistics of all login attempts Regulation specific data is being stored in DB All logged in players are in memory Time regulation features are based on Timer
44. What next Defense High throughput High availability Observability
45. High availability Dedicated servers Scalable private cloud Scalable cloud Disaster Recovery
46. HA: Dedicated servers Round robin LB with Health Checks
47. HA: Cloud
48. HA: Disaster recovery
49. HA: How to achieve
50. HA: Our choice Dedicated servers with LB Private cloud (Kubernetes) Disaster recovery
51. HA: Main points All requests should be signed with JWT All secrets are stored in Vault Consul as a LB for every API host
52. HA: Communication schema Client Login API JWT Secret Credentials Anonymous JWT Credentials Anonymous JWT OK Auth JWT Availability status Health check 1 2 3
53. Blue/Green deployment
54. Blue/Green deployment Login API-A Login API-B Client Credentials * Cluster (default=A) * Cluster A Cluster B Availability status Health check
55. What next Defense High throughput High availability Observability
56. Observability Observability Logs Metrics Journaling Distributed tracing
57. Dashboard
58. Metrics Based on Kibana • Success / Failed responses • Failed reasons • Blocks statistics Based on InfluxDB/Graphana • Response time for each MS • Total requests for each MS • Successful logins per min • Unsuccessful logins per min
59. Metrics
60. Metrics
61. Metrics
62. Metrics
63. Metrics
64. Load testing
65. Fails Kafka consumer in the main thread Wrong configuration: connect App to Orleans Improper Captcha counters calculation
66. Summary It’s impossible to handle all requests Quotes granting system Reject requests to save the system Load testing Know your common scenarios Identify performance degradation Feature toggles It’s better to disable some feature instead of broke the whole system Aggressive caching mechanism System health Alerts to prevent real issue on PROD Metrics to identify weak places
67. Questions FB: https://www.facebook.com/andrey.vinda Email: vindaav@gmail.com Skype: vinda.andrew
68. Additional info https://github.com/wavchik/orleans-demo