You don't have a production environment

"We have a problem in production! does it reproduce in staging?" is a sentence often uttered in many companies. Underlying this simple sentence lie implicit assumptions about environments which often turn out to be false in our modern services and applications. Clusters and micro-services complicate matters and make production environments hard to reproduce and model, but when A/B tests, feature toggles and ephemeral containers, serverless functions and multi-tenancy are introduced one must conclude we no longer have a "production environment", but rather "production environments", plural. In a sense, every transaction experiences a unique unreproducible universe. In this world, what is the value of Staging environments? how does such understanding change our development processes?

Transcript

1. You don’t have a production environment Avishai Ish-Shalom (@nukemberg)
2. You don’t have a staging environment either
4. What is it good for? • Testing new subsystems • Reproducing issues • Testing artificial conditions Basically, a laboratory Realistic feedback, safely
5. What is it good for? • Testing new subsystems • Reproducing issues • Testing artificial conditions Basically, a laboratory Realistic feedback, safely
7. • Conceptually: a system, hosted within an environment • Replicate/move the system between environments
8. Assumptions 1. Clear boundary between system and environment 2. System can be separated from environment 3. System is deterministic 4. Boundary can be replicated 5. System state can be replicated
9. • O/S configurations • Topology configurations • API dependencies • Data dependencies Complete boundary spec? easy!
10. • The environment always leaks into the system • The more scale, the more leakage, data, state
11. Clusters • Not homogenous • Every host a little different • Hosts have state • Transactions can be serviced by different hosts
12. Clusters (you do deploy sometimes, right?) • Hosts can have different versions • Even with blue/green deployment • V1→V2→V4 != V2→V3→V4 != V4 • Golden images to the rescue? (nope)
13. Micro-services (AKA The Opsocalypse) • Different transaction touch different services • On different hosts • Possibly async • And everyone is deploying all the time
18. A/B tests • Different code paths • Possibly different services • Multiple concurrent experiments
19. Personalization Caches Monitoring Client versions DNS CDN That stupid Cron job SSL certificates Traffic
20. You don’t have a production environment You have production environments
21. Production environments (plural) • Transaction coordinate space practically infinite • Every transaction experiences a unique, transient universe • Impossible to replicate exactly • Impossible to isolate Is your Facebook experience the same as mine?
22. So now what?
23. What is our goal? To get (most) realistic feedback, safely
24. Nothing is more realistic than reality Test in production
25. Why isn’t your staging environment hosted on AWS staging cloud™?
26. Multitenancy Logical isolation of a group of users including their data, configuration, user management, functional experience and non-functional aspects
27. Multitenancy Single system, logical separation Multi-instances Duplicate the entire system
28. But how do I…. • Test new service ABC version 0.23? • Test existing service XYZ under load? • Reproduce a bug? • Run automated tests? Open a tenant, connect your stuff, run the test, remove the tenant.
29. OK, i’m sold. How do I get there? • Pass the tenant ID everywhere • Mark everything with the tenant ID • Partition all data by the tenant ID • Never allow cross tenant operations • Integrate observability tools to everything
30. Multitenancy is a core feature for SaaS/Cloud
31. Questions?