- hosted microservices platform • • Designed as a backend integration layer Took architectural ownership mid flight • Seasoned in modernizing legacy code • Still surprised by what I found • Trained the team while fixing things. • Evolved far beyond its original scope • Multiple teams, many services, growing pains. The premise “Microservices multiply everything — good decisions and bad ones alike.” -
Services deployed separately… • …but tightly coupled in practice • All the complexity of microservices • None of the benefits. Architecture Modular Monolith First • Strong module boundaries inside one deploy • Easier to refactor and reason about • Extract services when you need independent scale • Conway's Law: org structure architecture. The rule “If you don't need independent scaling, you probably don't need microservices.” drives
Data Ownership • Each service owns its domain model • Every piece of data has one authoritative service • If two services call each other constantly… • Language is consistent within the boundary • No direct DB access across service boundaries • …they probably belong together • Map contexts explicitly at integration points • Read models can be replicated, not shared • High coupling is a decomposition failure • Don't share domain objects across services. • Ownership drives API contract design. • Merge first, re - split on a cleaner boundary. Chatty Services = Smell Key insight “Chatty inter - service calls are a design smell, not a network problem.”
(ADL) • Capture the 'why' behind design choices • ADRs survive team member turnover • Makes implicit knowledge explicit • Reduces repeated debates about past decisions. Team & Process Cognitive Load & Ownership • Too many services per team = burnout • Each team should fully own their services • Depending on a shared infra team = slowdown • On - call rotation improves quality your bugs. Process insight “If developers are paged for their own bugs, quality improves remarkably quickly.” — own
Standards • Private: between your own services • HTTP guidelines defined upfront — not per team • Internal: trusted external consumers • Commands: POST/PUT/DELETE (mutations) • Public: open to the world • Queries: GET (never mutate) • Each needs different versioning & SLA rules. • OpenAPI docs mandatory for every service. Contract rule “An undocumented API is a broken API waiting to happen.” API Design Contract Testing • Pact or similar for consumer driven contracts • Catches breaking changes before deployment • Versioning strategy agreed upfront • Don't discover breakage in production. -
• Serilog or Microsoft.Extensions.Logging • OpenTelemetry is now the .NET standard — add early • Correlation IDs across ALL service boundaries • Monitor rigorously transient • Log the right level ERROR • Set up alerts before you need them • • Distinguish business events from technical noise. Azure: return 204 (not 404) for not — prevents AppInsights flooding. — not everything is OpenTelemetry — fix every failure, even Hard lesson “Transient errors are not noise — they are early warnings of systemic failure.” - found
Async Messaging Polly or Microsoft.Extensions.Http.Resili ence (.NET 8+) • Azure Service Bus over synchronous HTTP where possible • Decouples producers from consumers • Retries with exponential backoff • Circuit breakers — fail fast, recover gracefully • Dead - letter queues need active monitoring • Timeouts everywhere trust the network. • Don't forget: what if a message arrives twice? — never Resilience rule “If you don't design for failure, you're designing for a bad day.” Resilience Distributed Transactions • Saga pattern for multi workflows - service • Understand idempotency build it in by default • Compensating transactions for rollback • Two - phase commit is almost never the answer —
What We Built • Unit tests testing implementation, not behaviour • No integration tests • No contract tests between services • Flaky pipelines accepted as normal • No smoke tests covering the full chain. — mocks everywhere • Unit tests: behaviour • Integration tests with Testcontainers dependencies • Contract tests (Pact) at every service boundary • Smoke tests: confirm the chain works end end • Azurite and in - memory emulators for Azure deps. Testing insight “A flaky pipeline is not a pipeline — it's a random number generator.” - focused, fast, no I/O — real - to-
Libraries Coding Guidelines Managing Debt • Seductive but introduce hidden coupling • Static analysis can't catch design decisions • Debt compounds faster in distributed systems • A shared lib change requires all services to update • Skills/guidelines docs align teams without forcing code • Prioritise by blast radius, not by age • Keep them thin — cross cutting concerns only • Don't strive for micro - level consistency across services. • Fix debt incrementally track it visibly • Prefer source - only NuGet packages for best practices. • Don't let broken windows accumulate. Warning “A shared library is a distributed monolith waiting to happen.” —
Discipline • Trunk - based development, short branches • Deployment Safe Release Strategies - lived • Feature flags: decouple deployment from release Never accept a flaky build or deployment pipeline • Blue/green or canary to reduce release risk • Separate repositories per service for independent deployment • Deploy to production quickly and often reduce batch size • Fix pipeline failures immediately block everyone. • Rollback must be fast and rehearsed, not theoretical. — they Deployment truth “If you're afraid to deploy, your deployment process is broken — not your code.” —
Code IaC Belongs with Code Drift is a Silent Killer Secret Management • Same repository as the service it provisions • Run what - if/preview regularly to detect drift • Azure Key Vault references — never hardcoded values • Reviewed in the same PR as application changes • Manual changes in the portal = invisible drift • Never put secrets in pipeline environment variables • Bicep for infra - team owned; Pulumi if devs maintain it • Treat drift like a failing test fix it immediately • Managed Identity over connection strings — always • Acceptance and production envs must mirror each other. • Automate drift detection in pipelines. • Rotate secrets regularly, automate where possible. — IaC discipline “If your infrastructure can't be reproduced from code, it doesn't really exist.”
Local Development Consistent Environments • .NET Aspire — orchestrate multiple services locally • Dev containers / devbox for team consistency • If spin - up takes > 10 minutes, people stop doing it • Eliminate 'works on my machine' permanently • Local env must mirror production as closely as possible • Onboarding should be: clone, open, run — nothing more • Testcontainers removes manual local DB setup. • Document the local setup, then automate it away. DX principle “If running the system locally is painful, developers will stop doing it.” - wide
Apps Azure Service Bus Identity & Networking • Functions: first - class ASB subscription support • Dead - letter queues need active monitoring • Managed Identity over connection strings — always • Web Apps: full ASP.NET Core power and middleware. • Poison messages silently stall processing • Private endpoints vs APIM: understand the trade - offs • Message TTL and lock duration need tuning • • Use App Configuration for centralized config. Return 204 (not 404) for not found — prevents AppInsights flooding. Azure hard lesson Dead - letter queues are where silent failures go to hide.
Otherwise, start with a modular monolith. 2 Chatty services are a design smell — reconsider the boundary, not the network. 3 Correlation IDs and structured logging are not optional in a distributed system. 4 Every HTTP call needs a retry policy, circuit breaker, and timeout. 5 Flaky pipelines are not pipelines. Fix or delete flaky tests. 6 Dead - letter queues need active monitoring — they're where silent failures hide. 7 IaC belongs in the same repo as the service it provisions. 8 Managed Identity over connection strings. Always. No exceptions. 9 Feature flags decouple deployment from release — use them. 10 On - call rotation changes quality behaviour more than any code review process.
dotnetday
daiz21
sansan33
masayoshi
soudai
oikon48
noriakioji
optim
ks91
kenichirokimura
minorun365
gurzu
aarron
mojombo
chriscoyier
bethany
tammyeverts
wjessup
cromwellryan
afnizarnur
skipperchong
katiecoart
aemeredith
