The Road to Continuous Deployments

The Road to Continuous
Deployments
Experience Report from CoLearn Engineering

View Slide

Swanand Pagnis
👨💼 CTO at CoLearn

🍻 meetup.com/Bangalore-Ruby-Users-Group/

📔 info.pagnis.in

👨🏫 postgres-workshop.com

View Slide

Background

View Slide

Company
•Education Platform for K
-
12

•In Indonesia (for now)

•Live Classes

•AI
-
Powered Homework Help

View Slide

>1 year ago
•8-month old codebases

•Service Oriented Architecture

•NodeJS backend
+
ReactJS Frontend

•Native Android in Kotlin

•Native iOS in Swift

View Slide

Now
•
~
2 year old codebases


•Rails/Django backend
+
ReactJS Frontend

•Native Android in Kotlin. Flutter WIP.

•Native iOS in Swift. Flutter WIP.

View Slide

Now
•
~
2 year old codebases


•Rails/Django backend
+
ReactJS Frontend

•Native Android in Kotlin. Flutter WIP.

•Native iOS in Swift. Flutter WIP.
Deprioritised, because 🚀 & 💰

View Slide

Legacy Systems
•Built for an MVP stage

•Came without thorough engineering
practices baked in

View Slide

Growth
•1 year period

•Engineering 10
->
36

•Product 2
->
9

•Design
+
Content 4
->
20

View Slide

Target Audience 🎯
•Product Engineering Teams

•Founders, CTOs, CPOs, VPs

•Software Developers

•Product Managers

View Slide

Why CD? 🤔

View Slide

What is this?

View Slide

Public Transport. ✅

View Slide

Every 1 Hour

View Slide

Every 5 Minutes

View Slide

Which is better? Every 1 Hour
Every 5 Minutes

View Slide

How about this?

View Slide

This belt is continuous.

Hop-on whenever.

Hop-off wherever.

View Slide

The Bottom Line
•Find and surface bugs faster

•Repeatable, reliable delivery

•Risk mitigation: "When the costs are
non-linear, keep it small"

View Slide

• Failure ☠ 🏦

View Slide

• Major Repairs 😱💰

View Slide

• Minor Repairs 😟 💵

View Slide

• Preventive Maintenance 😅 🪙

View Slide




• Preventive Maintenance 😅 🪙 ✅

View Slide

• Improved velocity 🏎

• Better product via rapid
iterations ♻

• Improved code quality,
reliability, architecture ☮

View Slide

What's the catch? 🎣

View Slide

• It needs rigour, which is not
always possible

View Slide

always possible
• High inertia — needs time,
effort, careful execution

View Slide

always possible
• High inertia — needs time,
effort, careful execution
• High short term costs

View Slide

Methodology

View Slide

Pairing, TDD, Trunk
Based Development,
On-Call Rotation:

Build Rigour

View Slide

Testing, Instrumentation,
Observability, Feature
Flags:

Make Verification Easy

View Slide

Infrastructure as Code,
Immutable Infra,
Pipelines, Playbooks:

Reduce Operating Friction

View Slide

A sustainable culture
of building & shipping
great products.

View Slide

1. Build Rigour

2. Make Verification Easy

3. Low Operating Friction

View Slide

1. Build Rigour

View Slide

1. Build Rigour
Putting the engineering in engineering

View Slide

1. Pair Programming

View Slide

1. Pair Programming
2. TDD

View Slide

1. Pair Programming
2. TDD
3. Trunk Based Development

View Slide

1. Pair Programming
2. TDD
4. On-Call Rotation

View Slide

1. Pair Programming

2. TDD


4. On-Call Rotation

View Slide

Why did we pick it?

View Slide

☝First, the results.

View Slide

Consistent high eNPS
•Min 73, Max 87

•Better connections, better work
relationships

•Pandemic induced remote anxiety went
down 📉

View Slide

Increasing Velocity
•Pairing velocity caught up with non-
pairing velocity

•Fewer delivery streams, same overall
speed

View Slide

Consistent

upward

trend.

Towards the end,

holidays and covid

knocked us down.

View Slide

Fast Onboarding
•Ship code to production in Week 1

•New languages, frameworks in a
sprint or two 🏎

•Internal transfers with zero friction 🧈

View Slide

Low Tech Debt
•Greenfield projects:
~
0 tech debt 👌

•Code quality up 📈

•Documentation quality and quantity up 📈

•Architecture has been flexible 💪

View Slide

Why did we pick it?

View Slide

A Combination Of
•Prior experience

•Established research

•First principles thinking

View Slide

What does research say?
•Improves design quality

•Reduces defects (people spend less time on defective
solutions)

•Reduces staffing risk

•Enhances technical skills

•Improves team communications

•Is considered more enjoyable at statistically significant levels.
The Costs and Benefits of Pair Programming; Alistair Cockburn, Laurie Williams, Feb 2000

View Slide

How to do it?

View Slide

Use Driver Navigator
•For an idea to go from Navigator's head to
the code, it must go through Driver's hands.

View Slide

•Switch roles periodically. Say, every hour.

View Slide

•Switch roles periodically. Say, every hour.
•Senior / Junior is immaterial. Both get both
roles.

View Slide

•Avoid giving line by line instructions,
convey the general idea.

View Slide

•Avoid giving line by line instructions,
convey the general idea.
•Seniors take the responsibility of
mentoring

View Slide

•When chopping onions, don't say
"cut the top off, now break in half,
make a slice" etc.

•Just say "finely chopped" or "diced"

View Slide

•Mentoring happens with both driving
and navigating.

•I leave it to you to figure out what are
the differences.

View Slide

•Remote pairing is better than in-person
because of the natural role selection

•Sharing screen? 👉 driver. Other 👉 navigator.

•Mobbing is incredibly easy in remote. Just
join the call and you're ready! 👍

View Slide

Switch Pairs Every Sprint
•Avoid pairing silos, they stall culture propagation

View Slide

•Often, a pair will be 💯, switch them anyway.

View Slide

•Exhaust senior-junior pairs first

View Slide

•Exhaust senior-junior pairs first
•When sprint ends, you swap even if WIP. This is
an effective litmus test.

View Slide

DevX is Crucial
•Let pairs figure out the balance between solo
focussed work and pairing

•Have routine health-checks about how
people are pairing, their experience, etc

•Let pairing feature in 1
:
1s and other
discussions

View Slide

Pilot + Co-Pilot

= Pairing

View Slide

You don't say

"Turn the lever by 10°
and push that button"

View Slide

You say

"Raise the elevation by
1000m"

View Slide

Pairing Summary
•Use Driver-Navigator

•Switch pairs every sprints; no silos

•Routine health checks with team

View Slide

1. Pair Programming

2. TDD


4. On-Call Rotation

View Slide

Why did we pick it?

View Slide

• TDD improves testability. This benefit
alone is enough to embrace TDD.

• TDD forces you to think in
specifications, hence improving
product thinking, along with code
quality.

View Slide

What are the effects?

View Slide

• Clear & significant uptick in quality
where TDD was followed vs where it
wasn't.

View Slide

wasn't.
• Legacy or greenfield doesn't matter

View Slide

wasn't.
• Legacy or greenfield doesn't matter
• TDD and Pairing are two incredible force
multipliers, they feed into each other
and create a strong positive gains loop.

View Slide

How to do it?

View Slide

View Slide

1. Have senior engineers who are
experienced in TDD

View Slide

1. Have senior engineers who are
experienced in TDD
2. Pair programming. Duh.

View Slide

1. Pair Programming

2. TDD


4. On-Call Rotation

View Slide

Why did we pick it?

View Slide

View Slide

• Impedance mismatch between long-
lived-branch
+
PR-based workflow
and how high-trust teams operate

View Slide

lived-branch
+
PR-based workflow
• Build a sense of ownership in the
codebase

View Slide

lived-branch
+
PR-based workflow
codebase
• Always be selling

View Slide

lived-branch
+
PR-based workflow

codebase

• Always be selling release ready

View Slide


View Slide

• Code reviews are faster

• Teams respond quicker to
urgent and important bugs

• We're running more iterations

View Slide

• Deploying to dev, stage has
become slightly awkward
because there's no 1
:
1 mapping

• Turn-key environments have
become a necessity rather than
nice-to-have

View Slide

How to do it?

View Slide

Have fast builds
•
<
1 min ideally, if possible

•15 min from git push to production
deploy including build

•Enable focussed tests i.e. run a single
test from a single file

View Slide

💯 Dev Machines
•Fast, capable laptops

•Must have automated & manual
testing setup

•Enable setting up any dependency

View Slide

1. Pair Programming

2. TDD


4. On-Call Rotation

View Slide

Observations?

View Slide

• Process still under iteration; no
"yes, this works" yet

View Slide

• Settled on functional rotations:
Backend, Frontend, Mobile, DevOps

View Slide

• Settled on functional rotations:
Backend, Frontend, Mobile, DevOps
• PTOs, leaves, Weekends still pose a
challenge from time to time

View Slide

• Team members that have done
really well during on-call have
also done really well in their
performance reviews.

• Correlation, yes. Causation? 🤷

View Slide

How to do it?

View Slide

Sharing our experience, not a
walkthrough for on-call.

Literature: PagerDuty Docs &
Google's SRE Book

View Slide

• Start with a robust triage process. First
response under 15 min.

• Have a playbook where common
problems and remedies are listed.

• In B2C products, a handful few situations
repeat like a persistent boomerang. FAB.
Frequently Annoying Bugs.

View Slide

• Use managed services as much as
possible; reduce operational on-call

• Try hard for "follow-the-sun"
model; i.e. no wee hours

• All alerts must be actionable, keep
adjusting until they are

View Slide

1. Pair Programming

2. TDD


4. On-Call Rotation

View Slide

Every single process /
methodology discussed so far
has ancillary benefits that go
way beyond just CD.

View Slide

2. Make
Verification Easy
How do you know what you've done is
working?

View Slide

View Slide

1. Testability

View Slide

1. Testability
2. Instrumentation

View Slide

1. Testability
2. Instrumentation
3. Feature Flags

View Slide

1. Testability
2. Instrumentation
3. Feature Flags
4. Static Verification

View Slide

1. Testability

2. Instrumentation

3. Feature Flags


View Slide

Why?

View Slide

• Testability is a core engineering
principle.

• To be able to answer questions
about a system by probing the
right points and looking at
indicators

View Slide

• Cars, bridges, rack & pinion —
you can't just restart them.

• Neither can you go and update
them at will

View Slide

Hit the hammer: $1.0

Knowing where to hit: $9999.0

View Slide

• The more testable your environment
is, the more people will actually test it.

• Make it easy to test something and it
will get tested.

• Conversely, make it difficult to test
and it's easy to slip.

View Slide

What are the
observations?

View Slide

• Not having Dev & Stage as close to
production has routinely caused problems

View Slide

• Static branches mapping to environments
(dev, stage, main) seem 👍, but are a 👎

View Slide

• Static branches mapping to environments
(dev, stage, main) seem 👍, but are a 👎
• Opaque 3rd party dependencies are
incredibly hard to test. e.g. WhatsApp
business APIs

View Slide

• SoA
+
inter-service dependencies =
complexity at a polynomial growth rate (or
worse, factorial)

View Slide

• SoA
+
worse, factorial)
• Cloud-Native systems are a pain to test,
but they do offer instrumentation.

View Slide

• SoA
+
worse, factorial)
• Cloud-Native systems are a pain to test,
but they do offer instrumentation.
• UIs are inherently hard to test, add probes
(
Metrics, Analytics, Traces, Errors, etc)

View Slide

So, how to go about it?

View Slide

There are two main
themes:

1. Development time

2. Runtime, in production

View Slide

Development Time
•Use TDD

•Add linters, code coverage to test builds

•Postman / equivalent API tools are 💯

•Powerful Type Systems*

View Slide

Runtime
• Make good use of lower order environments

View Slide

Runtime
•Heroku / Vercel style Review Apps are far
more powerful than they seem

View Slide

Runtime
•Heroku / Vercel style Review Apps are far
more powerful than they seem
•Dive down deep into important bugs and see
how they could've been tested earlier.
(
Which
is different from how to reproduce them)

View Slide

Runtime
•Add traces, specially to lower-order environments.
(
Example: AWS's X
-
Ray)

View Slide

Runtime
(
Example: AWS's X
-
Ray)
•Try and build idempotent units of work. APIs, Workers, etc.

View Slide

Runtime
(
Example: AWS's X
-
Ray)
•Pay special attention to non-idempotent units of work. Add
checks and balances.
(
Example: OTPs)

View Slide

Runtime
(
Example: AWS's X
-
Ray)
•Pay special attention to non-idempotent units of work. Add
checks and balances.
(
Example: OTPs)
•Eliminate String logs. All log statements are events, with
key value pairs*

View Slide

In both Environments
•Always test for contention:

• What must happen sequentially? Does it?

•Always test for coherence:

• How much and what information do two systems
need to collect from each other? Do they?

View Slide

Recommended Reading
•Neil Gunther's work on Universal
Scalability Law and Quantifying
Scalability and Performance

•Michael Nygard's "Release It!"

View Slide

1. Testability

2. Instrumentation

3. Feature Flags


View Slide

Why did we pick it?

View Slide

• Learning from other engineering
disciplines

• High velocity, but preferably not at
a very high upfront cost

• Wanted to build upfront, not after
the fact

View Slide


View Slide

• NewRelic routinely predicts a lot of problems
before they occur

• Tech spec quality has gone up — we add
metrics and dashboarding right into tech
specs

• Had our share of goof-ups. e.g. Shipped a
major feature, which nobody used in
production 🤦

View Slide

From our playbook

View Slide

• Number of bugs has gone down*

• Bug triage process is fast (and
getting faster; median first
response is down to 4 min)

• Consistently low tech debt; and we
assess and track regularly

View Slide

View Slide

• Number of bugs has gone down*

• Bug triage process is fast (and
getting faster; median first
response is down to 4 min)

• Consistently low tech debt; and we
assess and track regularly

View Slide

How to do it?

View Slide

• Have 3 levels of
instrumentation:

View Slide

instrumentation:
• Infra & Systems level

View Slide

instrumentation:
• Code & Application level

View Slide

instrumentation:
• Code & Application level
• Product & Business level

View Slide

• Have at least two kinds of
thresholds:

View Slide

thresholds:
• Too low and too high

View Slide

thresholds:
• Too low and too high
• Too long and too short

View Slide

• Envision your production dashboards
before even writing a single line of code

• We're running a trial with GQM
technique

• Answer the 🏅 question: How do you
know what you've built is working?

View Slide

• NewRelic, Cloudwatch & friends
are your friends

• Keep Logs, Metrics, APMs in
one place

View Slide

1. Testability

2. Instrumentation

3. Feature Flags


View Slide

Why?

View Slide

• Essential with Trunk Based
Development

• Works very well with product
experimentation

View Slide

Observations.

View Slide

• We now deploy "under development"
work to production on Day One

• Having fewer technologies has helped
in usage standardisation. Flipper is 🤘

• Code gets littered with branching.
Live with it.

View Slide

How to do it?

View Slide

3 Kinds of Feature Flags

View Slide

1. Infra / systems level (types of CPUs, Aurora
vs RDS, etc)

View Slide

vs RDS, etc)
2. Code level ( tied with continuous
deployments and trunk development )

View Slide

vs RDS, etc)
2. Code level ( tied with continuous
deployments and trunk development )
3. Product and business level
(
A/B tests,
experimentation )

View Slide

• Not all feature flags live forever, kill the
code branches when feature matures.

• Database changes have to be 100%
backward and forward compatible

• Prefer SDKs, libraries, code sharing over
a centralised service for feature flags

View Slide

1. Testability

2. Instrumentation

3. Feature Flags


View Slide

• Linters ✅

• Source Code Analysis for Security ✅

• Metrics ✅

• Exploring: TLA
+
🔮

• Formal Verification: At the moment 🛑

View Slide

1. Testability

2. Instrumentation

3. Feature Flags


View Slide

3. Reduce
Operating Friction
Help team focus on the important

View Slide

1. Pipelines

2. Infrastructure as Code

3. Playbooks

View Slide

• Automated pipelines ✅

• Manual deployment? 👎

• Manual approval? 👎

• Manual configuration? 👎

View Slide

• API service? Pipeline.

• iOS app? Pipeline.

• React App? Pipeline.

• Data pipeline? Well, duh!

View Slide

Make reliable
deployments a
foregone conclusion.

View Slide

What do we do?

View Slide

•
~
50 deployments per day

• Slowest deployment to prod is 15 min,
fastest is 3 min — this includes ALL THE
TESTING

• Deployments are completely transparent.
You push code and things happen. Teams
can focus on product and problems.

View Slide

• Entire infra is managed from the
pipeline, it's tied into the AWS
ecosystem.

View Slide

• Entire infra is managed from the
pipeline, it's tied into the AWS
ecosystem.
• Remember the golden rule: Every git
push goes to production under 15
minutes flat, with no manual approval
whatsoever.

View Slide

1. Pipelines


3. Playbooks

View Slide

Why did we pick it?

View Slide

• Declarative infrastructure; same code quality
focus on DevOps as well

View Slide

• We want the vertical teams to define and
manage their infra and not be blocked by a
horizontal team

View Slide

horizontal team
• Reduce operational on-call burden

View Slide

horizontal team
• Reduce operational on-call burden
• DevOps team works on hard platform problems
and security challenges

View Slide

• We picked: AWS CDK

• CDK Python makes it a low-
barrier for engineering.

View Slide


View Slide

Fast Turnaround
•New Rails project from scratch, goes from 0 to (dev
+ stage + production) in 2 hours.

View Slide

Fast Turnaround
•This includes Load balancer, DNS, HTTPS, Secrets,
Docker
(
Fargate) cluster, Redis, Workers, RDS
PostgreSQL, and all the things.

View Slide

Fast Turnaround
•This includes Load balancer, DNS, HTTPS, Secrets,
Docker
(
Fargate) cluster, Redis, Workers, RDS
PostgreSQL, and all the things.
•Out of this 2 hours, 45 min is taken by RDS to bring
up the server

View Slide

Fast Turnaround
•Adding a new AWS Lambda to dev + stage + prod:
15 to 30 minutes. Git push and you're in production.

•30-min when complex pieces like SQS / SNS are
involved

•New Redis server? Add code, git commit, 15 min
later: ✅

View Slide

• Infrastructure thinking and action is fully
absorbed into Engineering now.

• DevOps team has spent
<
1% of their total
time on on-call issues.

• They're working on pieces like turn-key
environments, load-testing setups, security
compliance, performance optimisations

View Slide

• Lower-order/sub-prime
environments are on a very high
parity with Production in terms
of infra.

• Remember better testing? This
makes it possible and easy.

View Slide

So, how do you do it?

View Slide

• Relentless focus on Developer
Productivity over infra costs.

View Slide

• Even in pure monetary terms, it's
cheaper

View Slide

• Even in pure monetary terms, it's
cheaper
• We routinely and constantly save costs
because developers have the headspace
to think about high impact problems.

View Slide

• Pick AWS CDK, Pick Cloud-
Native: The combination is
wildly effective.

• Similar combinations exist with
other providers

View Slide

• Treat infra team as an
engineering team, not a support
team.

• Actively help them avoid
becoming Jira card pushers

View Slide

1. Pipelines


3. Playbooks

View Slide

Playbooks for Nearly Everything
•Product Engineering? ✅

•Mobile Development? ✅

•Onboarding and Off-boarding? ✅

•Git Usage? ✅
(
WIP
)

•Feature Flags? ✅
(
WIP
)

View Slide

Templates for Nearly Everything
•Decision Records? ✅

•New code repositories? ✅

•PRDs? ✅

•Jira User Stories? ✅

•Interview Problems? ✅
(
WIP
)

View Slide

What is the idea?
•Reduce decision fatigue by codifying frequent
decisions.

•Improve compliance through written
procedures

•Encourage participation by making it open
and editable to all

View Slide

1. Pipelines


3. Playbooks

View Slide

Summary

View Slide

Rationale
1. Continuous Deployments are good for
you.

2. If you're not doing it, you're playing in hard
mode.

3. At minimum, think preventive maintenance

View Slide

Build Rigour:

Pairing, TDD, Trunk
Based Development,
On-Call Rotation

View Slide

Make Verification Easy:

Testing, Instrumentation,
Observability, Feature
Flags

View Slide

Reduce Operating Friction:

Infrastructure as Code,
Immutable Infra, Pipelines,
Playbooks

View Slide

A sustainable culture
of building & shipping
great products.

View Slide

Thank you! 🙏

View Slide

Questions?

View Slide

The Road to Continuous Deployments

The Road to Continuous Deployments

Swanand Pagnis

More Decks by Swanand Pagnis

Other Decks in Programming

Featured

Transcript