ArgoCD
Manatsawin Hanmongkolchai
Senior Architect, LINE MAN Wongnai
Slide 2
Slide 2 text
Pet vs Cattle
#connect people to good stuff
● Pet is when your servers have a name, and if it break all hell goes loose
● Cattle is when your servers are numbered, and if it break you kill it and
rebuild
● Kubernetes pod is a cattle, but what about your cluster?
Slide 3
Slide 3 text
Enforcing Infrastructure as Code
#connect people to good stuff
● How much do you think your current infrastructure is in Git?
Slide 4
Slide 4 text
Enforcing Infrastructure as Code
#connect people to good stuff
● Our answer is: we don’t even know!
● Have you ever kubectl edit -n wongnai-prod ?
○ Do you think your friends never do it?
Slide 5
Slide 5 text
Multiple deployment tools
#connect people to good stuff
● Some external software ships as Helm charts, so multiple sources of
deployment we have to manage
Slide 6
Slide 6 text
GitOps
Slide 7
Slide 7 text
GitOps
#connect people to good stuff
GitOps is a way to do Kubernetes cluster management and application
delivery. It works by using Git as a single source of truth for declarative
infrastructure and applications. With Git at the center of your delivery
pipelines, developers can make pull requests to accelerate and simplify
application deployments and operations tasks to Kubernetes.
Slide 8
Slide 8 text
GitOps
#connect people to good stuff
● All states are stored in Git repository - no more deploy time value
● All state changes are Git commit - can use Git tools to review/revert
and has audit trail
Slide 9
Slide 9 text
ArgoCD
#connect people to good stuff
● ArgoCD implements the GitOps pattern
● Multiple templating system support
○ Plain YAML file
○ Jsonnet
○ Helm v2 & v3
○ Kustomize
○ Plug your own with shell (We plug our previous system in)
Slide 10
Slide 10 text
ArgoCD
#connect people to good stuff
● ArgoCD has in-cluster daemon that read CRD, clone the Git repos,
template and apply
● All states are stored either in CRD or Git
● A nice web interface as a bonus
Slide 11
Slide 11 text
ArgoCD
#connect people to good stuff
Slide 12
Slide 12 text
ArgoCD @
Wongnai
Slide 13
Slide 13 text
Components
#connect people to good stuff
ArgoCD Jsonnet Argo Rollouts
Slide 14
Slide 14 text
Jsonnet
Slide 15
Slide 15 text
Jsonnet
#connect people to good stuff
Slide 16
Slide 16 text
Jsonnet
#connect people to good stuff
environment:
# load? overrides/${NAMESPACE}/env.yaml
environment: {
key: (
if namespace == 'wongnai-prod' then 'prod-value'
else 'other-value'
),
},
Slide 17
Slide 17 text
Jsonnet Merge
#connect people to good stuff
{
environment: {
key: 'VALUE',
}
} + {
environment: {
another: 'DATA'
}
}
{
environment: {
another: 'DATA',
}
}
Jsonnet Loop
#connect people to good stuff
{
["key" + i]: i
for i in [1, 2, 3, 4]
}
{
"key1": 1,
"key2": 2,
"key3": 3,
"key4": 4
}
Slide 20
Slide 20 text
Jsonnet Function
#connect people to good stuff
local kv(k, v) = {
name: key,
value: value,
};
{
environment: [
kv('key', 'VALUE'),
]
}
{
environment: [
{
name: 'key',
value: 'VALUE',
}
]
}
Slide 21
Slide 21 text
Jsonnet Reference
#connect people to good stuff
{
key: self.value
value: 'hello',
}
{
key: 'hello',
value: 'hello',
}
Slide 22
Slide 22 text
Jsonnet is Functional
#connect people to good stuff
{
key: self.value
} + {
value: 'hello'
}
{
key: 'hello',
value: 'hello',
}
Slide 23
Slide 23 text
Jsonnet Hidden Field
#connect people to good stuff
{
key: self.value
} + {
value:: 'hello'
}
{
key: 'hello',
}
Slide 24
Slide 24 text
Jsonnet Assertion
#connect people to good stuff
{
value: 10,
assert self.value < 5 :
"overflow"
}
RUNTIME ERROR: overflow
:3:27-37 thunk
During manifestation
Slide 25
Slide 25 text
Argo Rollouts
The “auto rollback”
Slide 26
Slide 26 text
Argo Rollouts
#connect people to good stuff
● Argo Rollout is in very early stage
● Rollout replaces Kubernetes Deployment
● Support multiple rollout strategy
Slide 27
Slide 27 text
Blue-green deployment
#connect people to good stuff
● Minimize the time that both versions are running together
Slide 28
Slide 28 text
Blue-green deployment
#connect people to good stuff
● Minimize the time that both versions are running together
Slide 29
Slide 29 text
Rolling deployment
#connect people to good stuff
● What Kubernetes Deployment does - existing deployment strategy
● Replace servers one by one
Slide 30
Slide 30 text
Canary deployment
#connect people to good stuff
● Replace a few servers, monitor, then roll forward
Slide 31
Slide 31 text
Canary analysis
#connect people to good stuff
● Argo Rollout use analysis to determine that the deployment is healthy
Slide 32
Slide 32 text
Canary analysis
#connect people to good stuff
● What can you analysis?
○ Evaluate Prometheus query result
○ Invoke Kubernetes Job
○ Send web request and evaluate JSON response
Slide 33
Slide 33 text
Canary analysis
#connect people to good stuff
● Analysis can run in the background during deployment, or at a given
step
● Can assert for success (error < 5%) or failure (error > 5%)
● If both success and failure are set, and none are met then it is
inconclusive
○ Inconclusive deployments are paused for human intervention
Slide 34
Slide 34 text
Canary deployment steps
#connect people to good stuff
● Canary deployment must have list of steps. Here’s how our deployment
works:
○ Rollout 10% of desired pods
○ Wait for 180 seconds (3 minutes)
○ Rollout 50% of desired pods
○ Wait for 120 seconds (2 minutes)
○ Rollout 100%
● If at any point background analysis fail, then rollback
Slide 35
Slide 35 text
Tales from production
#connect people to good stuff
● Argo Rollout saved a few botched deployments without operator
intervention
● But that requires it to be properly configured - can be annoying if not