Start a fullnode on GKE in <2 min - Alexander Peters, IOV

Start a fullnode on GKE in <2 min - Alexander Peters, IOV

Transcript

1. Start a full node on GKE in <2 min 2019 - Alexander Peters
2. About Me Alexander Peters Backend lead at IOV.one Go, K8s, DevOps and Security
3. Architecture V S S iNet S L B
4. Architecture V S S iNet S L B
5. Containers
6. Minimal Container Images • Scratch Image • Alpine Linux when OS required https://alpinelinux.org • Distroless base images https://github.com/GoogleContainerTools/distroless
7. State
8. State transfer • VolumeSnapshot Controller • VolumeSnapshot Provisioner > Disk snapshots #FTW https://github.com/kubernetes-incubator/external-storage
9. Best height? F F F F F C
10. Take snapshot F F F C V
11. CRDs: VolumeSnapshots + Data > kubectl get volumesnapshots NAME AGE godays-snap-1548763215780962127 17m godays-snap-1548763506627916883 12m godays-snap-1548763807378144367 7m apiVersion: volumesnapshot.external-storage.k8s.io/v1 kind: VolumeSnapshotData spec: gcePersistentDisk: snapshotId: pvc-d92e9529-2033-11e9-8eb6-42010a8400241548763507112695918
12. Let’s use it!
13. VolumeSnapshot Provisioning kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: snapshot-promoter provisioner: volumesnapshot.external-storage.k8s.io/snapshot-promoter reclaimPolicy: Delete volumeBindingMode: Immediate parameters: type: pd-ssd https://github.com/kubernetes-incubator/external-storage/blob/master/snapshot/doc/user-guide.md
14. Sentry - Full Node Config apiVersion: apps/v1 kind: StatefulSet spec: ... volumeClaimTemplates: spec: storageClassName: snapshot-promoter selector: matchLabels: group: "godays" matchExpressions: - {key: approved, operator: Exists}
15. Not a live demo
16. PVC - Provisioning > kubectl get pvc -w ... godays-fullnode-xrpc-0 Pending snapshot-promoter 19s godays-fullnode-xrpc-0 Pending snapshot-promoter 22s godays-fullnode-xrpc-0 Pending snapshot-promoter 24s godays-fullnode-xrpc-0 Pending snapshot-promoter 26s godays-fullnode-xrpc-0 Pending snapshot-promoter 28s godays-fullnode-xrpc-0 Pending snapshot-promoter 30s godays-fullnode-xrpc-0 Pending snapshot-promoter 32s godays-fullnode-xrpc-0 Pending pvc-23c64f9c-2308-11e9-8eb6-42010a840024 0 snapshot-promoter 34s godays-fullnode-xrpc-0 Bound pvc-23c64f9c-2308-11e9-8eb6-42010a840024 33Gi RWO snapshot-promoter 34s godays-fullnode-xrpc-0 Bound pvc-23c64f9c-2308-11e9-8eb6-42010a840024 33Gi RWO snapshot-promoter 34s
17. Pod - Deployment > kubectl get pods -w godays-fullnode-xrpc-0 0/3 Pending 0 0s godays-fullnode-xrpc-0 0/3 Pending 0 34s godays-fullnode-xrpc-0 0/3 Init:0/2 0 34s godays-fullnode-xrpc-0 0/3 Init:0/2 0 44s godays-fullnode-xrpc-0 0/3 Init:1/2 0 45s godays-fullnode-xrpc-0 0/3 PodInitializing 0 46s godays-fullnode-xrpc-0 2/3 Running 0 47s godays-fullnode-xrpc-0 3/3 Running 0 59s
18. • Rolling snapshots *) • PVC not deleted • Data integrity issues • Security concerns • High frequency Problems *) https://github.com/kubernetes-incubator/external-storage/issues/875
19. Summary • All good now! • Fast startup time < 2min • Works well with Horizontal Pod Autoscaler • Kubernetes v1.12 as an alpha feature
20. Thanks!
21. Bonus: Merkle tree https://en.wikipedia.org/wiki/Merkle_tree
22. Bonus: Working With Scratch Image • Go static compilation • COPY ca-bundle.pem /etc/ssl/certs/ca-certificates.crt https://curl.haxx.se/docs/caextract.html • ENV ZONEINFO /zoneinfo.zip COPY zoneinfo.zip / In $GOROOT/lib/time/zoneinfo.zip https://golang.org/pkg/time/#LoadLocation