[2018.01 Meetup] [TALK] Katarzyna Kittel & Hugo Ferreira - How we built scalable and secure serverless PaaS

UX Forms is PaaS for easy authoring of complex web forms. The core of the platform was initially implemented as OSGi. As Kubernetes developed and matured, in mid-2017 we decided to re-architect UX Forms around Docker and Kubernetes instead of OSGi, making Kubernetes' dynamic service provisioning the core of UX Forms' offering.

During this talk we will show you how we created a platform able to deploy client compiled code smoothly and efficiently and run it with great scalability. We will cover topics like infrastructure as code, serverless application and security and explain what else we gained with this transformation.

Katarzyna Kittel finished her M.Eng in Informatics in 2007. Since that time she has been working as software engineer. Over the years, she has worked with many technologies like Python, Perl, Java but more than a year ago she bet on Scala. Currently she is software engineer at Equal Experts.

Hugo Ferreira is a software engineer with over 20 years of Information Technology practice and wide international experience in the European market, currently working at Equal Experts in product development, jumping back and forth between Scala and Kubernetes.

Transcript

1. How we built scalable and secure serverless PaaS
2. A form builder, with a different spin!
3. A form builder, with a different spin!
4. A form builder, with a different spin!
5. • Business logic ◦ Running and deploying binaries on the fly ◦ Hot-swap of implementation ◦ Isolation ◦ Scalability ◦ Service discovery • Easily manageable infrastructure Key challenges DEPLOYER F A B . . . A B A? B?
6. Running client binaries /root /var /bin … /opt openjdk /opt/lib/form-executor.jar /opt/conf/config /opt/bin/entrypoint.sh form-executor FROM openjdk:latest WORKDIR /opt/docker ADD opt /opt EXPOSE 9000 USER daemon /opt/lib/form.jar formA FROM form-executor:latest WORKDIR /opt/docker ADD form.jar lib ENTRYPOINT ["bin/entrypoint"] CMD [ "-cp", "/opt/docker/lib/form.jar"] Base Image Form Image
7. Deploying forms INGRESS NODE #2 NODE #1 FORM A FORM C FORM B MASTER D EPLO YM EN TS SERVICE B FORM B FORM A FORM A FORM C FORM C SERVICE C SERVICE A PO D S INGRESS PO D S
8. NODE UPLOAD form.jar DEPLOYER PULL BASE IMAGE CREATE FORM IMAGE PUSH FORM IMAGE DEPLOY PULL FORM IMAGE REGISTRY Deploying forms on-the-fly
9. Security aspects while deploying binaries • Runtime isolation (forms cannot interact with each other) • Resources isolation (memory, filesys, processes, network) • Misbehaved forms affect only that container
10. Infrastructure as code . ├── base/ ├── clusters/ │ ├── dev/ │ └── ... ├── extra/ │ ├── dev/ │ └── ... └── kube/ ├── dev/ └── ... ⬅ Terraform: Base storage resources for Kops & Terraform ⬅ Kops: Create Kubernetes cluster ⬅ Terraform: Other supporting resources ⬅ Kubernetes: Spec definitions for Kubernetes resources
11. modules.git ├── ... ├── kops_state_aws/ ├── tf_backend_aws/ ├── uxf_constants/ ├── ... └── setup-generate-docs.sh Terraform modules • Reusable and parameterizable definitions • All modules in the same repo (mono-repo) • Flat structure: top level folder = module name • Git tags: modulename-0.0 • Auto-generate all module READMEs ./setup-generate-docs.sh
12. Terraform modules
13. Terraform “inception” base/ ├── main.tf └── terraform.tfstate module "constants" { source = "git::ssh://git@bitbucket.org/uxforms/ uxforms-infra-modules.git// uxf_constants?ref=uxf_constants-3.0" } module "kops_state_store" { Source = "git::ssh://git@bitbucket.org/uxforms/ uxforms-infra-modules.git// kops_state_aws?ref=kops_state_aws-1.0" bucket_name = "kops.${module.constants.scope_domain}" } • Not environment specific • State file kept in the repo • Simple instantiation of Terraform modules, e.g:
14. clusters/ ├── README.md ├── dev │ └── cluster.yml ├── ... ├── run-install-addons.sh ├── run-update-cluster.sh └── setup-cluster-spec.sh Kubernetes Ops (kops) clusters • Part of the Kubernetes project • Instantiates all AWS infra for you • Installs / re-installs all Kubernetes software ./run-update-cluster.sh Enter the cluster environment (ex: dev): _ • Maybe not needed now that AWS announced Kubernetes as a Service
15. extra/ ├── default │ ├── kms.tf │ └── main.tf ├── dev │ ├── backend.tf │ ├── kops.tf │ └── main.tf ├── ... ├── setup-env-create.sh └── setup-kops-import.sh Terraform extra resources • Supporting infrastructure… • KMS definitions for controlling secrets’ access • RDS instance to store inflight form data • Needs to know the network definitions of Kops (should have be done differently)
16. Kubernetes workload kube/ ├── dev/ │ ├── deployer/ │ │ ├── configmap.yml │ │ ├── deployment.yml │ │ ├── hpa.yml │ │ ├── ingress.yml │ │ └── service.yml │ └── ... ├── test/ └── ... Image version updated by Jenkins job: DeployKubernetes ⬅ ecr.amazonaws.com/deployer: 1.1 ./run-deploy-services.sh <kubectl-cmd> <path> ./run-deploy-services.sh apply dev/* ./run-deploy-services.sh delete dev/deployer ./run-deploy-services.sh delete dev/deployer/hpa.yml
17. Protecting secrets kube/ ├── dev/ │ ├── deployer/ │ │ ├── secret.yml │ │ └── secret.yml.gpg │ └── ... ├── recipients.rcp : ├── kubeconfig/ │ ├── dev.yml │ ├── test.yml │ └── ... : .gitignore: kube/*secret.yml Public key fingerprints of the authorized developers .gitignore: kubeconfig/*.yml Team shared portable kubeconfig files, saved in encrypted bucket in AWS S3 and retrieved via its integration with AWS KMS: aws s3 cp s3://<src> <tgt> --sse-kms-key-id <id>
18. Thank You