Utilize Kubernetes on OpenStack

Transcript

1. Utilize Kubernetes on OpenStack Cellular Genetics Informatics team (Sanger), Gene Expression team (EBI)
2. Deployment Production 4 m1.3xlarge nodes 54 vcpus, 464 Gb RAM Staging 4 m1.medium nodes 4 vcpus, 34 Gb RAM Development 4 o1.2xlarge nodes 26 vcpus, 28 Gb RAM utils https://github.com/cellgeni/kubespray
3. Applications Development: • Pipelines (Nextflow) • Web applications (Docker) One liners (using Helm): • Jupyterhub (notebooks) • Jenkins (CI) • Galaxy (EBI pipelines)
4. Nextflow Worfklow language/engine for data-driven computational pipelines. Java/Groovy • Sun Grid Engine (SGE) • AWS Batch • Load Sharing Facility (LSF) • SLURM (a soda in Futurama and an open source Linux job scheduler) • k8s (Kubernetes) • Local (ideal for testing) • PBS/Torque, NQSII, Ignite, GA4GH TES, HTCondor process { input: output: script: }
5. nextflow kuberun
6. Nextflow on Kubernetes (vs LSF) nextflow run cellgeni/rnaseq --samplefile $samplefile --studyid $sid --genome GRCh38 \ -profile farm3 nextflow run cellgeni/rnaseq --samplefile $samplefile --studyid $sid --genome GRCh38 \ -profile docker -c nf.config
7. -profile farm3 • executor { • name = 'lsf' • queueSize = 400 • perJobMemLimit = true • } • • process { • queue = 'normal' • } •
8. Shared config on all executors • process { • errorStrategy = 'ignore' • maxRetries = 2 • withName: irods { • memory = 2.GB • maxForks = 30 • } • withName: crams_to_fastq { • errorStrategy = { task.exitStatus == 130 ? 'retry' : 'ignore' } • cpus = 4 • memory = { 4.GB + 4.GB * (task.attempt-1) } • } • withName: star { • errorStrategy = { task.exitStatus == 130 ? 'retry' : 'ignore' } • cpus = { 8 * task.attempt, 'cpus' } • memory = { 40.GB * task.attempt * 1.6 ** (task.attempt - 1), 'memory' }
9. -profile docker • process { • withName: irods { • container = 'quay.io/cellgeni/irods' • pod = [secret: 'irods-secret', mountPath: '/secret'] • beforeScript = "/iinit.sh" • cpus = 4 • } • withName: crams_to_fastq { • container = 'quay.io/ biocontainers/samtools:1.8--4' • beforeScript = "export REF_PATH='http:://www.ebi.ac.uk/ena/cram/md5/%s'" • cpus = 4 • } • withName: star { • container = 'quay.io/biocontainers/star:2.5.4a--0' • cpus = 4 • } •
10. -c nf.config • process { • maxForks = 40 # adapt to our cluster size, fake batch system • afterScript = 'sleep 1' # deal with timestamps and caching • cache = 'lenient' # idem • } • • executor { queueSize = 40 } # adapt to cluster size • • k8s { • storageClaimName = 'nf-pvc' # gluster settings • storageMountPath = '/mnt/gluster’ # • } • •
11. Issues and notes • Samtools failure to populate reference sometimes. Still looking for solution, suggestions? • Batch processing; k8s and Nextflow don’t play nicely. The hot potato of ‘pending’? • So far we use GlusterFS rather than S3/NFS/lustre. • Benchmarking just started. Anecdotally: big scope for improvement. [E::cram_get_ref] Failed to populate reference for id 3 [E::cram_decode_slice] Unable to fetch reference #3 53251..1709029 [E::cram_next_slice] Failure to decode slice samtools merge: "23809_5#1.cram" is truncated
12. Kubernetes vs Farm (RNAseq pipeline) 2 times slower on Kubernetes, even with less steps! Farm - run on /lustre Kubernetes - run on GlusterFS
13. Kubernetes vs Farm (Memory Usage) Kubernetes Farm
14. Kubernetes vs Farm (Task execution in mins) Kubernetes Farm
15. Kubernetes vs Farm (Disk I/O - read) Kubernetes Farm
16. Kubernetes vs Farm (Disk I/O - write) Kubernetes Farm
17. Kubernetes vs Farm (CPU Usage) STAR is given 8 cpus and run with 8 threads STAR is given 4 cpus and run with 4 threads Kubernetes Farm
18. Kubernetes for web applications
19. K8s concepts • Pods • Replica sets • Deployments • Services • Persistent Volume Claim • Persistent Volume • Storage Class • Ingress controller • Helm
20. Request routing schema
21. Deployment flow 1. Dockerize application, upload image 2. Create manifests ▪ Deployment ▪ Service ▪ Persistent Volume Claims ▪ Persistent Volumes 3. Create Ingress record 4. Create Nginx record 5. Create Infoblox DNS record 6. [for public access] Web team approval
22. Example: partslab.sanger.ac.uk • Two independent services: ◦ FORECasT ◦ JACKS • FORECasT ◦ Front end ◦ Method back end • JACKS ◦ Front end ◦ Redis ◦ Celery
25. Third-party apps: JupyterHub • Interactive analysis with Python/R/Julia • RStudio Server on board • Ready installation: zero-to-jupyterhub (helm chart) ◦ helm upgrade --install jpt jupyterhub/jupyterhub --namespace jpt --version 0.7.0-beta.2 --values jupyter-config.yaml • Convenient storage management • Caveats ◦ TLS support didn’t work ◦ Can’t fine-grain resources
26. Jupyterhub
27. LIVE DEMO!!!1111
28. Jupyterhub
29. Jupyterhub
30. Jupyterhub
31. Jupyterhub
32. Jupyterhub
33. Jupyterhub
34. Jupyterhub
35. Jupyterhub
37. Pros and cons Pros: • Self-healing, rolling upgrades, health checks, auto scaling • Cloud provider independent • Resources efficiently utilized • Complex projects are easy to share and get running (via helm charts) • Vast community and knowledge base Cons: • Fresh cluster setup long and expertise demanding // we’ve been there for you, now you can run it on FCE in 20 min: https://github.com/cellgeni/kubespray • Significant learning curve • Limited SSL and Ingress management
38. Kubernetes future plans • Monitoring: ◦ Elasticsearch/Logstash/Kibana - logs ◦ Heapster/Grafana/InfluxDB - resources • Nextflow as a service - farm replacement for users • Heketi - better volument management for GlusterFS • GlusterFS alternatives ◦ CephFS ◦ NFS
39. Paolo Di Tommaso Pablo Moreno Phil Ewels Acknowledgments Helen Cousins Theo Barber-Bany Peter Clapham Tim Cutts Joe Garfoot James Smith Paul Bevan Nick Boughton