Cloud migration: Lift-and-Shift? Don’t be naive!

I will talk about how we tried to move our products from Rackspace Cloud to AWS in the least disruptive way and about the hindrances and disillusionment we experienced trying the lift-and-shift strategy.

Transcript

1. Cloud migration Lift-and-Shift? Don’t be naive!
2. About me: • Tech lead @Attendify; • Worked on migration of one product from Rackspace to AWS, preparing to migrate another one; • Passionate about Cloud Native Apps based on various tech stack. 2
3. Agenda: 1. Intro ◦ Drivers ◦ Approaches ◦ Other considerations 2. Ballads ◦ Certified Consulting partners ◦ Terraform ◦ CI/Jenkins/Chef ◦ Docker ◦ API Gateway ◦ RDS 3. Product evolution 4. Summary 3
4. If I had a chance, I would do it all over again without hesitation Even if it’s Lift & Shift, it’s usually a very challenging task and will require a lot of creativity and effort. Just make sure you get the support of the management and the rest of the teams. 4
5. Main drivers of migration • Operational Resilience; • Business Agility; • Workforce Productivity; • OPEX; • Cost Avoidance(simplified patching, risk reduction, etc.); • Cost reduction; • Innovation and digital transformation; • Acquisitions and/or divestiture; • Data centre consolidation; • Large scale compute intensive workloads. 5
6. Main drivers of migration • Operational Resilience; • Business Agility; • Workforce Productivity; • OPEX; • Cost Avoidance(simplified patching, risk reduction, etc.); • Cost reduction; • Innovation and digital transformation; • Acquisitions and/or divestiture; • Data centre consolidation; • Large scale compute intensive workloads. 6
7. CapEx -> OpEx 7
8. Why everyone talks about it so much? 8
9. High-level migration process by AWS: https://aws.amazon.com/cloud-migration/ 9
10. Okay...where do we start? 10
11. Our version: Phase Description 0 Prep work See: OW Migration Pre-work 1/ 2 Migrate&Refac tor Hybrid mode. Migration as-is with minor changes where the cost of as-is migration is high, fix breakages, load testing, perf. testing, automated testing, monitoring, test cloud parity, data replication. 3 Re-architect IaaS / Target mode Scalability, agility, caching, de-coupling, optimisation. 4 Retire legacy / Continual improvement Decommission Rackspace. 11
12. Pre-work: Keep your inventory in order 12
13. Lift-and-shift is always about tradeoffs 13
14. Our tradeoffs • Everything running in containers from day one; • No Configuration Management(Chef); • Get rid of nginx on servers with monolith apps; • Some components will need to stay in the old cloud until fully substituted. 14
15. Team 15 *by Jimmy Nelson
16. Team setup options • Dedicated Infra team with help from Dev teams(“classic” DevOps); • Dedicated Dev team with help from Infra team. 16
17. Team setup options • Dedicated Infra team with help from Dev teams; • Dedicated Dev team with help from Infra team. 17
18. We researched different strategies for migration The Pre-work step is not any less important than migration itself because it’s quite likely that you won’t have time/budget for re-architecture right after the initial Lift&Shift. Make sure you are comfortable with all the tradeoffs and carefully weigh in all pros and cons comparing to the existing infra. 18
19. Our core principle for migration(even lift-and-shift) • What we move into AWS should not require substantial non-functional changes/refactoring in AWS unless dictated by new business requirements; • We don’t use AWS-native technologies we are not comfortable with even if it provides a temporary benefit(no snake oil principle). 19
20. Some do it in more granular way *Based on Car Rentals presentation 20
21. Some start with Beanstalk first • It’s a SaaS wrapper on top of other wrappers/services(ELB, S3); • Can only be a problem in the future if more flexibility is required. 21
22. Areas where the team had better spend extra time • IAM(resource-based policies, trust relationships, cross-organizations, cross-account access) • CloudWatch(if you need customization, percentiles, high resolution metrics) 22
23. No matter how well you’ve done the Pre-work you won’t be able to predict all the hindrances, and here are some that me stumbled upon. 23
24. The ballad of 24
25. Truth about AWS partners There’s not much to tell: get ready for serious engagement and investment if you want to get ANY help. 25
26. The ballad of 26
27. It’s IaaC but it’s not “Code” aaC nonetheless 27
28. Terraform is stateful and hence slow module.rds_dev_db.module.db_instance.aws_db_instance.t his: Creation complete after 9m4s (ID: ow-dev-int-rds) Your Devs might already have accommodated themselves to microservice architecture where builds are in a 30-60 seconds range. 28
29. Terraform is stateful and hence non-idempotent If something changed on AWS side, re-applying an exact same code can have a different result. 29 You all know who this guy is if you worked with Terraform
30. No meaningful test tools You can test the code you’ve written, but you would hardly test the effect it has on the state of you infrastructure. 30
31. Terraform and AWS “deadlocks” db_instance: You won’t be able to delete your instance if you haven’t specified the name of a snapshot. skip_final_snapshot = true - required 31
32. Terraform and AWS “deadlocks” resource "aws_autoscaling_group" "container_instance" { lifecycle { create_before_destroy = true prevent_destroy = true } .... } 32
33. Terraform and AWS “deadlocks” resource "aws_autoscaling_group" "container_instance" { lifecycle { create_before_destroy = true prevent_destroy = true } .... } https://docs.aws.amazon.com/autoscaling/ec2/userguide/cha nge-launch-config.html 33
34. Terraform integration between AWS and MySql providers provider "mysql" { endpoint = "${aws_db_instance.mysql_rds.endpoint}" username = "${aws_db_instance.mysql_rds.username}" password = "${aws_db_instance.mysql_rds.password}" } ## Won’t Work 34
35. Terraform integration between AWS and MySql providers 35
36. The ballad of 36
37. CI coexistence Jenkinsfile runPipeline('githubflow') { appName = 'oneweb-nurvis-microservice' platform = 'maven' role= 'nurvis' notificationRoom = 'Jedi Team' notificationEmail = 'jedi@thomascookonline.com' } Jenkinsfile.aws runPipeline('awscontainerflow') { platform = 'awsmaven' awsUseIAMRoleForTask = true } 37
38. 2 types of artifacts: • Old - jar files or nodejs modules; • New - docker image agnostic of the underlying stack 38
39. Technical debt in old Chef infra 39
40. Technical debt in old Chef infra 40
41. Smoke test and a problem with /status 41
42. Initially we wanted to use HTTP /status endpoints for all our services and for all usecases: • Jenkins smoke tests • Load-balancing health checks Status endpoints 42
43. Smoke test and a problem with /status Initial approach Final approach if (response.status == 200) { try{ statusResult = readJSON text: (response.content) if ( (!statusResult["app_name"].equals(appName)) || (!statusResult["build_version"].equals(buildVersion)) ) { echo "WARNING: Smoke test not passed, waiting " + smokeTestWaitingPeriod.div(1000) +" seconds before checking status again..." } else { smokeTestReady = true } } catch (error) { echo "ERROR: Something happened with parsing JSON response, please check: " + error.message smokeTestReady = false } } aws ecs wait services-stable --region ${AWS_REGION} \ --cluster ${PROJECT}-${env}-cluster --service ${PROJECT}-${env}-${SERVICE}-service" 43
44. Lots and lots of parameters to move to Param store For all our systems in old infra, we used file-based configs. They were either passed to the service as-is, or converted to env. Vars by Chef automatically. It allowed us to not bother too much about the amount of such vars since they were all in one file. With Parameter store the situation was very different since we provisioned every parameter one by one. 44
45. We also reviewed some conventions in the process 45 Java Nodejs Used a lot of params with dashes. Example: remote.service.web-api.timeout=<value-fro m-chef> It’s Ok for file-based configs, but not for env vars, so we switched to purely env.vars-based config: remote.service.web-api.timeout=${REMOTE _SERVICE_WEBAPI_TIMEOUT} There was no unified approach. Started using a lib for all projects.
46. The ballad of 46
47. Artifact size 47
48. Artifact size Some services had an admin panel attached to them(in the same repo) and the final artifact included a lot of libs(nodejs+angular) that were unnecessary in runtime. It was not so obvious and critical without Docker. With docker we started to pay attention to it and cleaned up the dependencies. 48
49. Multi-stage Dockerfile: be ready for some challenges Maven/Npm dependencies: With current approach, we are mounting host ~/.m2 and ~/.npm folders into a container. With multi-stage Dockerfile we are not able to mount, because docker doesn't support mounts for docker build command. Maven/Npm repositories credentials: Folders ~/.m2 and ~/.npm contain credentials to Nexus/Sinopia. 49
50. COPYing files Docker added a level of isolation we didn’t have before. As a result we initially missed a couple of things: • .npmrc with registry location was not present in many repos • .dockerignore was also missed - as a result local node_modules were copied into the image(but shrinkwrap wasn't) 50
51. JVM below 9-10 • No support for cgroups memory limits • No support for CPU sets https://developers.redhat.com/blog/2017/03/14/java-inside-docker/ https://jaxenter.com/nobody-puts-java-container-139373.html 51
52. The ballad of 52
53. API Gateway issue - access to private upstream While API Gateway was substantially improved over 2017 and it can now forward external traffic to internal services, it can only do so via NLB(TCP level) because it depends on VPC Link endpoints, which in turn depend on NLB. And our services are resolved via ALB and Route 53(HTTP/level 7). 53
54. API Gateway + Lambda Allows for super fast prototyping 54
55. The ballad of 55
56. Aurora supports autoscaling But how will you let your application know about it? 56
57. RDS doesn’t support proxysql or similar 57
58. Connector/j jdbc:mysql:replication://[master host][:port],[slave host 1][:port][,[slave host 2][:port]]...[/[database]] » [?propertyName1=propertyValue1[&propertyName2=propertyValue2]...] Connector/J supports replication-aware connections. It can automatically send queries to a read/write master, or a failover or round-robin loadbalanced set of slaves based on the state of Connection.getReadOnly(). 58
59. Spring Cloud AWS support 59
60. Spring Cloud AWS - coupling 60
61. We are not the first one ancounterring it 61
62. Partitioning/sharding - will most likely require external tools and extra infra https://www.citusdata.com/blog/2015/07/15/scaling-postgres-rds-with-pg-shard/ 62
63. Postgres 10 - key features 63
64. And RDS follows suit and adds support for it 64
65. But what if you want to have different tablespaces for different shards like so: CREATE TABLESPACE fastspace LOCATION '/ssd1/postgresql/data'; 65
66. Additional considerations 66 In our case, we were just one team that worked on migration itself, but there were 10 other teams which were actively adding up features to the product.
67. The migration will most likely be done while the product is evolving 67
68. Don’t expect to stay synchronous Your product will move on while you are migrating 68
69. Don’t expect to get rid of all apaches and nginx (ALB - only path and host based routing) 69
70. Summary Even lift-and-shift migration will be a lot of fun and adventures! 70
71. Thank you yura@attendify.com yura.v.nosenko@gmail.com https://www.linkedin.com/in/yuranos/ 71