N things we learned building and deploying cloud applications

Presented at Redbrick DCU Tech Week, 19th Feb 2016

Transcript

1. ‘N’ things we learned about deploying cloud applications. Tony Ayres, School of Electronic Engineering @ DCU
2. What we build with Student information systems for academics and administrative staff Automated admin processes, statistics and graphs Development: Groovy, Grails, Java, Vertx, Spring Boot, Apache Camel, MySql, Oracle Deployment is AWS Elastic Beanstalk, RDS, S3, Route 53, EBS, Elastic Loadbalancer, CodeCommit, Email Service (SES), NAT Service, API Gateway
3. And so in conclusion... You WON’T listen to the advice in here. You WILL think - that wouldn’t have happened to me (You WILL be wrong) And you shouldn’t listen - go find things out for yourself. Don’t believe the tech hype Question everything
4. Keep it Simple Everyone wants to talk about scale Challenges of scale.. Problems we solved... Scaling languages, frameworks, architectures... I want to talk about The simple, stupid, mundane problems that will bite you (and cause you to have to work the weekend)
5. Building Applications is Hard Programming Databases Networks Security System Architecture
6. Stuff that can help Git Stash, branching, merging, rebasing, cherry-pick 3 Way Merge Tool - e.g. KDiff3 Don't be this guy -> Vi/Vim commands Learn the Linux command line grep, awk, sed, less, tail Shell scripting
7. Stuff that can help Develop code on the same OS family as it’s intended to run on Deploying Linux -> Develop on Linux (Mac is fine) Deploying on Windows -> Develop on Windows Otherwise you'll end up writing code like this if (OperatingSystem.current().isWindows()) { //lots of build script stuff } else { //lots more build script stuff } You’ll be dealing with ^M in your files Hacks to solve Windows path length limitations
8. Stuff that can help Learn to say ‘No’ Customers will ask for stuff they don’t need Project managers Can you lower your estimate? Can we skip testing? Can you work this weekend? (every weekend)
9. Stuff that can help Testing Who is responsible for testing your application? Answer - You Learn about Test Driven Development Learn about CI/CD
10. 1: Network Design Keep everything private Only expose port 80/443 Use NAT Bastion Host Firewall + MFA Hardened OS Development SSL by default LetsEncrypt - Free SSL Certs - No Excuses
11. 2: Simple Security Holes Configuration files with passwords in source control? dataSource: pooled: true jmxExport: true driverClassName: org.h2.Driver username: database_admin password: secret_123 dataSource: pooled: true jmxExport: true driverClassName: org.h2.Driver username: ${USER} password: ${PASSWORD}
12. 3: Configuration Cache Some values need to be customer configurable Schedules to run certain things at Page headings and text Email addresses to send error notifications to Customers need to be able to change them without downtime Read often - don’t want database hits! Can’t use environment variables or System properties Solution: Use Ehcache Library already in the project - via Hibernate
13. 3: Configuration Cache Cache cache = new Cache( new CacheConfiguration("systemCache", 10000) .memoryStoreEvictionPolicy(MemoryStoreEvictionPolicy.LFU) .eternal(false) .timeToLiveSeconds(86400)) cacheManager.addCache(cache) cache.put(new Element(someKey, configItem)) def myVal = cache.get(someKey) // Do something with myVal
14. 3: Configuration Cache Know your tools and libraries….RTFM Any dream library will do Don't guess! Know. Experiment Prototype Test Benchmark Be wary of vendor claims!
15. 4: Elastic Beanstalk ‘Gotchas’ If you use Tomcat - JVM monitoring isn’t there out of the box Flying blind Customise the image JMX and SSH Tunnels You can't turn off instances if you use autoscaling $ aws ec2 stop-instances --instance-ids i-de5b6d56 Customising with .ebextensions can be tricky YAML format is awkward
16. Battle 4: Elastic Beanstalk Gotchas Availability Zones and Beanstalk Wizards App Server (EC2) and Load Balancer in the same zone Can’t undo this later > Start Over! Solution: Use Cloud Formation Templates Tools to generate from existing infrastructure Temperamental Beanstalk Deployment / Rollback cycles
17. 4: Elastic Beanstalk Gotchas
18. 5 - S3 Storage and Integration Watch out for S3 storage costs Our CI build pipeline was storing our binary in S3 Beanstalk keeps copies of deployed apps on S3 also We stopped storing the binary artifacts on S3 Use Git tags Rebuild from source S3 is eventually consistent
19. 5 - S3 Storage and Integration ETL - Extract, Transform, Load Customer provides data in CSV - to put in our application Using Apache Camel: from(“aws:s3://my-bucket”) .process(new MyProcessor()) .to(“sql: insert into table values (:col1, :col2, col3)?dataSource=myDb”)
20. 5 - S3 Storage and Integration Great! Except it failed silently because of incorrect S3 bucket policy { "Version":"2012-10-17", "Statement":[ { "Sid":"AddPerm", "Effect":"Allow", "Principal": "*", "Action":["s3:GetObject"], "Resource":["arn:aws:s3::: examplebucket/*"] } ] }
21. 6: Logging Use the levels: DEBUG, INFO, ERROR, WARNING, TRACE Why? Too much logging can max your cpu and trigger autoscaling! Beanstalk “Freak Outs” Too little and you dont know whats going on! Customising log levels at runtime is a must - but more Beanstalk customisation! Example of how easy it is to do trivial things that cost you money! CPU logging causing an autoscale event
22. 7: Object Relational Mapping Beware of object relational mapping Understand what it’s doing in the database Know your database. Know SQL. (Don't use Mongodb - just because you don't like SQL) def departmentCode = “SC” Student.list().each { student -> def title = Department.findByDepartmentCode(departmentCode)?.title //do something cool with the Student and department title }
23. 7: ORM Perfect Storm What if you turned on logging for your ORM? What if your Student table has 1 million records? What if your Student/Department tables have no indexing? What if your both the Student and Department tables have 1 million records? What if you also had TRACE logging enabled?
24. Observation! Performance is important for you bank balance Different when you work for MegaCorp™ A faceless credit controller is paying for your servers! Technical choices and bad code cost money. Memory leaks CPU Usage Disk Usage
25. 8: Reduce Integration Points Hybrid Cloud Authentication - Active Directory/LDAP “On Prem” Databases SAML Authentication Difficult to test - but necessary All IDPs are not created equal!
26. 8: Reduce Integration Points ETL is a bit of a pain Requires software on customer end Scripts / Java Code Well tested - but temporary solution Nothing as permanent as a temporary solution Didn’t work when installed on customers server! Why?
27. 8: Reduce Integration Points Why? The customer “rewrote” the script Solution: Rest API Allow customers to push data to us! Easy...but wait!
28. Rest API, huh? Have you considered microservices
29. And inevitably... Something, something...Docker Blah, blah, blah...Kubernetes
30. I’m going to build microservices! Great! Now answer these questions: What is your API versioning strategy How will you do service discovery? What protocols will your services communicate over? How will you monitor services? How will you debug your system? How will you ensure loose coupling? How will you decide which service to put functionality in? How will you communicate a change to a service? Who owns and can make changes to a service? What is your database design strategy? Who owns the documentation? How will you integrate legacy services / systems?
31. Beware the hype cycle Don't accept claims without evidence Question things! Verify for yourself
32. 9: Yourself - Over Engineering We might need X in the future If you build X now you’re probably in over engineering territory AKA - YAGNI (You aint gonna need it) AKA - Scope Creep Speed and performance are relative You *probably* don't need 32 CPU cores for an app used by 100 people Old carpentry proverb: Measure twice, cut once
33. 9: Yourself - Accidental Complexity Example Grails 3.0 - application.yml and application.groovy and resources.groovy Multiple places for configuration System properties in Beanstalk Application configuration properties in the database Solution - Refactor Regularly (Btw - its not easy)
34. I leave you with this Learn Git Develop deep knowledge of tools/frameworks/programming languages Don’t believe the hype! Don’t accept claims without evidence! Including everything presented here! Try stuff out yourself Go build something great. Possibly even a disruptive mesh network with microservices and IOT Good Luck.