Introduction to DevOps on AWS

Transcript

1. Introduction to DevOps on AWS Principles & Practices Ananda Dwi Rahmawati @misskecupbung
2. About Me • Student in Software Eng, Universitas Gadjah Mada, Jul 2019 – present • Cloud Engineer, PT Boer Technology (Btech), Jul 2019 – present • AWS Community Builder, Okt 2021 – present • Tech background: System, Networking, IaaS & PaaS cloud, Programming • Open Source Enthusiast • https://linktr.ee/misskecupbung
3. Outline • Chapter 1: What Is DevOps? • Chapter 2: Foundational Terminology and Concepts • Chapter 3: Past Development Ways • Chapter 4: Present Development Method • Chapter 5: CI/CD • Chapter 6: DevOps on AWS
4. References •Effective DevOps, Jennifer Davis & Ryn Daniels, O’Reilly Media, Inc., 2016 •Effective DevOps with AWS, Yogesh Raheja , Giuseppe Borgese & Nathaniel Felsen, PacktPub, 2018 •Implementing DevOps on AWS, Veselin Kantsev, PacktPub, 2017 •https://aws.amazon.com/devops/
5. What Is DevOps? Chapter 1
6. Organization Traditional Culture Let's think about our organization's business and culture and how technologies related to them.
7. What Is DevOps? • A cultural movement that changes how individuals think about their work, values the diversity of work done, supports intentional processes that accelerate the rate by which businesses realize value, and measures the effect of social and technical change. • A way of thinking and a way of working that enables individuals and organizations to develop and maintain sustainable work practices. • A cultural framework for sharing stories and developing empathy, enabling people and teams to practice their crafts in effective and lasting ways. • About finding ways to adapt and innovate social structure, culture, and technology together in order to work more effectively.
8. The Old View and the New View • The old view (blameful culture): human error as the cause of trouble. a mindset in which the focus is on elimination of human error. It assumes that errors are often caused by malice or incompetence. Individuals responsible for failure must be blamed and shamed (or simply fired). • The new view (blameless culture): human error as a symptom of trouble deeper in the system. a mindset that sees human errors as structural rather than personal. People make choices and take the actions based on their context and what makes most sense to them, not intentional malice or incompetence. Organizations should consider systems holistically when looking to minimize or respond to issues.
9. The New View is key to DevOps •Lead to increased transparency and trust within a team; •Instruct our coworkers in how to avoid a costly mistake without having to directly experience it; and •Increase the time spent on solving new problems, allowing for more innovation. The new view impact the industry as a whole, creating new opportunities, knowledge, and shared understanding.
10. The DevOps Compact Teams work together, communicate their intentions and the issues that they run into, and dynamically adjust in order to work toward their shared organizational goals. Compact principles: •Shared, clearly defined goals •Ongoing communication •Dynamic adjustment and repairs of understanding
11. Foundational Terminology and Concepts Chapter 2
12. Software Development Methodologies Different phases of development work may include: •Specification of deliverables or artifacts. •Development and verification of the code with respect to the specification. •Deployment of the code to its final customers or production environment. Few methodologies: Waterfall, Agile, Scrum
13. Waterfall The waterfall methodology is a project management process with an emphasis on a sequential progression from one stage of the process to the next. Originating in the manufacturing and construction industries and adopted later by hardware engineering, the waterfall model was adapted to software in the early 1980s.
14. Agile Agile is designed to be more lightweight and flexible than waterfall. The Agile Manifesto, written in 2001, outlines its main principles: •individuals and interactions over processes and tools •working software over comprehensive documentation •customer collaboration over contract negotiation •responding to change over following a plan DevOps adopts and extends Agile principles and applies them to the entire organization, not only the development process.
15. Scrum (1) Scrum focuses on maximizing a development team’s ability to quickly respond to changes in both project and customer requirements. It uses predefined development cycles called sprints, usually between one week and one month long, beginning with a sprint planning meeting to define goals and ending with a sprint review and sprint retrospective to discuss progress and any issues that arose during that sprint.
16. Scrum (2) Daily Scrum or daily standup, a daily meeting where team members (rather rapidly) each answer three questions: • What did I do yesterday that helped the team meet its sprint goals? • What am I planning to do today to help the team meet those goals? • What, if anything, do I see that is blocking either me or the team from reaching their goals? Daily Scrum often facilitated by the Scrum master that also responsible in helping the team self-organize and coordinate work efforts, helping remove blockers so the team will continue making progress, and involving project owners and stakeholders so there is a shared understanding of what “done” means and what progress is being made.
17. Development, Release, and Deployment Concepts •Version control •Test-Driven Development •Application Deployment •Continuous Integration (CI) •Continuous Delivery (CD) •Continuous Deployment (CD) •Minimum Viable Product (MVP)
18. Version Control • A version control system records changes to files or sets of files stored within the system. This can be source code, assets, and other documents that may be part of a software development project. Developers make changes in groups called commits or revisions. Each revision, along with meta data such as who made the change and when, is stored within the system in one way or another. • Having the ability to commit, compare, merge, and restore past revisions to objects to the repository allows for richer cooperation and collaboration within and between teams. It minimizes risks by establishing a way to revert objects in production to previous versions.
19. Test-Driven Development • The code developer starts by writing a failing test for the new code functionality, then writes the code itself, and finally ensures that the test passes when the code is complete. The test is a way of defining the new functionality clearly, making more explicit what the code should be doing. •Test-driven not only greatly shortens feedback loops but also encourages developers to take more responsibility for the quality of the code they are writing.
20. Application Deployment •Application deployment is the process of planning, maintaining, and executing on the delivery of a software release. •The craft of application deployment needs to consider the changes that are taking place underneath the system. Having infrastructure automation build the dependencies required to run a specific application minimizes the impact of inconsistencies on the released software.
21. Continuous Integration (CI) •Continuous integration (CI) is the process of integrating new code written by developers with a mainline or “master” branch frequently throughout the day. •With bigger changesets, it is much more difficult to isolate and identify what caused something to break. With small, frequently merged changesets, finding the specific change that caused a regression is much easier.
22. Continuous Delivery (CD) •Continuous delivery (CD) is a set of general software engineering principles that allow for frequent releases of new software through the use of automated testing and continuous integration. •Continuous delivery means that CI changes can be deployed.
23. Continuous Deployment (CD) •Continuous deployment (CD) is the process of deploying changes to production by defining tests and validations to minimize risk. •Continuous deployment means that new changes get deployed into production.
24. Minimum Viable Product (MVP) •The idea of the minimum viable product (MVP) is to create a prototype of a proposed product with the minimum amount of effort required to determine if the idea is a good one. •Rather than developing something to 100 percent completion before getting it into users’ hands, the MVP aims to drastically reduce that amount, so that if significant changes are needed, less time and effort has already been spent. •The MVP might mean cutting down on features or advanced settings in order to evaluate the core concept, or focusing on features rather than design or performance
25. Infrastructure Concepts •Configuration Management (CM) •Cloud Computing (Cloud) •Infrastructure Automation •Artifact Management •Containers
26. Configuration Management (CM) •Configuration management is the process of establishing and maintaining the consistency of something’s functional and physical attributes as well as performance throughout its lifecycle. •CM includes the policies, processes, documentation, and tools required to implement the system of consistent performance, functionality, and attributes. •CM is the process of identifying, managing, monitoring, and auditing a product through its entire lifecycle, including the processes, documentation, people, tools, software, and systems involved.
27. Cloud Computing (Cloud) The cloud refers to a type of shared, internet-based computing where customers can purchase and use shared computing resources offered by various cloud providers as needed.
28. Infrastructure Automation •Infrastructure automation is a way of creating systems that reduces the burden on people to manage the systems and their associated services, as well as increasing the quality, accuracy, and precision of a service to its consumers. •Automation in general is a way to cut down on repetitious work in order to minimize mistakes and save time and energy for human operators.
29. Artifact Management • An artifact is the output of any step in the software development process. Depending on the language, artifacts can be a number of things, including JARs (Java archive files), WARs (web application archive files), libraries, assets, and applications. • Artifact management can be as simple as a web server with access controls that allow file management internal to your environment, or it can be a more complex managed service with a variety of extended features. • Artifact repository can serve as: ◦ a central point for management of binaries and dependencies; ◦ a configurable proxy between organization and public repositories; and ◦ an integrated depot for build promotions of internally developed software.
30. Containers •Software containers are isolated structures that can be developed and deployed relatively independently from the underlying operating system or hardware. •Containers makes it easier for developers to develop an application in a container in their local environment and deploy that same container into production, minimizing risk and development overhead while also cutting down on the amount of deployment effort required of operations engineers.
31. Past Development Ways Chapter 3
32. Monolithic Applications A uniform, massive, redoubtable, or inflexible quality or character of applications.
33. Two Layers Monolithic Applications
34. One Pipeline per Application Most CI/CD pipelines and processes are built around a single application and in mos cases, the pipeline is very coupled to the specific needs of the application being deployed. They are custom built to do everything needed by the application and can include things like: • Building the application • Automated tests • Running security or static code analysis • Provisioning/configuring infrastructure • Publishing artifacts • Injecting configuration • Deploying the application If organization adding more application, we need to build and maintain multiple pipelines.
35. Good CI/CD Practice
36. Bad CI/CD Practice (1)
37. Bad CI/CD Practice (2)
38. Bypass Pipeline Stages •A CI/CD pipeline can do a lot of work to ensure the code being delivered is a quality product. •Teams may find themselves up against tight timelines and realize that their code coverage is beginning to slip. A request may come down to bypass the static code analysis stage. They may import a new code library that has a security vulnerability in it. Another request is received to bypass another stage. Soon you’re left with the absolute barebones pipeline.
39. Long Development Time of Monolithic Application
40. Present Development Method Chapter 4
41. Microservices • Microservices represent the small and lightweight services that usually provide specific and very limited functionality. This allows them to focus on singular tasks, which makes developing them fast, debugging issues easier, and deployments nimbler. A microservice can be deployed independently of other services and is self-contained. • Microservices can be deployed at different times. • Each microservice is in its own repository, so teams are safe to make changes as needed. • Teams will need to ensure they use proper API versioning and dependency management. • Microservices allow the type of rapid deployment and as the gateway to faster release cycles
42. Typical Microservices Architecture
43. Microservices App Example
44. Dedicated Pipeline per Language Benefits •Faster on-boarding: others can use pipeline if they write their applications in the same language. Teams can on-board to the pipeline faster and begin providing business value much sooner. •Reuse of code: Pipeline code is reused across multiple applications. •Ease of maintenance: If a new feature is required of the pipeline or you need to perform a bug fix, everyone gets those changes. Imagine you have an Angular 4 pipeline and Angular 5 is released. Simply update your Angular pipeline, and all UIs benefit at the same time.
45. Language-specific Pipelines Illustration
46. Further Step Pipeline Reuse (1)
47. Further Step Pipeline Reuse (2)
48. Two Weeks Release Cycle
49. CI/CD Chapter 5
50. Defining Continuous (1) Continuous mean “always ready to run” that includes serveral core concepts and best practices: • Frequent releases, enabling the delivery of quality software at frequent intervals. Frequency here is variable and can be defined by the team or company.Regardless of the frequency, the goal is the same: deliver software updates of high quality to end users in a repeatable, reliable process. • Automated processes, having automated processes to handle nearly all aspects of software production. This includes building, testing, analysis, versioning, and, in some cases, deployment. It may not be possible to automate everything, and some things, such as user acceptance, may need to remain manual or may seem too difficult to automate.
51. Defining Continuous (2) • Repeatability, the processes in pipelines can be versioned and re-created. • Fast processing. Regardless of the frequency of software updates/releases, continuous processes are expected to transform source code changes into updated deliverables in an efficient manner. • Fast problem detection and remediation. Pipeline’s jobs are to quickly process changes; to monitor the different tasks/jobs that create the release; to notify developers of failures as soon as possible since code that doesn’t compile, or fails a test, can keep the pipeline from proceeding.
52. Continuous Integration (CI) • In the CI phase, changes from a developer are merged and validated. The intended outcome is to identify any problems in the code and automatically notify the developer. This helps ensure that the code base is not broken any longer than necessary. •The CI process detects when code changes are made, and runs any associated build processes to prove the code changes are buildable. It can also run targeted testing (unit tests) to prove that the code changes work in isolation.
53. Unit Tests •Developers should be responsible for creating unit test. Developers know the contexts about what the individual blocks of code are supposed to be doing and can best align that with the test cases. In fact, in test-driven development (TDD), requires unit tests to be designed first before the code is written. •Various unit testing frameworks, such as JUnit for Java development, are available to assist with this.
54. CI Tools •After completing the code and validating it with local testing, the developer can then push their code and unit tests into the shared source code repository (such as Git). If the code is pushed and does not merge cleanly with other users’ changes, the developer must manually fix the conflicts and try again. •After the code is merged in, a CI tool such as Jenkins can detect or receive a notification that a change has been made in the source code repository. It will then automatically grab the latest set of code from the repository and attempt to build it and execute any associated tests.
55. Prechecks • Prechecks on the code before it makes it all the way into the source code repository. A simple form of prechecks might use hooks or triggers. • Hooks and triggers are processes that can be run before or after a particular source control operation to execute a process. • Users wanting to contribute changes to a hosted, shared source repository can “fork” the repository. They can then push candidate changes to their copy. To get it merged into the original repository, they can submit a pull request (merge request). This asks the owner of the original repository to review and merge in their code changes. Most prechecks also include a way to review and comment on proposed code changes. Once changes are approved and merged, they can kick off the Continuous Delivery (CD) processes.
56. Continuous Delivery (CD) •CD refers to the chain of processes (the pipeline) that automatically gets code changes and runs them through build, test, packaging, and related operations to produce a deployable release. Typically, CD operate without much or any human intervention. •CD takes the changes pulled in by CI and executes the rest of the pipeline processes to produce the deliverables. Optionally, it may trigger Continuous Deployment processes to make releases from the pipeline automatically available to users.
57. Artifacts An artifact is an item that is either a deliverable (something directly used by the final product) or included in a deliverable. For example, an executable file created by compiling source that links in several other libraries or a compressed file such as a WAR or ZIP file that contains another set of files within it.
58. Versioning •Artifacts as it proceeds throughout the pipeline, it should be versioned via a process like Semantic Versioning. The versioning of the artifact should be able to be traced back to the exact set of source code that was used to produce it. •Versioned artifacts can be managed via an artifact repository tool such as Artifactory.
59. Continuous Testing Continuous Testing refers to the practice of running automated tests or other types of analysis, of broadening scope as code goes through the Continuous Delivery pipeline. There are various forms of testing: • Unit testing is typically integrated with the build processes as part of the CI stage and focused on testing code in isolation from other code interacting with it. • Integration testing validates that groups of components and services all work together. • Functional testing validates that the result of executing functions in the product is as expected. • Acceptance testing measures some characteristic of the system against acceptable criteria. Examples include performance, scalability, stress, and capacity.
60. Coding Metrics and Analysis Coding metrics and analysis assess the quality of code and testing. And they can be used to gate (block/pass) code at various points in the pipeline. Examples: • Analyzing the amount of code covered by test cases. This metric is called code coverage and can be measured by tools (such as JaCoCo for Java code). • Counting lines of code, measuring complexity, and comparing coding structure and style against known best patterns can be done by tools such as SonarQube. Tools like this run checks, measure results against desired thresholds, block/allow further pipeline processing, and provide integrated reporting on the results.
61. Continuous Deployment (CD) • Continuous Deployment (CD) refers to being able to take a release of code that has come out of the delivery pipeline and automatically make it available for end users. •Deployment may mean automatically deploying in a cloud, making an update available, updating a website, or simply updating the list of available releases. •Whether or not a release from a pipeline run is deployed may be gated by human decisions. These can be based on various methods employed to “try out” a release before fully deploying it.
62. Blue/Green Testing and Deployments
63. Canary Testing and Deployments
64. Feature Toogles / Flags
65. Dark Launch
66. DevOps on AWS Chapter 6
67. Deploying ing AWS • AWS is at the forefront of cloud providers. Launched in 2006, with SQS and EC2, Amazon quickly became the biggest IaaS provider. • Using AWS's managed tools and services is a way to drastically improve your productivity and keep your team lean.
68. Why AWS for DevOps? •Get started fast •Fully managed services •Built for scale •Programmable •Automation •Secure •Large partner ecosystem •Pay-as-you-go
69. How to take advantage of the AWS ecosystem? •Amazon services are like Lego pieces. If you can picture your final product, then you can explore the different services and start combining them, in order to build the stack needed to quickly and efficiently build your product.
70. How does AWS synergize with a DevOps culture? •AWS helps to accomplish the synergize with a DevOps culture in many different ways •AWS offers a number of services that fit DevOps methodologies and will ultimately allow us to implement complex solutions in no time.
72. CI/CD Tools on AWS •AWS CodePipeline - Software Release Workflows •AWS CodeBuild - Build and Test Code •AWS CodeDeploy - Deployment Automation •AWS CodeStar - Unified CI/CD Projects
73. Microservices Tool on AWS •Amazon Elastic Container Service - Production Docker Platform •AWS Lambda - Serverless Computing
74. Infrastructure as Code on AWS •AWS CloudFormation - Templated Infrastructure Provisioning •AWS OpsWorks - Chef Configuration Management •AWS Systems Manager - Configuration Management •AWS Config - Policy as Code
75. Monitoring and Logging •Amazon CloudWatch - Cloud and Network Monitoring •AWS X-Ray - Distributed Tracing •AWS CloudTrail Activity & API Usage Tracking
76. Platform as a Service • AWS Elastic Beanstalk - Run and Manage Web Apps
77. Version Control •AWS CodeCommit - Private Git Hosting
78. Discussion