P1 Storage Event: A DevOps State of Mind

Getting to DevOps with Red Hat

Transcript

1. A DEVOPS STATE OF MIND Chris Van Tuin Chief Technologist, West cvantuin@redhat.com
2. Retail Finance Media Transportation ? ? SOFTWARE DISRUPTS BUSINESS
3. + ≠ TAXI TRANSPORTATION DISRUPTER https://goo.gl/MP7QQH Ack: Andrew Ng APP IT’S NOT JUST A… +
4. Culture of experimentation A B 20% vs. 25% Empowered organization Time Change Rapid Innovation THE DISRUPTORS = AI /
 ML Data-driven intelligence Data, Data, Data
5. IT MUST EVOLVE 
 FROM A COST CENTER TO INNOVATION CENTER Development Model Application Architecture Deployment & Application Infrastructur Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
8. THE SOFTWARE FACTORY + + DEV QA OPS Culture Process Technology
9. 24x Faster Recovery From Failure 3x Lower Change Failure Rate 2555x Shorter Lead Times https://puppet.com/resources/whitepaper/2016-state-of-devops-report 200x More Deployments THE IMPACT…
10. Self-Service, On-Demand, Elastic Infrastructure Automation Ansible, Kubernetes CI & CD Deployment Pipeline Advanced Deployment Techniques Microservices (and flying elephants!) Re-Org to DevOps THE EVOLUTION
11. RE-ORG TO DEVOPS
12. DEV QA OPS Walled off people, walled off processes, walled off technologies “THROW IT OVER THE WALL”
13. ORGANIZATION COMMUNICATION AND DECISION MAKING Decision
 Making Decision Making Siloed Open
14. DEV QA OPS Collaborative and empowered teams, 
 consistent processes, consistent technologies MOVE FAST, BREAK THINGS Line of Business Security , DBA, Network, etc.
15. SELF SERVICE, ON-DEMAND ELASTIC INFRASTRUCTURE
16. How many weeks do you wait for a new VM to be provisioned? Why do expensive resources 
 like developers wait so long for inexpensive resources like VMs?
17. CLOUDFORMS
18. 18 AN EVOLUTIONARY PATH TO HYBRID CLOUD CONTAINERS PRIVATE CLOUD PUBLIC CLOUD VIRTUALIZATION SOFTWARE DEFINED NETWORKING VMware© Microsoft© Hyper-V Red Hat Virtualization Amazon© Web Services Microsoft Azure Google© Cloud Platform Red Hat Openstack© Platform Red Hat© OpenShift Container Platform Service Management Compliance & Governance Efficiency & Optimization
19. AUTOMATION AND ORCHESTRATION
20. It takes weeks (or months!) to provision an environment The application behaves different in production than it did in test Environments are all manually configured 
 (“pets vs. cattle”) Production deployments have a very low success rate I get paged in the middle of the night because of production failures I don’t have an environment to develop/test against Production deployments require outage windows THE CHALLENGES
21. From development… …to production. COMMUNICATION IS THE KEY TO DEVOPS. 
 ANSIBLE PLAYBOOK DEV/TEST Q/A OPERATIONS MANAGEMENT OUTSOURCERS Automation Language Automation Engine DEV/TEST Q/A ANSIBLE
22. 4 • Are there known vulnerabilities in the application layer? • Are the runtime and OS layers up to date? • How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION CONTAINERS AYER MATTERS CONTAINER OS RUNTIME APPLICATION JAR CONTAINER
23. docker.io Registry Private Registry Red Hat Certified FROM fedora:latest CMD echo “Hello” Build file Physical, Virtual, Cloud Image Container Build Run Ship CONTAINERS: BUILD, SHIP, RUN
24. code config data TREAT CONTAINERS AS IMMUTABLE
25. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…
26. THE SOFTWARE FACTORY
27. OPENSHIFT TECHNICAL OVERVIEW CrunchyData GitLab Iron.io Couchbase Sonatype EnterpriseDB and many more 27 ...and virtually any docker image out there! TRUE POLYGLOT PLATFORM PHP Python Java NodeJS Perl Ruby .NET Core Apache
 HTTP Server MySQL Redis nginx Tomcat Varnish JBoss EAP JBoss A-MQ JBoss Fuse JBoss BRMS JBoss BPMS JBoss Data Grid JBoss Data Virt RH Mobile RH SSO 3SCALE API mgmt JBoss Web Server Spring
 Boot Wildfly Swarm Vert.x PostgreSQ L MongoDB Phusion Passenger Third-party
 Language Runtimes Third-party
 Databases Third-party
 App Runtimes Third-party
 Middleware Third-party
 Middleware
28. JSON Kubernetes CONTAINER APPLICATION ORCHESTRATION
29. OPENSHIFT TECHNICAL OVERVIEW 29 ! Persistent Volume ◦ Tied to a piece of network storage ◦ Provisioned by an administrator (static or dynamically) ◦ Allows admins to describe storage and users to request storage PERSISTENT STORAGE NFS GlusterFS OpenStack Cinder Ceph RBD AWS Elastic Block Store (EBS) GCE Persistent Disk iSCSI Fibre Channel
30. OPENSHIFT TECHNICAL OVERVIEW 30 DYNAMIC VOLUME PROVISIONING Admin User define StorageClass create claim: Fastest Slow
 Azure-Disk Fast
 AWS-SSD Fastest
 NetApp-Flash NetApp
 Provisioner AWS
 Provisioner Pod claim PV OpenShift PV Controller provision Azure
 Provisioner bound
31. OPENSHIFT TECHNICAL OVERVIEW NODE NODE NODE CONTAINER-NATIVE STORAGE 31 NODE POD POD POD POD POD POD POD POD POD POD RHGS RHGS RHGS POD POD POD MASTER
32. CI/CD PIPELINE WITH CONTAINERS
33. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
34. CI/CD WITH OPENSHIFT
35. Security CI/CD PIPELINES WITH SECURITY SCAN
36. OPENSHIFT TECHNICAL OVERVIEW 36 SECURITY & COMPLIANCE
37. ADVANCED DEPLOYMENT
38. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Blue / Green deployment • Rolling updates
39. Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2
40. Version 1 Version 1 V1.2 Health Check: Readiness 
 Probe e.g. tcp, http, script V1 ROLLING UPDATES with ZERO DOWNTIME
41. MICROSERVICES
42. ”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
43. CONTINUOUS FEEDBACK LOOP
44. A/B TESTING USING CANARY DEPLOYMENTS
45. 50% 50% Version 1.2 Version 1 Route Version 1.2 25% Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS
46. OPENSHIFT TECHNICAL OVERVIEW 46 SERVICE HEALTH
47. OPENSHIFT TECHNICAL OVERVIEW 47 OPERATIONAL EFFICIENCY
48. OPENSHIFT TECHNICAL OVERVIEW 48 FINANCIAL MANAGEMENT
49. RED HAT 3SCALE API MANAGEMENT
50. F4726-102516 IT’S ALL HERE
51. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin
52. http://www.cnbc.com/2014/06/04/15-years-to-extinction-sp-500-companies.html “40% of F500 on S&P will not exist in 10 years” - John M. Olin School of Business at Washington University “In short, software is eating the world.” - Marc Andreessen, Wall Street Journal, August 2011 “Only the paranoid survive” - Andy Grove, Intel MARKET INFLECTION POINT
53. UBER, LYFT FALLOUT
 TAXI RIDES PLUNGE 65% IN SAN FRANCISCO San Francisco’s largest cab company, Yellow Cab Co-Op, which filed for bankruptcy in 2016, was sold for $810,000 — less than it costs to buy a house in the City by the Bay.
54. $1.3M (2010), 
 $700K (2016) NOT JUST AN APP, BUT A 
 DYNAMIC PRICING STRATEGY Taxi Medallion Taxi Uber $0 $2.50/mile, $0.50/idle Fare Static Dynamic vs. vs. Ack: William Benton Uber fare 
 based on 
 real-time and historical data: supply/demand distance, time
 route, traffic etc.
55. NETWORKING
56. OPENSHIFT TECHNICAL OVERVIEW 56 ! Pluggable routing architecture ◦ HAProxy Router ◦ F5 Router ! Multiple-routers with traffic sharding ! Router supported protocols ◦ HTTP/HTTPS ◦ WebSockets ◦ TLS with SNI ! Non-standard ports via cloud load-balancers, ExternalIP, and NodePort ROUTING AND LOAD-BALANCING
57. OPENSHIFT TECHNICAL OVERVIEW 57 ROUTE SPLIT TRAFFIC SERVICE A App A App A SERVICE B App B App B ROUTE 10% traffic 90% traffic Split Traffic Between Multiple Services For A/B Testing, Blue/Green and Canary Deployments
58. OPENSHIFT TECHNICAL OVERVIEW NODE
 IP-3 NODE
 IP-2 NODE
 IP-1 58 EXTERNAL TRAFFIC WITH EXTERNALIP SERVICE EXT: IP-10:8080 INT: INT-IP:8080 EDGE ROUTERS IP-10, IP-11, IP-12 POD
 
 Port: 8080 POD Port: 8080 POD Port: 8080 IP FAILOVER
 POD IP FAILOVER
 POD connect 
 IP-10:8080 CLIENT ! Route external traffic to a service on any TCP/UDP port ! Available on non-cloud clusters ! External IP automatically assigned from a pre-defined pool of external IPs ! IP failover pods provide high availability for the pool of external IPs
59. OPENSHIFT TECHNICAL OVERVIEW ! NodePort exposes a unique port on all the nodes in the cluster ! Ports in 30K-60K range which usually differs from the service ! Traffic received on any node redirects to a node with the running service ! Firewall rules must allow traffic to all nodes on the specific port NODE
 IP-3 NODE
 IP-2 NODE
 IP-1 59 EXTERNAL TRAFFIC WITH NODEPORT SERVICE INTERNAL-IP:8080
 NODEPORT: 32010 POD
 
 Port: 8080 POD Port: 8080 POD Port: 8080 connect 
 IP-1:3201 0 CLIENT
60. OPENSHIFT TECHNICAL OVERVIEW 60 CONTROL SOURCE IP WITH EGRESS ROUTER NODE
 IP1 EGRESS ROUTER
 POD
 IP1
 EGRESS SERVICE
 INTERNAL-IP:8080 EXTERNAL 
 SERVICE
 Whitelist: IP1 POD POD POD
61. OPENSHIFT TECHNICAL OVERVIEW 61 ! Built-in internal DNS to reach services by name ! Split DNS is supported via SkyDNS ◦ Master answers DNS queries for internal services ◦ Other nameservers serve the rest of the queries ! Software Defined Networking (SDN) for a unified cluster network to enable pod-to-pod communication ! OpenShift follows Kubernetes network plug-in model ! Supported plug-ins ◦ OpenShift SDN (Open vSwitch or Flannel) ◦ Nuage SDN (Virtualized Services Platform) OPENSHIFT NETWORKING
62. OPENSHIFT TECHNICAL OVERVIEW NODE
 172.16.1.10 62 OPENSHIFT NETWORKING POD
 10.1.2.1 POD
 10.1.4.1 NODE
 172.16.1.20 POD
 10.1.2.2 POD
 10.1.4.2 Core Network VxLan Overlay Network
63. OPENSHIFT TECHNICAL OVERVIEW 63 FLAT NETWORK ! All pods can communicate with each other across projects MULTI-TENANT NETWORK ! Project-level network isolation ! Granular policies for network traffic ! Multicast support ! Egress network policies OPENSHIFT SDN NODE POD POD POD POD NODE POD POD POD POD PROJECT A PROJECT B DEFAULT NAMESPACE ✓ PROJECT C
64. OPENSHIFT TECHNICAL OVERVIEW Container to Container on the Same Host 64 OPENSHIFT SDN - OVS PACKET FLOW NODE POD 1 veth0
 10.1.15.2/24 br0
 10.1.15.1/24 192.168.0.100 eth0 POD 2 veth1
 10.1.15.3/24 vxlan0
65. OPENSHIFT TECHNICAL OVERVIEW NODE 2 NODE 1 65 OPENSHIFT SDN - OVS PACKET FLOW POD 1 veth0
 10.1.15.2/24 br0
 10.1.15.1/24 vxlan0 POD 2 veth0
 10.1.20.2/24 br0
 10.1.20.1/24 vxlan0 192.168.0.100 eth0 192.168.0.200 eth0 Container to Container on the Different Hosts
66. OPENSHIFT TECHNICAL OVERVIEW Container Connects to External Host Container to Container on Different Hosts 66 OPENSHIFT SDN - OVS PACKET FLOW NODE 1 POD 1 veth0
 10.1.15.2/24 br0
 10.1.15.1/24 tun0 192.168.0.100 External
 Host eth0
67. OPENSHIFT TECHNICAL OVERVIEW 67 OPENSHIFT SDN - FLANNEL PACKET FLOW NODE 1 POD 1 veth0
 10.1.15.2/24 docker0
 10.1.15.1/24 Routing Table flanneld NODE 2 POD 2 veth0
 10.1.20.2/24 docker0
 10.1.20.1/24 Routing Table flanneld etcd 192.168.0.100 eth0 192.168.0.200 eth0
68. LOGGING & METRICS
69. OPENSHIFT TECHNICAL OVERVIEW 69 ! EFK stack to aggregate logs for hosts and applications ◦ Elasticsearch: an object store to store all logs ◦ Fluentd: gathers logs and sends to Elasticsearch. ◦ Kibana: A web UI for Elasticsearch. ! Access control ◦ Cluster administrators can view all logs ◦ Developers can only view logs for their projects ! Ability to send logs elsewhere ◦ External elasticsearch, Splunk, etc CENTRAL LOG MANAGEMENT WITH EFK
70. OPENSHIFT TECHNICAL OVERVIEW CENTRAL LOG MANAGEMENT WITH EFK APPLICATION LOGS OPERATION LOGS ELASTIC ELASTIC 70 RHEL NODE POD POD POD POD FLUENTD RHEL NODE POD POD POD POD FLUENTD ELASTICSEARCH RHEL NODE POD POD POD POD FLUENTD USER ELASTIC ELASTIC KIBANA ELASTIC ELASTIC ELASTICSEARCH ELASTIC ELASTIC KIBANA ADMIN
71. OPENSHIFT TECHNICAL OVERVIEW 71 CONTAINER METRICS
72. OPENSHIFT TECHNICAL OVERVIEW CONTAINER METRICS 72 RHEL NODE POD POD POD POD FLUENTD CONTAINER METRICS RHEL NODE POD POD POD POD FLUENTD HEAPSTER RHEL NODE POD POD POD POD CADVISOR HAWKULAR OPENSHIFT WEB CONSOLE ELASTIC ELASTIC CASSANDRA RED HAT CLOUDFORMS CUSTOM DASHBOARDS API USER
73. SECURITY
74. OPENSHIFT TECHNICAL OVERVIEW 74 TEN LAYERS OF CONTAINER SECURITY Container Host & Multi-tenancy Container Content Container Registry Building Containers Deploying Container Container Platform Network Isolation Storage API Management Federated Clusters
75. OPENSHIFT TECHNICAL OVERVIEW NODE MASTER ! Secure mechanism for holding sensitive data e.g. ◦ Passwords and credentials ◦ SSH Keys ◦ Certificates ! Secrets are made available as ◦ Environment variables ◦ Volume mounts ◦ Interaction with external systems ! Encrypted in transit ! Never rest on the nodes 75 SECRET MANAGEMENT Container Distributed Store Container
76. PERSISTENT STORAGE
77. OPENSHIFT TECHNICAL OVERVIEW 77 ! Persistent Volume ◦ Tied to a piece of network storage ◦ Provisioned by an administrator (static or dynamically) ◦ Allows admins to describe storage and users to request storage PERSISTENT STORAGE NFS GlusterFS OpenStack Cinder Ceph RBD AWS Elastic Block Store (EBS) GCE Persistent Disk iSCSI Fibre Channel
78. OPENSHIFT TECHNICAL OVERVIEW PROJECT POOL OF PERSISTENT VOLUMES 78 PERSISTENT STORAGE NFS PV iSCSI
 PV NFS PV Admin User register PV create claim NFS PV GlusterFS PV Pod claim Pod claim Pod claim Ceph
 RBD
 PV
79. OPENSHIFT TECHNICAL OVERVIEW 79 DYNAMIC VOLUME PROVISIONING Admin User define StorageClass create claim: Fastest Slow
 Azure-Disk Fast
 AWS-SSD Fastest
 NetApp-Flash NetApp
 Provisioner AWS
 Provisioner Pod claim PV OpenShift PV Controller provision Azure
 Provisioner bound
80. OPENSHIFT TECHNICAL OVERVIEW 80 ! Containerized Red Hat Gluster Storage ! Native integration with OpenShift ! Unified Orchestration using Kubernetes for applications and storage ! Greater control & ease of use for developers ! Lower TCO through convergence ! Single vendor Support DISTRIBUTED, SECURE, SCALE-OUT STORAGE CLUSTER APPLICATION CONTAINER APPLICATION CONTAINER APPLICATION CONTAINER STORAGE CONTAINER STORAGE CONTAINER STORAGE CONTAINER CONTAINER-NATIVE STORAGE
81. OPENSHIFT TECHNICAL OVERVIEW NODE NODE NODE CONTAINER-NATIVE STORAGE 81 NODE POD POD POD POD POD POD POD POD POD POD RHGS RHGS RHGS POD POD POD MASTER
82. OPERATIONAL 
 MANAGEMENT
83. OPENSHIFT TECHNICAL OVERVIEW 83 TOP CHALLENGES OF RUNNING CONTAINERS AT SCALE SERVICE
 HEALTH SECURITY & COMPLIANCE FINANCIAL MANAGEMENT OPERATIONAL EFFICIENCY
84. OPENSHIFT TECHNICAL OVERVIEW 84 Operational Management Across the Stack ! Real-time discovery ! Visualize relationships ! Monitoring and alerts ! Vulnerability scanning ! Security compliance ! Workflow and policy ! Automation ! Chargeback
85. OPENSHIFT TECHNICAL OVERVIEW ! CloudForms continuously discovers your infrastructure in near real time. ! CloudForms discovers and visualizes relationships between infra components ! CloudForms cross references inventory across technologies. ! CloudForms offers custom automation via control policy or UI extensions 85 OPERATIONAL EFFICIENCY
86. OPENSHIFT TECHNICAL OVERVIEW 86 OPERATIONAL EFFICIENCY
87. OPENSHIFT TECHNICAL OVERVIEW ! CloudForms monitors resource consumption and shows trends ! CloudForms alerts on performance thresholds or other events ! CloudForms offers right-sizing recommendations ! CloudForms enforces configuration and tracks it over time. 87 SERVICE HEALTH
88. OPENSHIFT TECHNICAL OVERVIEW 88 SERVICE HEALTH
89. OPENSHIFT TECHNICAL OVERVIEW ! CloudForms finds and marks nodes non- compliant with policy. ! CloudForms allows reporting on container provenance. ! CloudForms scans Containers Images using OpenSCAP. ! CloudForms tracks genealogy between images and containers. 89 SECURITY & COMPLIANCE
90. OPENSHIFT TECHNICAL OVERVIEW 90 SECURITY & COMPLIANCE
91. OPENSHIFT TECHNICAL OVERVIEW ! Define cost models for infrastructure and understand your cost. ! Rate schedules per platform and per tenant with multi-tiered and multi-currency support
 ! CloudForms shows top users for CPU, memory, as well as cost. ! Chargeback/showback to projects based on container utilization. 91 FINANCIAL MANAGEMENT
92. OPENSHIFT TECHNICAL OVERVIEW 92 FINANCIAL MANAGEMENT