Lean Enterprise with Microservices and Big Data

Slide 1

Slide 1 text

How to enable the Lean Enterprise Johann Romefort co-founder @ rainbow

Slide 2

Slide 2 text

My Background • Seesmic - Co-founder & CTO Video conversation platform Social media clients…lots of pivots :) • Rainbow - Co-founder & CTO Enterprise App Store

Slide 3

Slide 3 text

Goal of this presentation • Understand what is the Lean Enterprise, how it relates to big data and the software architecture you build • Have a basic understanding of the technologies and tools involved

Slide 4

Slide 4 text

What is the Lean Enterprise? http://en.wikipedia.org/wiki/Lean_enterprise “Lean enterprise is a practice focused on value creation for the end customer with minimal waste and processes.”

Slide 5

Slide 5 text

Enabling the OODA Loop ! ! “Get inside your adversaries' OODA loop to disorient them” ! OBSERVE ORIENT DECIDE ACT USAF Colonel John Boyd on Combat: OODA Loop

Slide 6

Slide 6 text

Enabling the OODA Loop OODA Loop

Slide 7

Slide 7 text

The OODA Loop for software image credit: Adrian Cockcroft

Slide 8

Slide 8 text

OODA Loop • (Observe) Innovation and (Decide) Culture are mainly human-based • Orient (BigData) and Act (Cloud) can be automated

Slide 9

Slide 9 text

ORIENT

Slide 10

Slide 10 text

What is Big Data? • It’s data at the intersection of 3 V: • Velocity (Batch / Real time / Streaming) • Volume (Terabytes/Petabytes) • Variety (structure/semi-structured/unstructured)

Slide 11

Slide 11 text

Why is everybody talking about it? • Cost of generation of data has gone down • By 2015, 3B people will be online, pushing data volume created to 8 zettabytes • More data = More insights = Better decisions • Ease and cost of processing is falling thanks to cloud platforms

Slide 12

Slide 12 text

Data ﬂow and constraints Generate Ingest / Store Process Visualize / Share The 3 V involve heterogeneity and make it hard to achieve those steps

Slide 13

Slide 13 text

What is AWS? • AWS is a cloud computing platform • On-demand delivery of IT resources • Pay-as-you-go pricing model

Slide 14

Slide 14 text

Cloud Computing + + Storage Compute Networking Adapts dynamically to ever changing needs to stick closely to user infrastructure and applications requirements

Slide 15

Slide 15 text

How does AWS helps with Big Data? • Remove constraints on the ingesting, storing, and processing layer and adapts closely to demands. • Provides a collection of integrated tools to adapt to the 3 V’s of Big Data  • Unlimited capacity of storage and processing power ﬁts well to changing data storage and analysis requirements.

Slide 16

Slide 16 text

Computing Solutions for Big Data on AWS Kinesis EC2 EMR Redshift

Slide 17

Slide 17 text

Computing Solutions for Big Data on AWS EC2 All-purpose computing instances. Dynamic Provisioning and resizing Let you scale your infrastructure at low cost Use Case: Well suited for running custom or proprietary application (ex: SAP Hana, Tableau…)

Slide 18

Slide 18 text

Computing Solutions for Big Data on AWS EMR ‘Hadoop in the cloud’ Adapt to complexity of the analysis and volume of data to process Use Case: Oﬄine processing of very large volume of data, possibly unstructured (Variety variable)

Slide 19

Slide 19 text

Computing Solutions for Big Data on AWS Kinesis Stream Processing Real-time data Scale to adapt to the ﬂow of inbound data Use Case: Complex Event Processing, click streams, sensors data, computation over window of time

Slide 20

Slide 20 text

Computing Solutions for Big Data on AWS RedShift Data Warehouse in the cloud Scales to Petabytes Supports SQL Querying Start small for just $0.25/h Use Case: BI Analysis, Use of ODBC/JDBC legacy software to analyze or visualize data

Slide 21

Slide 21 text

Storage Solution for Big Data on AWS DynamoDB RedShift S3 Glacier

Slide 22

Slide 22 text

Storage Solution for Big Data on AWS DynamoDB NoSQL Database Consistent Low latency access Column-base ﬂexible data model Use Case: Oﬄine processing of very large volume of data, possibly unstructured (Variety variable)

Slide 23

Slide 23 text

Storage Solution for Big Data on AWS S3 Use Case: Backups and Disaster recovery, Media storage, Storage for data analysis Versatile storage system Low-cost Fast retrieving of data

Slide 24

Slide 24 text

Storage Solution for Big Data on AWS Glacier Use Case: Storing raw logs of data. Storing media archives. Magnetic tape replacement Archive storage of cold data Extremely low-cost optimized for data infrequently accessed

Slide 25

Slide 25 text

What makes AWS diﬀerent when it comes to big data?

Slide 26

Slide 26 text

Given the 3V’s a collection of tools is most of the time needed for your data processing and storage. Integrated Environment for Big Data AWS Big Data solutions comes integrated with each others already AWS Big Data solutions also integrate with the whole AWS ecosystem (Security, Identity Management, Logging, Backups, Management Console…)

Slide 27

Slide 27 text

Example of products interacting with each other.

Slide 28

Slide 28 text

Tightly integrated rich environment of tools On-demand scaling sticking to processing requirements + = Extremely cost-eﬀective and easy to deploy solution for big data needs

Slide 29

Slide 29 text

• Error Detection: Real-time detection of hardware problems • Optimization and Energy management Use Case: Real-time IOT Analytics Gathering data in real time from sensors deployed in factory and send them for immediate processing

Slide 30

Slide 30 text

First Version of the infrastructure Aggregate Sensors data nodejs stream processor On customer site evaluate rules over time window in-house hadoop cluster mongodb feed algorithm write raw data for further processing backup

Slide 31

Slide 31 text

Version of the infrastructure ported to AWS Aggregate Sensors data On customer site evaluate rules over time window write raw data for archiving Kinesis RedShift for BI analysis Glacier

Slide 32

Slide 32 text

ACT

Slide 33

Slide 33 text

Cloud and Lean Enterprise

Slide 34

Slide 34 text

Let’s start with a personal example

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

First year @seesmic • Prototype becomes production • Monolithic architecture • No analytics/metrics • Little monitoring • Little automated testing

Slide 37

Slide 37 text

I built a monolith

Slide 38

Slide 38 text

or…at least I tried

Slide 39

Slide 39 text

Early days at Seesmic First year @seesmic

Slide 40

Slide 40 text

Everybody loves a good horror story

Slide 41

Slide 41 text

We crashed Techcrunch

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

What did we do?

Slide 44

Slide 44 text

Add a QA Manager

Slide 45

Slide 45 text

Add bearded SysAdmin

Slide 46

Slide 46 text

We added tons of process so nothing can’t go wrong

Slide 47

Slide 47 text

Impact on dev team • Frustration of slow release process • Lots of back and forth due to bugs and the necessity to test app all over each time • Chain of command too long • Feeling no power in the process • Low trust

Slide 48

Slide 48 text

Impact on product team • Frustration of not executing fast enough • Frustration of having to ask for everything (like metrics) • Feeling engineers always have the last word

Slide 49

Slide 49 text

Impact on Management

Slide 50

Slide 50 text

• Break down software into smaller autonomous units • Break down teams into smaller autonomous units • Automating and tooling, CI / CD • Plan for the worst What can you do?

Slide 51

Slide 51 text

= Break down software into smaller autonomous units

Slide 52

Slide 52 text

Introduction to Microservices

Slide 53

Slide 53 text

Monolith vs Microservices - 10000ft view -

Slide 54

Slide 54 text

Monolith vs Microservices - databases -

Slide 55

Slide 55 text

Monolith vs Microservices - servers -

Slide 56

Slide 56 text

Microservices - example -

Slide 57

Slide 57 text

Break down team into smaller units

Slide 58

Slide 58 text

Amazon’s “two-pizza teams” • 6 to 10 people; you can feed them with two pizzas. • It’s not about size, but about accountability and autonomy • Each team has its own ﬁtness function

Slide 59

Slide 59 text

• Full devops model: good tooling needed • Still need to be designed for resiliency • Harder to test Friction points

Slide 60

Slide 60 text

Continuous Integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day

Slide 61

Slide 61 text

Continuous Deployment

Slide 62

Slide 62 text

Continuous Deployment

Slide 63

Slide 63 text

Tools for Continuous Integration • Jenkins (Open Source, Lot of plugins, hard to conﬁgure) • Travis CI (Look better, less plugins)

Slide 64

Slide 64 text

Tools for Continuous Deployment • GO.cd (Open-Source) • shippable.com (SaaS, Docker support) • Code Deploy (AWS) + Puppet, Chef, Ansible, Salt, Docker…

Slide 65

Slide 65 text

Impact on dev • Autonomy • Not afraid to try new things • More conﬁdent in codebase • Don’t have to linger around with old bugs until there’s a release

Slide 66

Slide 66 text

Impact on product team • Iterate faster on features • Can make, bake and break hypothesis faster • Product gets improved incrementally everyday

Slide 67

Slide 67 text

Impact on Management

Slide 68

Slide 68 text

• Enabling Microservices architecture • Enabling better testing • Enabling devops model • Come talk to the Docker team tomorrow!

Slide 69

Slide 69 text

Thank You follow me: @romefort [email protected]