Slide 1

Slide 1 text

the technical story of From 1 to 20 million users

Slide 2

Slide 2 text

Matthieu Moquet @MattKetmo web engineer at

Slide 3

Slide 3 text

Leader ride-sharing service Our goal is to become the 1st travel platform

Slide 4

Slide 4 text

Why this talk? ❖ History of the BlaBlaCar platform ❖ Overview our main technical choices ❖ Understand of our culture & methodologies

Slide 5

Slide 5 text

v1.0 2005–2006

Slide 6

Slide 6 text

The prototype

Slide 7

Slide 7 text

v2.0 (2008) – Plain PHP

Slide 8

Slide 8 text

2008 2010 2011 2012 v2.0

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Homemade framework + No code convention + Messy = Not maintainable in the long term

Slide 12

Slide 12 text

$  wc  -­‐l  lib.trip.php Longest method: 1000+ lines 3678

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No tests

Slide 15

Slide 15 text

Going further will be a pain Hiring will be hard We need to change things

Slide 16

Slide 16 text

Refactor all the code

Slide 17

Slide 17 text

Symfony 2.0

Slide 18

Slide 18 text

Because community matters

Slide 19

Slide 19 text

Attract new talented 
 & motivated people

Slide 20

Slide 20 text

Separation of Concerns Dependencies Injection Integration & Unit tests etc. Embrace best practices

Slide 21

Slide 21 text

Update product design

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Agility is our best strength

Slide 24

Slide 24 text

— Eberhardt Von Kuenheim « The big will not always eat the small, 
 but the fast ones will overtake the slow ones »

Slide 25

Slide 25 text

10+ deployments per day

Slide 26

Slide 26 text

DEVOPS… before it was cool « if you break it you fix it »

Slide 27

Slide 27 text

Automation Automatize as much as possible to reduce the time between dev and prod

Slide 28

Slide 28 text

Simple development workflow Master Branch

Slide 29

Slide 29 text

git push origin feat-awesome Deploy on staging environment is as simple as

Slide 30

Slide 30 text

Staging env. for QA & product owners https://feat-awesome.staging.blablacar.com

Slide 31

Slide 31 text

Continuous Integration https://github.com/alexandresalome/behat-launcher

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Localization Update translations on the fly

Slide 34

Slide 34 text

Frontend server Databases Backoffice Sync translations Update translations

Slide 35

Slide 35 text

Download last translations when app boot Update translations Mobile app CDN Backoffice

Slide 36

Slide 36 text

openl10n.io

Slide 37

Slide 37 text

Progressive Rollout ❖ Open new countries with v3 one by one ❖ It took about 2.5 years to run v3 everywhere ❖ Today we can deploy new features for a set of users (by attributes or random)

Slide 38

Slide 38 text

Monitoring

Slide 39

Slide 39 text

ELK

Slide 40

Slide 40 text

Monitoring

Slide 41

Slide 41 text

Departure to new horizons…

Slide 42

Slide 42 text

Opportunity to change backend But we still keep the same primary database

Slide 43

Slide 43 text

Database Updates Sync replication & No Master SPOF

Slide 44

Slide 44 text

Better archiving strategy

Slide 45

Slide 45 text

Photo Storage Don’t store static BLOB into MySQL. Use an elastic filesystem storage. MySQL AWS S3

Slide 46

Slide 46 text

elasticsearch ❖ Horizontally scalable ❖ Geo type index ❖ Aggregations

Slide 47

Slide 47 text

Varnish Fast reverse proxy cache 30% HIT/MISS Firemode to handle high traffic (TTL) Be careful with authenticated user blocks (Javascript is your friend)

Slide 48

Slide 48 text

SPDY (HTTP 2.0) Fast API calls for (compliant) mobile clients ngx_http_spdy_module

Slide 49

Slide 49 text

Asynchronous jobs Command Command Handlers Commands Queueing Ack

Slide 50

Slide 50 text

workers/      mail      sms      push      image-­‐resize      indexer      cache-­‐invalidation      elasticsearch-­‐indexation      trip-­‐publication      ... github.com/swarrot

Slide 51

Slide 51 text

Horizontal scalability

Slide 52

Slide 52 text

Event dispatcher Every business events is dispatched into RabbitMQ Easy to watch the events (in real-time or batchs)

Slide 53

Slide 53 text

Data Warehouse Log every business event into Hadoop user.register user.edit_bio user.left_rating user.post_trip ...

Slide 54

Slide 54 text

user.register user.edit_bio user.left_rating user.post_trip ... user.register user.post_trip ... Real-Time dashboard Log every business event into Elasticsearch

Slide 55

Slide 55 text

Such performances. Many users. Wow.

Slide 56

Slide 56 text

How to serve the product worldwide?

Slide 57

Slide 57 text

Datacenter centric One datastore to rule them all? It ain’t gonna work. ?? ??

Slide 58

Slide 58 text

Monolithics Micro-Services

Slide 59

Slide 59 text

Today we are mainly Monolithics But we would love using more Micro-Services Easier to deploy Development workflow Legacy database Smaller teams Faster deployments Easier to scale out

Slide 60

Slide 60 text

Gateway Principles (Clean Architecture) Decouple models (not db) Isolate business & data accesses (in the Monolytics to better decouple in micro services) Restrictive rules Enter the

Slide 61

Slide 61 text

Gateway RegisterUser PublishTrip PostMessage UpdateBio PostRating Users Messages Trips Business Data

Slide 62

Slide 62 text

But this is a long term project…

Slide 63

Slide 63 text

caching In the short term we invested on

Slide 64

Slide 64 text

Points of presence & Varnish

Slide 65

Slide 65 text

Mobile First

Slide 66

Slide 66 text

But how to cache the API? GET  /api/trips?from=Paris&to=Berlin   Authorization:  Bearer  7c82e855b0415f27bd92d   HTTP/1.1  200  OK   {          "trips":  [...]   }

Slide 67

Slide 67 text

{ Reverse proxy is useless if only the app knows the authorizations Reverse Proxy Client User Scopes Access Token

Slide 68

Slide 68 text

Let the reverse proxy check authorization directly Reverse Proxy But here is a latency problem

Slide 69

Slide 69 text

Use of stateless token Reverse Proxy So we don’t need a database anymore

Slide 70

Slide 70 text

Json Web Token

Slide 71

Slide 71 text

What mobile apps send GET  /api/trips?from=Paris&to=Berlin   Authorization:  Bearer  eyJhbGciOiJIUzI1NiI...

Slide 72

Slide 72 text

What backend servers receive GET  /api/trips?from=Paris&to=Berlin   X-­‐Auth-­‐User:  1337   X-­‐Auth-­‐Client:  android   X-­‐Auth-­‐Scope:  user_info,messages

Slide 73

Slide 73 text

App gets an Access Token from the origin Reverse Proxy App submit request with Access Token Reverse Proxy transform Access Token header into custom X-Auth headers

Slide 74

Slide 74 text

Cachable response HTTP/1.1  200  OK   Content-­‐Type  application/json   Vary:  X-­‐Auth-­‐Scope   ! {  "trips":  [  ...  ]  }

Slide 75

Slide 75 text

Long term solution to deliver data close to the client

Slide 76

Slide 76 text

❖ Manage massive amounts of data ❖ High Availability ❖ Multi Datacenter replication

Slide 77

Slide 77 text

‣ Know the read requests before creating your data models ‣ Create as many tables (KeySpaces) than you have views ‣ Denormalize the data (no join allowed)

Slide 78

Slide 78 text

CQRS & Event Sourcing ❖ Separate Read & Write ❖ Eventual consistency ❖ But hard to do with legacy software / database See talk PHPTour 2015 at moquet.net

Slide 79

Slide 79 text

C* C* C* Primary Data C* Read Data User post message Endpoint

Slide 80

Slide 80 text

Messaging (micro) service Expose an HTTP API

Slide 81

Slide 81 text

2008 2012 2013 2014 2015 2016 2 6 15 30+ … ?

Slide 82

Slide 82 text

blablatech.com We’re hiring Follow us

Slide 83

Slide 83 text

Thank You Slides available at moquet.net/talks/ipc-2015-blablacar Leave feedbacks at @MattKetmo