How Frontend Developers can drive the Switch to Microservices

F4612f9492cccc0c61c9b5f80d0ca8a1?s=47 jng
December 09, 2017

How Frontend Developers can drive the Switch to Microservices

My slides including speaker notes for the closing keynote at the 2017 Frontend Conference Munich on 9 December 2017 at Microsoft Germany.

-----

Original talk description:

Software is easy until you need to scale. But luckily for you, JavaScript and frontend developers have a clear advantage when moving to event-driven architectures and microservices. Events on clicks and blurs were your training wheels. Asynchronous is a given. You may have used a statement management framework like Redux or Vuex. You are destined for this.

To demonstrate, we'll examine a full stack JavaScript upload service written with Vue, Express and RabbitMQ. Ignore the frameworks. We'll focus on the messaging and state patterns that trip up many developers, preventing them from building true microservices. At the end of the talk, you will understand how to chunk file uploads on the front- and backend for happy users and developers.

-----

Note: I changed the topic from JavaScript to Frontend developers in general, inspired by a tweet from Yehuda Katz about how frontend development is the hardest kind of development out there.

F4612f9492cccc0c61c9b5f80d0ca8a1?s=128

jng

December 09, 2017
Tweet

Transcript

  1. FRONTCONF How Frontend Developers can drive the Switch to Microservices

    Julie Ng Architect, Allianz Germany 9 December 2017
  2. WHO AM I Designer • Developer • Architect Architect today

    Rich design & frontend experience got me here.
  3. Web Designer Back in 1999 Websites as a hobby since

    1999 Virtual Classroom As a designer, you don't make the rules. Someone else does. Designers are used to change. For a developer changing systems is expensive.
  4. Interaction Designer Flash - click to Program Flash - early

    first experience with programming for UX and business value Example: deciper Sütterlin for potential customers
  5. Interaction Developer(?) Flash - Actionscript Flash developers are in full

    control Limited Plugin from 1 vendor: more consistency You control screensize and interface size
  6. Full-Stack Developer Frontend to Backend (Ruby) Basic frontend, HTML/CSS, limited

    JavaScript, mostly jQuery for years Left Ancestry and freelanced for startups and myself. Learned Backbone.JS, Rails MVC I was full-stack. I built monoliths. Now what?
  7. Mike Monteiro on "What is a Designer?" solves problems within

    a set of constraints understands goals gathers information imposes order is a gatekeeper Enterprise Architect Designer at ♥ Why go from startups to corporate? I like problems and challenges. We have BIG problems. Plus I ran out of money
  8. Allianz Germany Financial Year 2016 Revenue Operating Profit € 32.3

    Billion € 2.0 Billion – Allianz Deutschland AG Zahlen, Daten und Fakten The bigger you are, the slower you are to move. But we're going through an IT transformation. Why?
  9. Challenges from Outside • New Business Models • New Markets

    • 100% Digital
  10. Hello Lemonade Insurance claims are processed in 3 Minutes and

    paid out
  11. A Fast Architecture Event driven Microservices Background Video: Microservices at

    Netflix Scale: Principles, Tradeoffs & Lessons Learned By Ruslan Meshenberg, GOTO 2016, This is not easy! It's a complex mind fuck. https://youtu.be/57UK46qfBLY?t=28m33s
  12. ATC: Open Offices My job is to help developers move

    in that direction. I work in our Agile Training Center (ATC).
  13. ATC: we have Post-Its

  14. ATC: we do Standups

  15. MICROSERVICES Why is it so hard?

  16. Offloading tasks to the cloud Why? For a better user

    experience. Google Photos AI analyzes my photos And made this lovely animated gif of Kilimanjaro from above for me without using my battery and resources
  17. — Jonas Bonér, Reactive Microservices Architecture What is a microservice?

    That's Easy It's small. It does one thing. It's fast. Because it's small? ⁃ ⁃ Some developers split monoliths into multiple apps and think "that's a microservice!" But they have to be deployed together , i.e. coupled and not a microservice Where do most developers fail? State / Data
  18. What is a microservice? Actually, it's hard State is not

    at rest → but in motion No beginning, no end → no indexes Distributed → where is my data (source of truth)? ⁃ ⁃ ⁃ Uncertainty philosophical mind fuck.
  19. You have no control Everything is uncertain Tough problem to

    solve!
  20. We never had control — Kevin Webber. A Journey into

    Reactive Streams. User picked their device and browser of choice We always think of all use cases responsive design We're more creative about worst case screnarios. Remember IE7? We've always juggled backend and frontend data and tried to re-connect
  21. What beginning? Only reactions. Not when. Whenever. mouseenter mouseover mousemove

    mousedown click dblclick wheel select ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ "when" is synchronous, coupled to time. "whenever" is reactive, no coupling to time.
  22. What Which end? All your bases are covered. document.addEventListener("dragend", function(event)

    { … }, false); document.addEventListener("dragexit", function(event) { … }, false); document.addEventListener("dragleave", function(event) { … }, false);
  23. DOM Event bubbling anyone? The force is with you frontend

    developers!
  24. STATE AS STREAMS Many sources & always moving

  25. Concat() — RxJS Official Docs

  26. Merge() — RxJS Official Docs

  27. CombineLatest() Looks intimidating… — RxJS Official Docs Looks Hard. Where

    many backend developers get lost Basically "holding" 2 values b2, b3, b4 -> always "b", no values from first stream
  28. CombineLatest() BMI is always moving… var weight = Rx.Observable.of(70, 72,

    76, 79, 75); var height = Rx.Observable.of(1.76, 1.77, 1.78); var bmi = Rx.Observable.combineLatest(weight, height, (w, h) => w / (h * h)); bmi.subscribe(x => console.log('BMI is ' + x)); 5 weights & 3 heights Streams -> determines calculated BMI sequence You might not get same BMIs every time.
  29. Forms & Streams Example: Rate Calculator Browser is really fast.

    You can't control sequence Data may be invalid at a certain time But so fast, it's valid later
  30. Forms & Models How to lose millions on the frontend

    '5.000.000,00' 500000000 5000000 '1,7' For a better UI: automatically insert thousands-separator in real-time on input Problem: What's a sum? rate (e.g. float)? currency (string)? Challenge: converting 1.000,00 to 1000, strings to floats and back at browser speeds - when you have state at rest. Streams are better. You will have inconsistent data at some point. But get to eventual consistency without user noticing.
  31. Forms & Streams Functional programming — export const INITIAL_STATE: IAppState

    = { count: 0, }; export function rootReducer(lastState: IAppState, action: Action): IAppState { switch(action.type) { case CounterActions.INCREMENT: return { count: lastState.count + 1 }; case CounterActions.DECREMENT: return { count: lastState.count - 1 }; } return lastState; } @angular-redux/store Beginners' Tutorial Question: Isn't functional programming blindly following the rules? Example: what if my account has no money? Answer: you get charged a fine (new way to make money).
  32. Similarly in the Backend Parallel requests are nice, but now

    what? — Jonas Bonér, Reactive Microservices Architecture What data do I send back? Worried about sending back inconsistent data?
  33. Backend streams Brain hurt yet? — Jonas Bonér, Reactive Microservices

    Architecture
  34. FRONTEND Most challenging development

  35. — @wycats on Twitter retired member of the Ruby on

    Rails retired member jQuery Core Team creator of ember.js member of the Rust Core Team
  36. Frontend Developers Advantage app development with SPAs real-time interactions asynchronous

    communication from multiple sources already event-driven user facing you never had control ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ Not 1999. Today every frontend developer knows JavaScript. There is no request/response, only NOW. User facing: shorter feedback loop. And you can SEE what you do. Less abstract. Not fair to backend devs. Control → mindfuck
  37. DEFINING BOUNDARIES Where Business meets Development Your user experience and

    short feedback loop helps you think differently, abstractly. Business = Design As an enterprise architect, you have to align business processes to technology, not just software.
  38. Cloud Computing Challenges Cloud architectures are distributed systems What can

    you do in parallel? What's synchronous? What's asynchronous? Where do you split business logic? ⁃ ⁃ ⁃ ⁃ Not an all-or-nothing game Charts with parallel lines are easy to understand. Translating business to code is not
  39. Charts with parallel lines are easy Translating business logic to

    code is not
  40. One Customer

  41. Competing Customers Who will get their coffee(s) first?

  42. How & where to optimize?

  43. Divide and Conquer

  44. Smart is messy

  45. CLOUD UPLOAD SERVICE Demo & Deep dive

  46. How do you upload a 50MB file? A single HTTP

    request with SOAP?
  47. Distributed Upload Service Demo

  48. Demo shows broken worker with a bug → no need

    to redeploy app Queued uploads wait for available workers Large files are stored in chunks and streamed back as single file
  49. Offload uploads to Workers Frontend (3 MB) Backend Proxy message

    broker worker (1 MB) worker (1 MB) worker (1 MB) e.g. /api/worker01/upload/123 upload is saved in n chunks and streamed back as single file ⁃ ⁃ ⁃ Everything scalable: frontend, backend, workers Bottle is backend, so break off worker job missing direct line between proxy and worker I stupidly realized this morning, instead of absolutely positioning div s, I should have used PowerPoint and screenshots
  50. Uploading - Get identifier app.get('/api/upload/new', function (req, res) { let

    id = crypto.randomBytes(16).toString('hex') res.json({ subscription: { id: id, } }) }) synchronously fetch ID via REST better error handling. No need for timeout.
  51. Uploading - worker waits for chunks queue.on('connect', () => {

    http.listen(port, () => { console.log(`=== [${workerName}] listening on ${port}… ===` ) }) }) queue.on('message', (attrs) => { let route = `/api/${workerName}/upload/${attrs.id}` queue.transmit('ready', Object.assign({ route: route }, attrs)) http.post(`/upload/${attrs.id}`, (req, res) => { let upload = new Upload(attrs.id, req.headers) queue.transmit('begin', attrs) upload.on('file:end', () => { queue.transmit('done', attrs) res.writeHead(200, { 'Connection': 'close' }) res.end("That's all folks!") }) return req.pipe(upload) }) }) worker listens for particular URL writes file/chunk to disk, could be stored in memory
  52. Downloading - stream 4 chunks back as 1 file app.get('/api/download/:id',

    function (req, res) { let id = req.params.id Upload.findOne({id: id}, (err, upload) => { if (!err && upload) { let chunks = upload.chunks.split(',') res.set({ 'Content-Type': upload.contentType, 'Content-Disposition': `attachment; filename="${upload.name}"` }) // What is this? pipeChunks(chunks, res) } else { if (err) console.log(err) res.status(404).send('File not found') } }) })
  53. Client (vueJS) Messages Command Destination Description SEND /upload/new send file

    details to backend SUBSCRIBE /upload/ready receive chunk upload end points SUBSCRIBE /upload/create receive download url a er upload is created on backend
  54. Message Example, ack s and s >>> SEND (app) x-upload-id:73750a171c67a8d54d1cfe67f81853c3

    x-upload-event:done content-type:application/json subscription:done@73750a171c67a8d54d1cfe67f81853c3 ack:client destination:/topic/upload.done timestamp:1504711122064 content-length:345 {"upload":{"id":"73750a171c67a8d54d1cfe67f81853c3","name":"jqueryconf_final_2.pdf","contentType":"app
  55. Backend Messages - Chunks Command Destination Description SUBSCRIBE /topic/chunk.ready transform

    into upload/ready message for frontend SUBSCRIBE /topic/chunk.done check if all chunks are done. If so, send upload.done message subscribes are whenevers reactive programming change message route for frontend (in retrospect I'm not sure if this was a good idea?)
  56. SUMMARY Advantages & Challenges

  57. Microservices: advantages For best user experience Fast and real-time Resilient

    and fault tolerant ⁃ ⁃
  58. Microservices: challenges But easier for frontend developers Easy in JavaScript

    - not much code. You already know real-time and streams. Leverage both synchronous and asynchronous calls. Leverage both REST and web sockets - error handling. Messy is OK, and by design. You don't need 100% perfect control. Splitting logic is difficult. ⁃ ⁃ ⁃ ⁃ ⁃ ⁃
  59. Microservices - Beauty in Chaos Julie Ng @jng5