Deploying client-side apps, 1,000 (or so) at a time Rebecca Murphey / CascadiaJS 2015

@rmurphey ~ rmurphey.com

this is a talk about how we’re rethinking the systems we 
 use to build and deploy client-side web applications 
 at Bazaarvoice this is not a talk about how we should all start building 
 and deploying client-side web applications

the scene ratings & reviews for brands & retailers SPA in OPP* single code base, multiple customers one build per site/locale combo lots of sites, lots of locales * Credit to Mike Pennisi for this way-cooler acronym for 3PJS.

what’s a build? verify API fetch config generate config make core app js make scout js make css embed font in css download images publish to s3 verify display publish scout file purge akamai probably some 
 other stuff i forgot

return util.format( 'grunt %s %s', args.join(':'), options.join(' ') );

how does it even no clear contract between steps primitive, blocking retries builds are lost for most failures no demand-based scaling of resources limited observability overall

what’s a build? a set of steps some serial, some parallel triggered by some initial input producing some final output always with the possibility of failure

what’s a step? a unit of work likely asynchronous possibly requiring input possibly depending on a service ideally producing output

No content

{ steps : [ { name : 'step1', activityType : 'readMessageBody', input : 'message:workflow' }, { name : 'step2', activityType : 'randomNumber' }, { name : 'step3', activityType : 'email', input : 'step1:body,step2:number' } ] }

var Q = require('q'); var SES = new AWS.SES(); module.exports = function (input) { return Q.ninvoke(SES, 'sendEmail', { // ... Message : { Subject : { Data : 'A message from Pontiac' }, Body : { Text : { Data : input.number + '\n' + input.body } } } }); };

No content

No content

No content

No content

orchestration the automated arrangement, coordination, 
 and management of complex computer systems, middleware, and services

No content

“The one thing about Simple Workflow is it can be really hard to reason about, because things can happen out of order.” — (approximately) Joe Pollard

“The one thing about Simple Workflow is it can be really hard to reason about, because things can happen out of order.” — (approximately) Joe Pollard

pontiac a tool for describing and performing a workflow, using SWF, where a workflow consists of 1. a set of steps, 2. information about the input 
 required for each step, 3. information about the ideal duration and
 concurrency of those steps, and 4. the strategy for retrying failed steps

chopper receive a build message fetch locales create new messages per locale delete original message

No content

module.exports = { name : 'chopper', steps : [ { name : 'getDeploymentLocales', activityType : 'chopper!getDeploymentLocales', input : 'message:workflow', retries : 1, timeout : 10 }, { name : 'createChoppedMessages', activityType : 'chopper!createChoppedMessages', input : 'message:workflow,locales:getDeploymentLocales' }, { name : 'deleteMessage', activityType : 'shared!deleteMessage', input : 'message:workflow' } ] };

getDeploymentLocales module.exports = function (input, log) { var message = parseContents(input.message); var deploymentUrl = message.deploymentUrl; var deploymentRevision = message.deploymentRevision; var request = getDeploymentDoc({ v : deploymentRevision }, deploymentUrl); var promise = request(function (response) { return response.locales; }, function (err) { log.error(err, 'BHive request failed'); throw err; }); return promise; };

size of step output is limited and must be a string; so, we have to store large or binary data no guarantee that steps will run on the same machine; can’t rely on filesystem retries, concurrency are first class clear contract between steps framework abstracts known complexities adding a step is exactly as hard as writing the step itself, and no harder

reality check none of this is in production yet; rewriting a mission-critical build system that isn’t visibly broken is a hard sell planning to pilot this instead with teams that have more critical deployment issues good news: lots of common steps across projects, and sharing will be easy now

lessons / reminders grunt is tools are easy to abuse identify and abstract common complexity write all code as testable modules there is great power in convention smart people have probably already solved 
 [at least part of] your problem

thank you @rmurphey @ttl_podcast rmurphey.com ttlpodcast.com