Advanced JavaScript build pipelines using Gulp.js

It has been some time since JavaScript build tools like Grunt or Gulp were just the "next big thing" for web developers. Working without them is nearly unimaginable nowadays and it seems that there's almost no problem in our day to day workflow which cannot be solved by simply using just another plugin.

But are build tools really the answer to everything?

In this talk, we will take a look at the ten most common problems which seem to be unsolved by using just the build tool basics. We will create advanced building pipelines for our custom processes and find out reusable patterns which can be applied to similar issues.

Transcript

1. Advanced JavaScript
   Build Pipelines
   with Gulp.js
   

   View Slide

2. Let's talk a
   short bit about
   the JS build tool
   revolution
   

   View Slide

3. HTML
   CSS
   JavaScript
   

   View Slide

4. Sass CoffeeScript LESS
   P o s t C S S H A M L J a d e
   U g l i f y E S 6 R e a c t J S
   B r o w s e r i f y A n g u l a r J S
   E m b e r C S S M i n J S L i n t
   ESHint ImageOptim Mocha
   Jasmine TypeScript
   

   View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. 1530 lines of code
   original Ant tasks used:
   concat — copy — delete — mkdir
   

   View Slide

10. View Slide

11. Maybe Java isn't the right
    tool?
    Java is to JavaScript what

    Alf is to Gandalf
    

    View Slide

12. Grunt started a boom
    

    View Slide

13. Gruntfiles get long
    

    View Slide

14. Grunt tasks get slow:
    lots of reads
    and writes
    

    View Slide

15. One unmanageable horde
    of (sometimes)
    low quality plug-ins
    

    View Slide

16. And then came Gulp
    

    View Slide

17. React is in love with
    Webpack
    

    View Slide

18. Ember.js digs Broccoli.js
    

    View Slide

19. Ember.js digs Broccoli.js
    

    View Slide

20. View Slide

21. So… every community seems
    to have its preference
    Stick with your communities
    preference!
    

    View Slide

22. For all the rest?
    Gulp might be a good,
    very good option
    

    View Slide

23. Disclaimer
    

    View Slide

24. I (occasionally) contribute
    to Gulp
    

    View Slide

25. I'm writing a book on Gulp
    http://bit.ly/gulp-tool-book
    
    39% off with 39baumgar

    coupon code!
    

    View Slide

26. gulp.src(…) gulp.dest(…)
    Reads files Writes files
    

    View Slide

27. gulp.src(…) .pipe(uglify()) gulp.dest(…)
    

    View Slide

28. gulp.src(…) .pipe(uglify()) gulp.dest(…)
    .pipe(concat())
    

    View Slide

29. task
    manager
    streaming
    file
    system
    file watcher
    

    View Slide

30. gulp.task('scripts', function() {
    return gulp.src(‘src/**/*.js')
    .pipe(uglify())
    .pipe(concat('main.min.js'))
    .pipe(gulp.dest('dist'));
    });
    gulp.task('default', function(done) {
    gulp.watch(‘src/**/*.js', gulp.parallel(‘scripts'));
    done();
    });
    

    View Slide

31. undertaker.task('scripts', function() {
    return vinyl.src(‘src/**/*.js')
    .pipe(uglify())
    .pipe(concat('main.min.js'))
    .pipe(vinyl.dest('dist'));
    });
    undertaker.task('default', function(done) {
    chokidar.watch(‘src/**/*.js', undertaker.parallel(‘scripts'));
    done();
    });
    

    View Slide

32. so similar … yet so
    different?
    Gulp
    Streams
    Browserify 

    Streams
    

    View Slide

33. var source = require(‘vinyl-source-stream’);
    var b = browserify({
    entries: ['_js/main.js']
    });
    var bundle = function() {
    return b.bundle()
    .pipe(source(‘main.js’))
    .pipe(gulp.dest('js'));
    }
    gulp.task(‘bundle’, bundle);
    

    View Slide

34. This also goes the
    other way around…
    

    View Slide

35. var app = express();
    var router = express.Router();
    router.get('/*', function(req, res) {
    var stream = request('http://host.to.forward.to' + req.originalUrl);
    stream.pipe(res);
    stream
    .pipe(source('.' + req.originalUrl))
    .pipe(gulp.dest('./cachedir'));
    });
    app.use(express.static('_site'));
    app.use(express.static('cachedir'));
    app.use(router);
    

    View Slide

36. var app = express();
    var router = express.Router();
    router.get('/*', function(req, res) {
    var stream = request('http://host.to.forward.to' + req.originalUrl);
    stream.pipe(res);
    stream
    .pipe(source('.' + req.originalUrl))
    .pipe(gulp.dest('./cachedir'));
    });
    app.use(express.static('_site'));
    app.use(express.static('cachedir'));
    app.use(router);
    

    View Slide

37. So start thinking
    streams …
    

    View Slide

38. Multiple input formats
    return gulp.src(‘src/**/*.coffee)
    .pipe(coffee())
    .pipe(uglify())
    .pipe(concat(‘main.js’))
    .pipe(gulp.dest(‘build’))
    

    View Slide

39. Multiple input formats
    return gulp.src(‘src/**/*.js)
    .pipe(uglify())
    .pipe(concat(‘main.js’))
    .pipe(gulp.dest(‘build’))
    

    View Slide

40. Multiple input formats
    return gulp.src(‘src/**/*.js)
    .pipe(uglify())
    .pipe(concat(‘main.js’))
    .pipe(gulp.dest(‘build’))
    the same!
    

    View Slide

41. Multiple input formats
    And actually, you just want
    one bundle in the end
    

    View Slide

42. What if we could
    reuse parts of the
    stream?
    

    View Slide

43. return gulp.src(‘app1/src/**/*.coffee’)
    .pipe(coffee())
    .pipe(uglify()
    .pipe(concat(‘main.js’))
    .pipe(gulp.dest('build'));
    

    View Slide

44. var merge = require(‘merge2’);
    return merge(gulp.src(‘app1/src/**/*.coffee’)
    .pipe(coffee()),
    gulp.src(‘app1/src/**/*.js’))
    .pipe(uglify()
    .pipe(concat(‘main.js’))
    .pipe(gulp.dest('build'));
    

    View Slide

45. var merge = require(‘merge2’);
    return gulp.src(‘app1/src/**/*.coffee’)
    .pipe(markdown()),
    .pipe(gulp.src(‘app1/src/**/*.js’), {passthrough: true})
    .pipe(rename(blogFn))
    .pipe(wrap(layoutStr))
    .pipe(swig())
    .pipe(gulp.dest('build'));
    

    View Slide

46. Multiple bundles
    And now we need this
    over and over again
    for all our applications…
    

    View Slide

47. var elems = [
    { dir: ‘app1’, bundleName: ‘app1.min.js’ },
    { dir: ‘app2’, bundleName: ‘app2.min.js’ }
    ];
    

    View Slide

48. var streams = elems.map(function(el) {
    return gulp.src(el.dir + ‘src/**/*.coffee’)
    .pipe(coffee()),
    .pipe(gulp.src(el.dir + ‘src/**/*.js’), {passthrough: true})
    .pipe(uglify())
    .pipe(concat(el.bundleName))
    });
    return merge(streams)
    .pipe(gulp.dest('build'));
    

    View Slide

49. Incremental
    builds
    

    View Slide

50. Some tasks take long
    gulp.src(‘scripts/*.js’)
    .pipe(uglify())
    .pipe(gulp.dest())
    .pipe(concat())
    

    View Slide

51. Some tasks take long
    gulp.src(‘scripts/*.js’)
    .pipe(uglify())
    .pipe(gulp.dest())
    .pipe(concat())
    

    View Slide

52. Some tasks take long
    gulp.src(‘scripts/*.js’)
    .pipe(uglify())
    .pipe(gulp.dest())
    .pipe(concat())
    Too much is going on!
    Each change: Uglify all
    the files?
    

    View Slide

53. Some tasks take long
    gulp.src(‘scripts/*.js’)
    .pipe(uglify())
    .pipe(gulp.dest())
    .pipe(concat())
    

    View Slide

54. View Slide

55. filter files
    that have changed
    

    View Slide

56. filter files
    that have changed
    do performance
    heavy operations
    

    View Slide

57. filter files
    that have changed
    do performance
    heavy operations
    remember the
    old files
    

    View Slide

58. filter files
    that have changed
    do performance
    heavy operations
    remember the
    old files
    and continue
    with the other ops
    

    View Slide

59. gulp.task('scripts', function () {
    return gulp.src('app/scripts/**/*.js')
    .pipe(cached('ugly'))
    .pipe(uglify())
    .pipe(remember('ugly'))
    .pipe(concat('main.min.js'))
    .pipe(gulp.dest('dist/scripts'));
    });
    

    View Slide

60. gulp.task('scripts', function () {
    return gulp.src('app/scripts/**/*.js')
    .pipe(cached('ugly'))
    .pipe(uglify())
    .pipe(remember('ugly'))
    .pipe(concat('main.min.js'))
    .pipe(gulp.dest('dist/scripts'));
    });
    we use the cache to
    check if files have
    changed
    

    View Slide

61. gulp.task('scripts', function () {
    return gulp.src('app/scripts/**/*.js')
    .pipe(cached('ugly'))
    .pipe(uglify())
    .pipe(remember('ugly'))
    .pipe(concat('main.min.js'))
    .pipe(gulp.dest('dist/scripts'));
    });
    once we are done,
    we remember all
    the other files
    we stored in
    the cache
    

    View Slide

62. gulp.task('lint', function () {
    return gulp.src(‘src/**/*.js’)
    .pipe(jshint())
    .pipe(jshint.reporter(‘default’))
    .pipe(jshint.reporter(‘fail’));
    });
    

    View Slide

63. gulp.task('lint', function () {
    return gulp.src(‘src/**/*.js’, { since: gulp.lastRun(‘lint’) })
    .pipe(jshint())
    .pipe(jshint.reporter(‘default’))
    .pipe(jshint.reporter(‘fail’));
    });
    

    View Slide

64. gulp.task('images', function () {
    return gulp.src(‘src/images/**/*.*’)
    .pipe(imagemin())
    .pipe(gulp.dest(‘dist’));
    });
    

    View Slide

65. gulp.task('images', function () {
    return gulp.src(‘src/images/**/*.*’)
    .pipe(newer(‘dist’))
    .pipe(imagemin())
    .pipe(gulp.dest(‘dist’));
    });
    

    View Slide

66. When architecting
    Gulp build pipelines…
    

    View Slide

67. Separate the process
    from the content
    

    View Slide

68. Think about how the
    basic pipeline would
    look like
    

    View Slide

69. Think about how
    your data looks like
    

    View Slide

70. And then posh your pipes up
    with stream tools that
    change your data accordingly
    

    View Slide

71. View Slide

72. @ddprrt
    

    View Slide