Roll Your Own Asset Pipeline with Gulp

I’ve found myself using Gulp for just about everything involving HTML/CSS/JS these days. It’s super fast, quick to write scripts for and flexible. I’m at a point now where I have a ton of projects I can just cd into, run gulp and be up and running. It’s the best solution I’ve found for delivering static assets whether for local development or production.

If you haven’t used a tool like this before, you might be thinking that your cobbled together scripts (or Asset Pipeline) work just fine. The value of using Gulp is not within its ability to concatenate files, minify files or any of that. Gulp’s true value lies in its ability to compose these things well. Using disparate components might make it easy to concatenate JS, but mixing that in with Rev will immediately compound the complexity of your home-grown scripts. It’s value compared to Asset Pipeline is that it is still flexible.

If you’ve been on Rails’ Asset Pipeline, or on a totally different stack, Gulp should offer you a great solution to static asset compilation. Here’s how:

Rails Asset Pipeline Woes

One of my favorite parts of Rails has been the Asset Pipeline. The ability to just start throwing assets into the application and have rails spit out CDN-ready assets has been super helpful. After using it for a few years now though, I’m feeling the pain that many of you are. Asset pre-compilation errors, slow deploys, inflexibility, tying yourself to the Rails stack. It’s rough.

Enter Gulp

Gulp is a build system. It’s like Grunt, Make, Rake, and the like. It’s easy to use for the person running it. While it does have a slight learning curve, you’ll find it a super useful tool for all kinds of tasks. It’ll be the fastest weapon in your toolbox for asset compilation (speed for both time to develop and runtime).

CSS Compilation

I imagine just about everyone these days is using a CSS preprocessor. So let’s see how to do this with Gulp.

First, create a new app folder and add an empty package.json to store your node.js dependencies (I’m assuming you have Node installed here). Then we install Gulp and its LESS compiler:

Alright, now create your gulpfile.js:

This file just defines a single task less that will compile the css/app.less file into dist/

And a basic LESS file at css/app.less:

Now if we run gulp less we’ll see our compiled css in dist/app.css. Great!

Watching for Changes

This is great for building the assets once (perhaps in a deploy), but for local development it would be a pain to have to type gulp less every time we wanted new assets. gulp.watch() will be our helper here. Rewrite your gulpfile like so:

Note a couple things:

  • We watch for any less file but only compile app.less. This is because with LESS you would likely @import whatever other files you need. This strategy will not work for JavaScript (but we’ll cover that later).
  • The watch task depends on less. This ensures that the first run will have the assets there already.

Now, run a gulp watch, make an edit to your file and see gulp recompiling the stylesheets each time:


✗✗✗ myapp:(gh-pages) ✗ gulp watch
[gulp] Using gulpfile ~/src/myapp/gulpfile.js
[gulp] Starting 'watch'...
[gulp] Finished 'watch' after 6.17 ms
[gulp] Starting 'less'...
[gulp] Finished 'less' after 24 ms
[gulp] Starting 'less'...
[gulp] Finished 'less' after 2.73 ms

Now we have Gulp compiling our assets each time we touch the file: great! On my projects I like to include livereload to refresh the browser automatically on updated files. (it’s not as hard to setup as it sounds, really!)

Concatenating JavaScript

Now that we’ve mostly reimplemented asset pipeline for CSS, it’s time to look at JavaScript. JavaScript, unlike LESS does not have the convenient @import to support specifying what files to import, so we will just concatenate all of them into one file. (If you want something like that, look into Browserify, but that’s out of scope for this guide).

First install gulp-concat:

npm install --save-dev gulp-concat

Then add this script task to your gulpfile.js:

And the following JS file to src/users.js (not the most interesting code in the world):

console.log('user.js is called');

Now a gulp scripts or a gulp watch will concatenate your JS as well!

Minification

Minification is easy mode. Just pipe the stream JS through Uglifier (make sure you npm install) and set the compress flag on less:

Rev

This is my favorite part of using Gulp. Rev will give you that friendly app-ef62e7.js filename output that Asset Pipeline is famous for. The reason for it is you can cache it forever. New requests will just point to new files. CDNs love this. To get the files to have the hash is pretty easy with Rev.

Now the filename has the hash appended to it! Note the digest I generate as well; that looks like the following:

Here is where a bit of custom code would come into play. When you generate your index.html (or wherever else you reference the CSS/JS) you will have to swap out the URL for the one in the digest file. Should only be a matter of parsing this JSON file in your framework of choice, or having Gulp rewrite your index.html to replace the CSS/JS include with the correct filename.

Note for Angular.js users

Most of my front-end apps are built with Angular. Two issues crop up when using angular with this method: ordering of the module getters/setters and minification, both of which can break Angular’s dependency injection.

If you’ve used Angular, you know there is a way to ‘get’ the module, and a way to ‘set’ the module. angular.module('appname', []); vs angular.module('appname');. You must call the setter once before the getter, then can use the getter as much as you want.

To make this work with my concat strategy, first create a module.js for each of your modules, then edit the gulpfile’s script task like so:

Also note that ngmin is thrown in there to solve the minification/DI issue.

There are a lot of places you can go with this. It is trivial to deploy this to Google Pages, S3, or the public folder of a web app. You could put all of your static assets into its own Git repo, or make this part of your deploy automation. The point here is not to dictate how your app should be built, but that Gulp can offer you flexibility to compose these tools in a workflow that works for you.

So next time you need asset pre-compilation, take a look at Gulp and let me know what you think.

About Jeff Dickey

Twitter: @dickeyxxx
This entry was posted in Web and tagged , , , , , , . Bookmark the permalink.
  • http://tehbilly.com/ TehBilly

    I’ve recently discovered gulp and am absolutely loving it. My old process involved manually (I know, hush) concatenating my css/js files and minifying them. This did let me easily preserve concatenation order, which is something I wish I could specify with gulp. Or maybe I can and just haven’t discovered how yet.

    • http://dickey.xxx/ Jeff Dickey

      you can, just pass an array in (you can see this in the article during the angular bit)

      • http://tehbilly.com/ TehBilly

        Actually figured that out last night from the little bit in your post! Lots of good info here, the posts I’ve read are very thorough but aren’t dry. Please keep it up, this series in particular was a very pleasant and timely find for me. :)

  • efvincent

    Nice post. I spent last weekend converting a good size project from grunt to gulp to get a feel for it. While I definitely like the imperative / streaming approach of gulp better than grunt, I ran into some real problems. Mostly it’s been around sequencing events. First cleaning target directories, and only when that’s done starting the concat / uglify (with source maps), copy the index.html and only when that’s done and the JS has been rev()’d would I be ready to inject the new names into index.html.

    Gulp tries to do everything at the same time, asynchronously. Which is great, unless you need to coordinate between streams. It’s the same problem as running asynchronous workflows in general; there needs to be coordination between threads (or streams) in all but the most trivial cases.

    Have you come across this problem, and if so, how did you address it? I’ve seen some attempts, and there is a beta of Gulp in the works with far more sophisticated workflow control, but right now I can’t find anything reliable.

    • http://dickey.xxx/ Jeff Dickey

      I have. The parallelize up front portion does have problems. Gulp works great as an asset pipeline because the only sort of orchestration you need is dependencies (which work well in gulp).

      What doesn’t work so well is a mutex sort of thing (these 2 can’t run at the same time, but don’t depend on each other either).

      Ultimately I think that gulp needs different types of tasks. Some for building assets, some for shelling out to processes, etc.

      Still though, I love how concise gulp is compared to grunt, and the fact it handles the assets in memory and in parallel results in huge performance. I think adding on functionality like this really wouldn’t be that hard.

    • Maciej Dudziński

      For synchronous task running use https://www.npmjs.org/package/run-sequence
      With this you can run clean, copy, and next tasks with sure, that dist folder contain actual data.

      • efvincent

        I did end up looking at run-synchronous, but it says right up front that it’s a temporary, hack solution that may stop working when orchestrator is updated in gulp. Translation: there’s no clean way of achieving this result; so for me, while I *like* gulp’s approach better, I’ll stick with grunt which is still a great tool, and handles all the use cases I need more cleanly.

        When gulp’s next version is stable I can switch production code over. In the meantime, I’m keeping my eye on the beta.

        • Joshua Bellamy-Henn

          It may claim that it’s a hack solution but it solved all my problems.

    • Steven Daniels

      You can make tasks wait for other tasks to complete first by using this syntax:

      task( ‘secondTask’, ['firstTask'], function(){…});

      http://stackoverflow.com/questions/23458428/gulp-how-to-pipe-to-another-task

  • http://nfriedly.com/ Nathan Friedly

    Nice intro! One suggestion though: you probably meant to add `cd mynewapp` to your first gist.

    • http://dickey.xxx/ Jeff Dickey

      fixed!

  • http://joakim.beng.se/ Joakim Bengtson

    Good post!
    For sorting angular files I’ve made the gulp-angular-filesort, which analyzes your js files and sorts them according to module definitions and dependencies. Because of this you don’t need to name your files in a certain way, and can therefor fully follow Google’s app structure best practices.

    • http://dickey.xxx/ Jeff Dickey

      great tip!

  • Pingback: Roll Your Own Asset Pipeline with Gulp | Flippin' Awesome

  • Nic Johnson

    Thanks for the write-up, Jeff.

    I love the idea of gulp-rev but it seems like it falls short. What do you do for references to assets in CSS files?

    I’ve run across gulp-rev-all which seems to address that issue but I haven’t completely nailed it down yet.

  • paveldemyanenko

    Also there exist a js port of sprockets https://github.com/nodeca/mincer

  • Gregory McCue

    Thanks for this! Super useful intro to Gulp.

  • Pingback: Best materials #web development Nr.8 (4-10 May 2014)

  • thelinuxlich

    The problem of replacing Rails Asset Pipeline with Gulp for me lies on dealing with assets inside engines.