Rendering large lists with Backbone

One of the first things that you do when starting with Backbone after sorting out your view hierarchy and memory management is an automatic way to display a collection. If you've read the blog post Backbone at Dataminr then you know we're using mixins to just tell a view that it needs to automatically list the collection for us. We found though that this was pretty slow for extremely large lists as it tries to keep everything in order and will add in sub views one at a time.

After having a look at where a lot of the time was being taken we could see it was in the calls to parseHTML and inserting the element in to the DOM. These were being called 100 times each for 100 elements, instead we wanted to batch all the HTML parsing and inserting to the dom together.

Below is the mixin we attach to a collection:

    Mixin.ComponentView.autoBigList = function(options) {

        // create the base element for the template
        var createEl = function(tagName, className, attributes, id, innerHTML) {
            var html = '<' + tagName;
            html += (className ? ' class="' + className + '"' : '');
            if (attributes) {
                _.each(attributes, function(val, key) {
                    html += ' ' + key + '="' + val + '"';
                });
            }
            html += (id ? ' id="' + id + '"' : '');
            html += '>' + innerHTML + '</' + tagName + '>';
            return html;
        };

        // get the needed variables
        this.after('initialize', function() {
            var iv = this.itemView.prototype;
            this._autobiglist = {
                html: '',
                template: Handlebars.compile(createEl(
                    iv.tagName,
                    iv.className,
                    iv.attributes,
                    iv.id,
                    (options.itemTemplate || iv.template)
                )),
                toAdd: []
            };
        });

        // setup and teardown listeners
        this.after('enterDocument', function() {
            this.listenTo(this.collection.on, 'reset', this.autlist_, this);
            this.listenTo(this.collection.on, 'add', this.autlist_, this);
            this.doneListing_ = _.debounce(this.doneListing_, 100);
            this.autolist_(this.collection);
        });

        this.setDefaults({
            // component view doesn't remove a decorated element, override
         rem_: function(model) {
          var view = _.find(this.getAllChildren(), function(child) {
           return child.model == model;
          });
          this.removeChild(view);
          view.$el.remove();
          view.dispose();
         },
            // collect the HTML together
            autolist_: function(model, silent) {
                var toAdd = this._autobiglist.toAdd;
                var template = this._autobiglist.template;
                if (model instanceof Backbone.Model) {
                    var setup = _.extend({}, options.setup);
                    setup.model = model;
                    var item = new this.itemView(setup);
                    toAdd.push(item);
                    if (!options.reverseOrder || this.reverseOrder)
                        this._autobiglist.html += template(item.serialize());
                    else
                        this._autobiglist.html = template(item.serialize()) + this._autobiglist.html;
                // if not single model, run each model
                } else if(model) {
                    $(this.getContentElement()).empty();
                    _.each(model.models, function(mod) {
                        this.autolist_(mod, true);
                    }, this);
                }
                if (silent !== true)
                    this.doneListing_();
            },
            // after all the HTML is collected put in DOM and attach views
            doneListing_: function() {
                var html = this._autobiglist.html;
                var toAdd = this._autobiglist.toAdd;
                if (!html)
                    return;
                // put html in document
                var div = document.createElement('div');
                div.insertAdjacentHTML("beforeend", html);
                var els = _.toArray(div.childNodes);
                var l = els.length;
                html = '';
                var frag = document.createDocumentFragment();
                for (var i = 0; i < l; i++) {
                    frag.appendChild(els[i]);
                };
                if (options.reverseOrder || this.reverseOrder) {
                    this.getContentElement().insertBefore(frag, this.getContentElement().firstChild);
                } else {
                    this.getContentElement().appendChild(frag);
                }
                // attach views
                for (i = 0; i < l; i++) {
                    var n = i;
                    if (options.reverseOrder || this.reverseOrder)
                        n = l - i - 1;
                 this.addChild(toAdd[n]);
                    toAdd[n].decorate(els[n]);
                };
                this._autobiglist.toAdd = [];
                this._autobiglist.html = '';
                if (this.afterList)
                 this.afterList();
            }
        });

    };

This can just be dropped in instead of our autolist mixin and works like a charm. So how does it work?

When we initialize the view for the collection we have a look at the defined ItemView and grab out any information Backbone uses to create a view's top level element: tagName, className, attributes & id. We save this along with the template function and create our own template function that will take a model's serialized object and return the full HTML back as a string.

Now that we can get the HTML for an item we need a way to collect these all together when our models come in. That why we've got these lines:

this.doneListing_ = _.debounce(this.doneListing_, 100);
this.autolist_(this.collection);

autolist_ is our function that will collect the HTML we need. We debounde doneListing so it is only called once we've collected all the HTML we need and save a second array which is the view for each bit of HTML

doneListing_ will create the DOM from the HTML which means we only need to parse the HTML once and then add it on to the dom. This is where our performance boost comes in. Once we've attached that though we still need to go through each of the new child nodes added and make that the HTML for the view. Using Backbone.ComponentView means we have a "decorate" function that does this for us but plain Backbone also has the setElement function that does the same thing.

And that's it, pretty simple. It should be noted though that this does not have any logic to handle a change in sort order, or if new items are inserted anywhere but the bottom (or the top) of the list. Removal should be fine though.

Hope that helps anyone with performance issues rendering large lists. If you want to know more about mixins, Backbone.Advice and Backbone.ComponentView see my blogpost on Backbone at Dataminr

Aspect Oriented Programming in JavaScript

You may have heard about AOP recently or about some of the new libraries that are coming out. The first I had heard about it was with Angus Croll's talk on How we learned to stop worrying and love JavaScript, although they just called it advice (and is now being used in Twitter Flight). So just what is Aspect-Oriented, and why do you care?

Aspect Oriented programming is actually a subset of Object Oriented programming. It helps in code re-use where there are cross-cutting concerns that don't fit well in to the single inheritance model. These can be things like logging which you may want to apply to objects throughout your program that don't share a common ancestor that would make sense to add the functionality to.

So what we really need is functionality (called "advice", which is what Angus named his library) and a mechanism to add it to an object (called a "pointcut") and these two things together are called an "aspect". In the speaker deck we are given "before", "after" and "around" as our pointcuts, and our Aspect is actually the function that contains these. It turns out that using functions for describing these aspects are quite useful.

I wrote a plugin for Backbone called Backbone.Advice which we use at work. We've created a file which contains a lot of these mixins that we can now apply across different contsructors. This has allowed us to separate out logic which was repeated in disparate parts of the system, and f you look at the github repo you will see some examples which will do things from automatically listing views to giving you keyboard navigatable lists. Aspects turn out to be very handy.

So now the downsides. Using AOP makes it extremely difficult to debug. You will use functionality across different parts in your system but not necessarily know where it's going to be put when writing. You will also have a fairly ugly callstack, especially if you apply a few different aspects.

So how do we get around this? The trick is to keep the aspects simple and testable. If you have a look at the mixins you can see that often we're making another call to this.mixin(Mixin....); which gives us a sort of inheritance structure. We're also careful about our naming, keeping things consistent across aspects. Also we only allow mixing in on a constructor and not an instance which means we can easily find out what mixins are being used.

Some other AOP javascript libraries you can look in to:

• YouAreDaChef
• meld
• Aop.js

Monads in plain JavaScript

So you want to know what these Monads thing is? Douglas Crockford in an interview said that if you understand them you lose the ability to explain them, and then went on to do a Monads & Gonads talk. In the talk he said you don't have to understand category theory or Haskell and I happen to agree, but then he jumps in to some generic monad constructor that may be confusing at first. So what's a monad in plain english?

Simply put it's a wrapper around a value. There are also some laws that it should conform to, but more on that later.

As with any wrapper we need two methods, one to get and one to set the value. These are called "return" and "bind". Let's construct the "Maybe" monad - the easiest monad to get your head around:

var Maybe = function(value) {
  this.value = value;
};

So we have a constructor, monads are something called a "functor" which basically is a "functional object" that allows you to move values between sets. In this case the maybe monad will allow us to use "undefined" in places which expect an actual value and we would see an error otherwise.

So let's look at the return function:

Maybe.prototype.ret = function() {
  return this.value;
};

Nice and easy, it gives us a method to get the value of the monad. Now comes the fun part, bind:

Maybe.prototype.bind = function(fn) {
  if (this.value != null)
    return fn(this.value);
  return this.value;
};

So the bind function runs a given function with the value of the monad. In the case of the maybe monad it just skips running the function if the value doesn't exist - and that's it!

Now we should talk about "lift". you'll see that the bind will return whatever the function returns for the value. Well we want a monad as a return so you should be passing in functions that return a monad - but who wants to do that? instead we'll just create a "lift" function that can take in a function that returns a normal value and changes it to a monad. pretty easy, it'd look something like this:

Maybe.lift = function(fn) {
  return function(val) {
    return new Maybe(fn(val));
  };
};

so now we can do things like have an addOne function:

var addOne = function(val) {
  return val + 1;
};

lift it:

var maybeAddOne = Maybe.lift(addOne);

then you can use it with bind! But what if we have a function that takes two monads? say we want to add two together?

Maybe.lift2 = function(fn) {
  return function(M1, M2) {
    return new Maybe(M1.bind(function(val1) {
      return M2.bind(function(val2) {
        return fn(val1, val2);
      });
    }));
  };
};

This one's a bit more complicated, but basically it just uses closures to get the values from the two monads before running it through the function, and because it's a maybe monad it will just pass back undefined  so we can safely use undefined values without errors. You can try it out like this:

var add = function(a, b) {return a + b;};
m1 = new Maybe(1);
m2 = new Maybe(2);
m3 = new Maybe(undefined);

var liftM2Add = Maybe.lift2(add);

liftM2Add(m1, m2).ret(); //3
liftM2Add(m3, m2).ret(); //undefined
liftM2Add(m1, m3).ret(); //undefined

And that's it. So to recap a monad is just a container. You can pass in a function to operate on it and get a returned value, or ask for it's value. You've probably used monads in the past without knowing (like promises, you just send it a function to run on it's value and it returns back itself - so it's a lifted bind) and perhaps even created some.