Rendering large lists with Backbone

One of the first things that you do when starting with Backbone after sorting out your view hierarchy and memory management is an automatic way to display a collection. If you've read the blog post Backbone at Dataminr then you know we're using mixins to just tell a view that it needs to automatically list the collection for us. We found though that this was pretty slow for extremely large lists as it tries to keep everything in order and will add in sub views one at a time.

After having a look at where a lot of the time was being taken we could see it was in the calls to parseHTML and inserting the element in to the DOM. These were being called 100 times each for 100 elements, instead we wanted to batch all the HTML parsing and inserting to the dom together.

Below is the mixin we attach to a collection:

    Mixin.ComponentView.autoBigList = function(options) {

        // create the base element for the template
        var createEl = function(tagName, className, attributes, id, innerHTML) {
            var html = '<' + tagName;
            html += (className ? ' class="' + className + '"' : '');
            if (attributes) {
                _.each(attributes, function(val, key) {
                    html += ' ' + key + '="' + val + '"';
                });
            }
            html += (id ? ' id="' + id + '"' : '');
            html += '>' + innerHTML + '</' + tagName + '>';
            return html;
        };

        // get the needed variables
        this.after('initialize', function() {
            var iv = this.itemView.prototype;
            this._autobiglist = {
                html: '',
                template: Handlebars.compile(createEl(
                    iv.tagName,
                    iv.className,
                    iv.attributes,
                    iv.id,
                    (options.itemTemplate || iv.template)
                )),
                toAdd: []
            };
        });

        // setup and teardown listeners
        this.after('enterDocument', function() {
            this.listenTo(this.collection.on, 'reset', this.autlist_, this);
            this.listenTo(this.collection.on, 'add', this.autlist_, this);
            this.doneListing_ = _.debounce(this.doneListing_, 100);
            this.autolist_(this.collection);
        });

        this.setDefaults({
            // component view doesn't remove a decorated element, override
         rem_: function(model) {
          var view = _.find(this.getAllChildren(), function(child) {
           return child.model == model;
          });
          this.removeChild(view);
          view.$el.remove();
          view.dispose();
         },
            // collect the HTML together
            autolist_: function(model, silent) {
                var toAdd = this._autobiglist.toAdd;
                var template = this._autobiglist.template;
                if (model instanceof Backbone.Model) {
                    var setup = _.extend({}, options.setup);
                    setup.model = model;
                    var item = new this.itemView(setup);
                    toAdd.push(item);
                    if (!options.reverseOrder || this.reverseOrder)
                        this._autobiglist.html += template(item.serialize());
                    else
                        this._autobiglist.html = template(item.serialize()) + this._autobiglist.html;
                // if not single model, run each model
                } else if(model) {
                    $(this.getContentElement()).empty();
                    _.each(model.models, function(mod) {
                        this.autolist_(mod, true);
                    }, this);
                }
                if (silent !== true)
                    this.doneListing_();
            },
            // after all the HTML is collected put in DOM and attach views
            doneListing_: function() {
                var html = this._autobiglist.html;
                var toAdd = this._autobiglist.toAdd;
                if (!html)
                    return;
                // put html in document
                var div = document.createElement('div');
                div.insertAdjacentHTML("beforeend", html);
                var els = _.toArray(div.childNodes);
                var l = els.length;
                html = '';
                var frag = document.createDocumentFragment();
                for (var i = 0; i < l; i++) {
                    frag.appendChild(els[i]);
                };
                if (options.reverseOrder || this.reverseOrder) {
                    this.getContentElement().insertBefore(frag, this.getContentElement().firstChild);
                } else {
                    this.getContentElement().appendChild(frag);
                }
                // attach views
                for (i = 0; i < l; i++) {
                    var n = i;
                    if (options.reverseOrder || this.reverseOrder)
                        n = l - i - 1;
                 this.addChild(toAdd[n]);
                    toAdd[n].decorate(els[n]);
                };
                this._autobiglist.toAdd = [];
                this._autobiglist.html = '';
                if (this.afterList)
                 this.afterList();
            }
        });

    };

This can just be dropped in instead of our autolist mixin and works like a charm. So how does it work?

When we initialize the view for the collection we have a look at the defined ItemView and grab out any information Backbone uses to create a view's top level element: tagName, className, attributes & id. We save this along with the template function and create our own template function that will take a model's serialized object and return the full HTML back as a string.

Now that we can get the HTML for an item we need a way to collect these all together when our models come in. That why we've got these lines:

this.doneListing_ = _.debounce(this.doneListing_, 100);
this.autolist_(this.collection);

autolist_ is our function that will collect the HTML we need. We debounde doneListing so it is only called once we've collected all the HTML we need and save a second array which is the view for each bit of HTML

doneListing_ will create the DOM from the HTML which means we only need to parse the HTML once and then add it on to the dom. This is where our performance boost comes in. Once we've attached that though we still need to go through each of the new child nodes added and make that the HTML for the view. Using Backbone.ComponentView means we have a "decorate" function that does this for us but plain Backbone also has the setElement function that does the same thing.

And that's it, pretty simple. It should be noted though that this does not have any logic to handle a change in sort order, or if new items are inserted anywhere but the bottom (or the top) of the list. Removal should be fine though.

Hope that helps anyone with performance issues rendering large lists. If you want to know more about mixins, Backbone.Advice and Backbone.ComponentView see my blogpost on Backbone at Dataminr