Templates: The unsolved mystery

This post is all about Templating (and HTML templating for JavaScript applications in particular) and how we really don't know how to do it yet. I'm sure there are a great many people out there that just go with the recommended templating engine for whatever tools they are using. But which should we really be using for out project? The truth is that we haven't found the best solution yet. So let's look at some of the soutions we have now.

Types of template engines

Render as a string

For the majority of templating solutions out there the output is a string. Why is this? Well there are two very good reasons

1. It's easier. You're HTML is written as a string so it's easy to just replace those placeholders and output that string
2. Thats how data is sent from the backend. So if your output is a string it can also be used on the backend to render your HTML templates.

The issue with this? Well on the frontend we still have to take that string and change it in to DOM nodes. This usually means we use innerHTML to convert the string and this means the browser then has to parse the string and create the nodes. If you look at your timeline in chrome developer tools you may see "parse HTML", and it can take a fair bit of time:

You can see in an unoptimized setting where you have lots of views come in that this may take a while. One way to get around this is to batch them all together before you do an innerHTML call. This will be a bit faster but is still slower than adding in a detached DOM tree.

If you're coming up against this problem then see my post on Rendering large lists with Backbone

Another big issue (perhaps the biggest issue) with this method is that you're creating a whole new DOM structure from the string. This means that any event listeners or state (such as current user focus or selection) will be lost. Backbone gets around this by having a top level element and using event delegation to handle events but this still does not address the issue of lost state, or even HTML that is added by subviews (there are plenty of plugins to help with the subview problem, you can lookup the backbone plugins wiki under Views or Frameworks).

Lightweight DOM representation

We all know the DOM is slow. However React (and I believe AngularJS) has taken a different approach where they keep their template will output their own representation of the DOM. This means they can compute the smallest amount of transforms (operations) they need to perform on the DOM to make changes. This is a good method if there are several changes happening to the DOM and means that we can still output a string but have our DOM exist on the page. However there is still a performance hit because we need to change the string to a DOM representation.

Output DOM

There isn't much out there that will give you straight DOM for a template. There is a project called HTMLBars which is promising and should eventually replace EmberJS's method of templating (which currently outputs a string that contains extra script tags that are used as markers for bound values). The plus side is that it's fast as what you're getting out of it is what you want to put in the page.

Clone DOM

Another way we could get a template is to clone a DOM structure and theoretically this should be even faster that outputting dom. This is the approach I outlined in my blog post about Optimizing often used templates and works well for Closure and PlastronJS as there may be operations between actually creating the DOM and putting it in the page so your data may have changed. This means that creating the template and filling it with data have to be two different operations whereas most traditional approaches will fill the template with data when it is created.

Speed

So knowing all this, let's see what the speed actually is like for a string vs dom based:

It's not the most scientific of results but you can see the test page code here.

// the string template function
var stringTemplate = function(data) {
 var str = '<div class="Header">';
 str += data.heading;
 str += '<div class="body"><div class="list-heading">';
 str += data.listName;
 str += '</div>';
 str +='<ul>';
 for (var i = 0; i < data.list.length; i++) {
  str += '<li>';
  str += data.list[0].name;
  str += '</li>';
 }
 str += '</ul></div></div>';
 return str;
};

// the dom template function
var domTemplate = function(data) {
 var base = document.createElement('div');
 var div = document.createElement('div');
 div.className = 'Header';
 div.innerText = data.heading;
 var body = document.createElement('div');
 body.className = 'body';
 var listHead = document.createElement('div');
 listHead.className = 'list-heading';
 listHead.innerText = data.listName;
 var list = document.createElement('ul');
 for (var i = 0; i < data.list.length; i++) {
  var li = document.createElement('li');
  li.innerText = data.list[i].name;
  list.appendChild(li);
 }
 body.appendChild(listHead);
 body.appendChild(list);
 base.appendChild(div);
 base.appendChild(body);
 return base;
};

Then for the string template I set it as an innerHTML and for the domTemplate I just run it. For the fastString test I added all the strings together before doing the innerHTML and for the binding I ran the domTemplate but also returned an object with functions that would change the values when needed. Times are in milliseconds and the number is the number of templates I generate

From this we can see that creating a pre-poulated DOM structure is certainly the way to go in terms of speed and even if we pass back functions that can handle the bindings it's still faster than using a string.

Data Binding

Data binding is important. It may be possible on simple sites to get away with re-rendering all your views but as soon as any meaningful user interaction is needed then you need to bind specific parts of your DOM to the data to keep the user state when data changes. Most libraries handle it natively and for those that don't you can try Rivets.js. To try and use this at work I wrote a handlebars to rivets converter.

Converting Handlebars to Rivets

I've since abandoned the project but it's interesting to look at what is needed in conversion because it gives some insight in to the differences between outputting DOM and outputting a string. For those that don't know how rivets works it looks at data attributes on nodes and then binds to them. This means you get conversions like this:

<div>{{name}}</div>
<!-- becomes -->
<div data-text="model.name"></div>

This works well for simple cases and the converter I wrote would handle a fair few cases like toggling class names:

<div class="fred {{#if bob}}foo{{/if}}">
<!-- becomes -->
<div class="fred" data-class-foo="bob">

The trickiest though were if statments that contained DOM nodes under them. There is no real "if" statement in rivets and the whole point of changing to the DOM was so that we wouldn't have to recreate new DOM nodes when there is a change in the model. To get around this the nodes under an if statement used the data-show attribute so they would be hidden when it evaluated to false. It actually worked rather well but there are three reasons why I'm abandoning it

1. There are better data binding tools out there - i.e. AngularJS
2. Rivets.js has computed properties which are essential to the converter but they don't take the values of the bound properties as arguments, instead you have to have those properties provided to the function separately through a closure which stop the functions being reused and makes it impossible to convert when the scope changes (say inside an each loop). To get around this I patched a version of rivetsjs that is included in the github repo
3. It was slow. Creating the DOM was fast as we only needed to clone a static DOM structure but then that structure had to be parsed to setup the bindings and those bindings run to populate the data. This made it slow at startup.

Logic

A big issue I take with templates is the amount of logic that is put in to them. You can read more of my thoughts in my Logic in templates post. This is a major reason handlebars is slower than other engines, because you can do so much with it. One of the things I wanted to do was reduce the logic in our templates and so I created Backbone.ModelMorph. The idea being that we create a new model from what we already have using a schema that holds our computed properties but still acts as a model sending out changes. This means we can take out the logic from the actual template and put it in a "presentation model" that sits between the template and the actual data model. This means we've separated all our concerns with the structure in the template, the data in the actual model and the presentation of the data in the ModelMorph.

So what's next?

Next up I'm going to be taking the guts from my handlebars converter and seeing if I can't make a simplified HTMLBars that will have less logic (say only one IF statement or EACH depth allowed) that can output a DOM stucture but also return an object that will allow you to hook up bindings. Only allowing one level should still give flexibility yet force people to think about how much logic they're putting in while making it much easier to write for me and also means that I won't have to deal with intricate layers of scope so the final function should also perform well.

So why am I calling it the unsolved mystery? Well as far as I know the correct way hasn't been found yet even though I have an idea of what I'd like. What will be interesting is to see how well it all fits in with the new <template> tag and web components.

Rendering large lists with Backbone

One of the first things that you do when starting with Backbone after sorting out your view hierarchy and memory management is an automatic way to display a collection. If you've read the blog post Backbone at Dataminr then you know we're using mixins to just tell a view that it needs to automatically list the collection for us. We found though that this was pretty slow for extremely large lists as it tries to keep everything in order and will add in sub views one at a time.

After having a look at where a lot of the time was being taken we could see it was in the calls to parseHTML and inserting the element in to the DOM. These were being called 100 times each for 100 elements, instead we wanted to batch all the HTML parsing and inserting to the dom together.

Below is the mixin we attach to a collection:

    Mixin.ComponentView.autoBigList = function(options) {

        // create the base element for the template
        var createEl = function(tagName, className, attributes, id, innerHTML) {
            var html = '<' + tagName;
            html += (className ? ' class="' + className + '"' : '');
            if (attributes) {
                _.each(attributes, function(val, key) {
                    html += ' ' + key + '="' + val + '"';
                });
            }
            html += (id ? ' id="' + id + '"' : '');
            html += '>' + innerHTML + '</' + tagName + '>';
            return html;
        };

        // get the needed variables
        this.after('initialize', function() {
            var iv = this.itemView.prototype;
            this._autobiglist = {
                html: '',
                template: Handlebars.compile(createEl(
                    iv.tagName,
                    iv.className,
                    iv.attributes,
                    iv.id,
                    (options.itemTemplate || iv.template)
                )),
                toAdd: []
            };
        });

        // setup and teardown listeners
        this.after('enterDocument', function() {
            this.listenTo(this.collection.on, 'reset', this.autlist_, this);
            this.listenTo(this.collection.on, 'add', this.autlist_, this);
            this.doneListing_ = _.debounce(this.doneListing_, 100);
            this.autolist_(this.collection);
        });

        this.setDefaults({
            // component view doesn't remove a decorated element, override
         rem_: function(model) {
          var view = _.find(this.getAllChildren(), function(child) {
           return child.model == model;
          });
          this.removeChild(view);
          view.$el.remove();
          view.dispose();
         },
            // collect the HTML together
            autolist_: function(model, silent) {
                var toAdd = this._autobiglist.toAdd;
                var template = this._autobiglist.template;
                if (model instanceof Backbone.Model) {
                    var setup = _.extend({}, options.setup);
                    setup.model = model;
                    var item = new this.itemView(setup);
                    toAdd.push(item);
                    if (!options.reverseOrder || this.reverseOrder)
                        this._autobiglist.html += template(item.serialize());
                    else
                        this._autobiglist.html = template(item.serialize()) + this._autobiglist.html;
                // if not single model, run each model
                } else if(model) {
                    $(this.getContentElement()).empty();
                    _.each(model.models, function(mod) {
                        this.autolist_(mod, true);
                    }, this);
                }
                if (silent !== true)
                    this.doneListing_();
            },
            // after all the HTML is collected put in DOM and attach views
            doneListing_: function() {
                var html = this._autobiglist.html;
                var toAdd = this._autobiglist.toAdd;
                if (!html)
                    return;
                // put html in document
                var div = document.createElement('div');
                div.insertAdjacentHTML("beforeend", html);
                var els = _.toArray(div.childNodes);
                var l = els.length;
                html = '';
                var frag = document.createDocumentFragment();
                for (var i = 0; i < l; i++) {
                    frag.appendChild(els[i]);
                };
                if (options.reverseOrder || this.reverseOrder) {
                    this.getContentElement().insertBefore(frag, this.getContentElement().firstChild);
                } else {
                    this.getContentElement().appendChild(frag);
                }
                // attach views
                for (i = 0; i < l; i++) {
                    var n = i;
                    if (options.reverseOrder || this.reverseOrder)
                        n = l - i - 1;
                 this.addChild(toAdd[n]);
                    toAdd[n].decorate(els[n]);
                };
                this._autobiglist.toAdd = [];
                this._autobiglist.html = '';
                if (this.afterList)
                 this.afterList();
            }
        });

    };

This can just be dropped in instead of our autolist mixin and works like a charm. So how does it work?

When we initialize the view for the collection we have a look at the defined ItemView and grab out any information Backbone uses to create a view's top level element: tagName, className, attributes & id. We save this along with the template function and create our own template function that will take a model's serialized object and return the full HTML back as a string.

Now that we can get the HTML for an item we need a way to collect these all together when our models come in. That why we've got these lines:

this.doneListing_ = _.debounce(this.doneListing_, 100);
this.autolist_(this.collection);

autolist_ is our function that will collect the HTML we need. We debounde doneListing so it is only called once we've collected all the HTML we need and save a second array which is the view for each bit of HTML

doneListing_ will create the DOM from the HTML which means we only need to parse the HTML once and then add it on to the dom. This is where our performance boost comes in. Once we've attached that though we still need to go through each of the new child nodes added and make that the HTML for the view. Using Backbone.ComponentView means we have a "decorate" function that does this for us but plain Backbone also has the setElement function that does the same thing.

And that's it, pretty simple. It should be noted though that this does not have any logic to handle a change in sort order, or if new items are inserted anywhere but the bottom (or the top) of the list. Removal should be fine though.

Hope that helps anyone with performance issues rendering large lists. If you want to know more about mixins, Backbone.Advice and Backbone.ComponentView see my blogpost on Backbone at Dataminr

Aspect Oriented Programming in JavaScript

You may have heard about AOP recently or about some of the new libraries that are coming out. The first I had heard about it was with Angus Croll's talk on How we learned to stop worrying and love JavaScript, although they just called it advice (and is now being used in Twitter Flight). So just what is Aspect-Oriented, and why do you care?

Aspect Oriented programming is actually a subset of Object Oriented programming. It helps in code re-use where there are cross-cutting concerns that don't fit well in to the single inheritance model. These can be things like logging which you may want to apply to objects throughout your program that don't share a common ancestor that would make sense to add the functionality to.

So what we really need is functionality (called "advice", which is what Angus named his library) and a mechanism to add it to an object (called a "pointcut") and these two things together are called an "aspect". In the speaker deck we are given "before", "after" and "around" as our pointcuts, and our Aspect is actually the function that contains these. It turns out that using functions for describing these aspects are quite useful.

I wrote a plugin for Backbone called Backbone.Advice which we use at work. We've created a file which contains a lot of these mixins that we can now apply across different contsructors. This has allowed us to separate out logic which was repeated in disparate parts of the system, and f you look at the github repo you will see some examples which will do things from automatically listing views to giving you keyboard navigatable lists. Aspects turn out to be very handy.

So now the downsides. Using AOP makes it extremely difficult to debug. You will use functionality across different parts in your system but not necessarily know where it's going to be put when writing. You will also have a fairly ugly callstack, especially if you apply a few different aspects.

So how do we get around this? The trick is to keep the aspects simple and testable. If you have a look at the mixins you can see that often we're making another call to this.mixin(Mixin....); which gives us a sort of inheritance structure. We're also careful about our naming, keeping things consistent across aspects. Also we only allow mixing in on a constructor and not an instance which means we can easily find out what mixins are being used.

Some other AOP javascript libraries you can look in to:

• YouAreDaChef
• meld
• Aop.js

Monads in plain JavaScript

So you want to know what these Monads thing is? Douglas Crockford in an interview said that if you understand them you lose the ability to explain them, and then went on to do a Monads & Gonads talk. In the talk he said you don't have to understand category theory or Haskell and I happen to agree, but then he jumps in to some generic monad constructor that may be confusing at first. So what's a monad in plain english?

Simply put it's a wrapper around a value. There are also some laws that it should conform to, but more on that later.

As with any wrapper we need two methods, one to get and one to set the value. These are called "return" and "bind". Let's construct the "Maybe" monad - the easiest monad to get your head around:

var Maybe = function(value) {
  this.value = value;
};

So we have a constructor, monads are something called a "functor" which basically is a "functional object" that allows you to move values between sets. In this case the maybe monad will allow us to use "undefined" in places which expect an actual value and we would see an error otherwise.

So let's look at the return function:

Maybe.prototype.ret = function() {
  return this.value;
};

Nice and easy, it gives us a method to get the value of the monad. Now comes the fun part, bind:

Maybe.prototype.bind = function(fn) {
  if (this.value != null)
    return fn(this.value);
  return this.value;
};

So the bind function runs a given function with the value of the monad. In the case of the maybe monad it just skips running the function if the value doesn't exist - and that's it!

Now we should talk about "lift". you'll see that the bind will return whatever the function returns for the value. Well we want a monad as a return so you should be passing in functions that return a monad - but who wants to do that? instead we'll just create a "lift" function that can take in a function that returns a normal value and changes it to a monad. pretty easy, it'd look something like this:

Maybe.lift = function(fn) {
  return function(val) {
    return new Maybe(fn(val));
  };
};

so now we can do things like have an addOne function:

var addOne = function(val) {
  return val + 1;
};

lift it:

var maybeAddOne = Maybe.lift(addOne);

then you can use it with bind! But what if we have a function that takes two monads? say we want to add two together?

Maybe.lift2 = function(fn) {
  return function(M1, M2) {
    return new Maybe(M1.bind(function(val1) {
      return M2.bind(function(val2) {
        return fn(val1, val2);
      });
    }));
  };
};

This one's a bit more complicated, but basically it just uses closures to get the values from the two monads before running it through the function, and because it's a maybe monad it will just pass back undefined  so we can safely use undefined values without errors. You can try it out like this:

var add = function(a, b) {return a + b;};
m1 = new Maybe(1);
m2 = new Maybe(2);
m3 = new Maybe(undefined);

var liftM2Add = Maybe.lift2(add);

liftM2Add(m1, m2).ret(); //3
liftM2Add(m3, m2).ret(); //undefined
liftM2Add(m1, m3).ret(); //undefined

And that's it. So to recap a monad is just a container. You can pass in a function to operate on it and get a returned value, or ask for it's value. You've probably used monads in the past without knowing (like promises, you just send it a function to run on it's value and it returns back itself - so it's a lifted bind) and perhaps even created some.