Monolithic Node.js

Are large-scale Node.js systems possible? Empirically, the answer is yes. Walmart and Paypal have both shown that it can be done. The quick criticism is that you need 10X engineers. This a classic, and well-founded criticism. New ways of doing things are often found to be exceptionally productive, precisely because brilliant minds self-select for the new and interesting.

So let’s rephrase the question. Are large-scale Node.js systems possible with mainstream developers? If you believe that these large-scale Node.js systems will resemble the large-scale Java and .Net systems you have known and loved, then the answer is, emphatically, no. JavaScript is far too weak a language to support the complexity inherent in systems of such scale. It’s not exactly type-safe, and half the language is unusable. There’s a reason the best-selling book on the language is called JavaScript, The Good Parts.

Despite this, we’ve managed to build quite a few large-scale systems at my company, nearForm. Here’s what we’ve learned, and how we do it.

The defining attribute of most large-scale, mainstream traditional systems is that they are monolithic. That is, a single large codebase, with many files, thousands of classes, and innumerable configuration files (in XML, if you’re lucky). Trying to build a system like this in JavaScript is indeed the path to madness. The visceral rejection of Node.js that you see from some quarters is often the spidey-sense of an experienced enterprise developer zapping them between the eyes. JavaScript? No! This reaction is entirely justified. Java and .Net have been designed to survive enterprise development. They enable monolithic architecture.

There are of course systems built in Java and .Net that are not monolithic, that are more structured. I’ve built in both styles myself. But it takes effort, and even the best systems fall to technical debt over time. It’s too easy to fall back into the monolithic trap.

Monolithic Systems are Bad

What is so bad about monolithic systems anyway? What does it mean for a system to be “monolithic”? The simplest definition is a system that cannot survive the loss of any of its parts. You pull one part out, and the whole thing fails. Each part is connected to the others, and interdependent on them.

The term monolith means single stone, and is derived from the ancient greek. The ancient city of Petra in modern-day Jordan is one of the best examples of monolithic architecture. Its buildings are constructed in one piece, hewn directly from the cliff face of a rocky mountain. It also provides a perfect example of the failure mode of monolithic systems. In 363AD an earthquake damaged many of the buildings, and the complex system of aqueducts. As these were carved directly into the mountain, they were impossible to repair, and the city fell into terminal decline.

So it goes with monolithic software. Technical debt, the complexity built into the system over time, makes the system impossible to repair or extend at reasonable cost as the environment changes. You end up with things like month-long code freezes in December so that the crucial Christmas shopping season is not affected by unknowable side-effects.

The other effect of monolithic software is more pernicious. It generates software development processes and methodologies. Because the system has so many dependencies, you must be very careful how you let developers change it. A lot of effort must be expended to prevent breakage. Our current approaches, from waterfall to agile, serve simply to enable monolithic systems. They enable us to build bigger and add more complexity. Even unit testing is an enabler. You thought unit testing was the good guy? It’s not. If you do it properly, it just lets you build bigger, not smarter.

Modular Systems are Good

So what are we supposed to do, as software developers, to avoid building monolithic systems. There are no prizes for knowing the answer. Build modular systems! The definition of a modular system is simply the inverse: each part stands alone, and the system is still useful when parts are missing.

Modular software should therefore be composed of components, each, by definition, reusable in many contexts. The idea of reusable software components is one of the Holy Grails of software development.

The greatest modular system humanity has created to date is the intermodal shipping container. This is a steel box that comes in a standard set of sizes, most commonly 8′ wide, 8’6” tall, and 20 or 40 feet long. This standardisation enables huge efficiency in the transport of goods. Each container is re-usable and has a standardised “API”, so to speak.

Sadly, software components are nothing like this. Each must necessarily have it’s own API. There are dependency hierarchies. There are versioning issues. Nonetheless, we persist in trying to build modular systems, because we know it is the only real way to deal with complexity.

There have been some success stories, mostly at the infrastructure level. UNIX pipes, and the UNIX philosophy of small programs that communicate over pipes, works rather well in practice. But it only takes you so far.

Other attempts, such as CORBA, or Microsoft’s OLE, have suffered under their own weight. We’ve grown rather fond of JSON-based REST services in recent years. Anyone who’s been at the mercy of third party APIs, and I’m looking at you, Facebook, will know that this is no promised-land either.

Objects are Hell

The one big idea for modular software that seems to have stuck to the wall, is the object-oriented paradigm.

Objects are supposed to be reusable representations of the world, both abstract and real. The tools of object-oriented development; interfaces, inheritance, polymorphism, dynamic methods, and so on, are supposed to provide us with the power to represent anything we need to build. These tools are supposed to enable us to build objects in a modular reusable way.

The fundamental idea of objects is really quite broken when you think about it. The object approach is to break the world into discrete entities with well-defined properties. This assumes that the world will agree to being broken up in such a way. Anyone who has tried to create an well-designed inheritance hierarchy will be familiar with how this falls apart.

Let’s say we have a Ball class, representing, well, a ball. We then define a BouncingBall, and a RollingBall, both inheriting from the base Ball class, each having suitable extensions of properties and methods. What happens when we need a ball than can both bounce and roll

Admittedly, inheritance is an easy target for criticism, and the solution to this problem is well-understood. Behaviours (bouncing and rolling) are not essential things, and should be composed instead. That this is known does not prevent a great deal of inheritance making it’s way into production systems. So the problem remains.

Objects are really much worse than you think. They are derived from a naïve mathematical view of the world. The idea that there are sets of pure, platonic things, all of which share the same properties and characteristics. On the surface this seems reasonable. Scratch the surface and you find that this idea breaks down in the face of the real world. The real world is messy. It even breaks down in the mathematical world. Does the set of all sets that do not contain themselves contain itself? You tell me.

The ultimate weakness of objects is that they are simply enablers for more monolithic development. Think about it. Objects are grab bags of everything a system needs. You have properties, private and public. You have methods, perhaps overridden above or below. You have state. There are countless systems suffering from the Big Ball of Mud anti-pattern, where a few enormous classes contain most of the tangled logic. There are just too many different kinds of thing that can go into an object.

But objects have one last line of defense. Design patterns! Let’s take a closer look at what design patterns can do for us.

Bad Patterns are Beautiful

In 1783 Count Hans Axel von Fersen commissioned a pocket watch for the then Queen of France, Marie Antoinette. The count was known to have had a rather close relationship with the Queen, and the extravagance of the pocket watch suggests it was very close indeed. The watch was to contain every possible chronometric feature of the day; a stopwatch, an audible chime, a power meter, and a thermometer, among others. The master watchmaker, Abraham-Louis Breguet was tasked with the project. Neither Marie Antoinette, Count Fersen, nor Breguet himself lived to see the watch completed. It was finally finished in 1837, by Breguet’s son. It is one of the most beautiful projects to have been delivered late and over-budget.

It is not for nothing that additional features beyond basic time keeping are known as complications in the jargon of the watchmaker. The watches themselves possess a strange property. The more complex they are, the more intricate, the more monolithic, the more beautiful they are considered. But they are not baroque without purpose. Form must follow function. The complexity is necessary, given their mechanical nature.

We accept this because the watches are singular pieces of artistry. You would find yourself under the guillotine along with Marie Antoinette in short order if you tried to justify contemporary software projects as singular pieces of artistry. And yet, as software developers, we revel in the intricacies we can build. The depth of patterns that we can apply. The architectures we can compose.

The complexity of the Marie Antoinette is seductive. It is self-justifying. Our overly complex software is seductive in the same way. What journeyman programmer has not launched headlong into a grand architecture, obsessed by the aesthetic of their newly imagined design? The intricacy is compounded by the failings of their chosen language and platform.

If you have built systems using one of the major object-oriented languages, you will have experienced this. To build a system of any significant size, you must roll out your Gang-of-Four design patterns. We are all so grateful for this bible that we have forgotten to ask a basic question. Why are design patterns needed at all? Why do you need to know 100+ patterns to use objects safely? This is code smell!

Just because the patterns work, does not mean they are good. We are treating the symptoms, not the disease. There is truth in the old joke that all programming languages end up being incomplete, buggy versions of LISP. That’s pretty much what design patterns are doing for you. This is not an endorsement of functional programming either, or any language structure. They all have similar failings. I’m just having a go at the object-oriented languages because it’s easy!

Just because you can use design patterns in the right way does not mean using design patterns is the right thing to do. There is something fundamentally wrong with languages that need design patterns, and I think I know what it is.

But before we get into that, let’s take a look at a few things that have the right smell. Let’s take a look at the Node.js module system.

Node.js Modules are Sweet

If you’ve built systems in Java or .Net, you’ll have run into the dreaded problem of dependency hell. You’re trying to use component A, which depends on version 1 of component C. But you’re also trying to use component B, which depends on version 2 of component C. You end up stuck in a catch-22, and all of the solutions are a kludge. Other platforms, like Ruby or Google’s new Go language may make it easier to find and install components, but they don’t solve this problem either.

As an accident of history, JavaScript has no built-in module system (at least, not yet). This weakness has turned out to be a hidden strength. Not only has it created the opportunity to experiment with different approaches to defining modules, but it also means that all JavaScript module systems must survive within the constraints of the language. Modules end up being local variables that do not pollute the global namespace. This means that module A can load version 1 of module C, and module B can load version 2 of module C, and everything still works.

The Node Package Manager, npm, provides the infrastructure necessary to use modules in practice. As a result, Node.js projects suffer very little dependency hell. Further, it means that Node.js modules can afford to be small, and have many dependencies. You end up with a large number of small focused modules, rather than a limited set of popular modules. In other platforms, this limited set of popular modules end up being monolithic because they need to be self-sufficient and do as much as possible. Having dependencies would be death.

Modules also naturally tend to have a much lower public API to code ratio. They are far more encapsulated than objects. You can’t as a rule misuse them in the same way objects can be misused. The only real way to extend modules is to compose them into new modules, and that’s a good thing.

The Node.js module system, as implemented by npm, is the closest anybody has come in a long time to a safe mechanism for software re-use. At least half the value of the entire Node.js platform lies in npm. You can see this from the exponential growth rate of the number of modules, and the amount of downloads.

Node.js Patterns are Simple

If you count the number of spirals in the seed pattern at the centre of a sunflower, you’ll always end up with a fibonacci number. This is a famous mathematical number sequence, where the next fibonacci number is equal to the sum of the previous two. You start with 0 and 1, and it continues 1, 2, 3, 5, 8, 13, 21, 34, and so on. The sequence grows quickly, and calculating later fibonacci numbers is CPU intensive due to their size.

There’s a famous blog post attacking Node.js for being a terrible idea. An example is given of a recursive algorithm to calculate fibonacci numbers. As this is particularly CPU intensive, and as Node.js only has one thread, the performance of this fibonacci service is terrible. Many rebuttals and proposed solutions later, it is still the case that Node.js is single-threaded, and CPU intensive work will still kill your server.

If you come from a language that supports threads, this seems like a killer blow. How can you possibly build real systems? The are two things that you do. You delegate concurrency to the operating system, using processes instead of threads. And you avoid CPU intensive tasks in code that needs to respond quickly. Put that work on a queue and handle it asynchronously. This turns out to be more than sufficient in practice.

Threads are notoriously difficult things to get right. Node.js wins by avoiding them altogether. Your code becomes much easier to reason about.

This is the rule for many things in Node, when compared to object-oriented languages. There is simply no need for a great many of the patterns and architectures. There was a discussion recently on the Node.js mailing list about how to implement the singleton pattern in JavaScript. While you can do this in JavaScript using prototypical inheritance, there’s really very little need in practice, because modules tend to look after these things internally. In fact, the best way to achieve the same thing using Node.js is to implement a standalone service that other parts of your system communicate with over the network.

Node.js does require you to learn some new patterns, but they are few in number, and have broad application. The most iconic is the callback pattern, where you provide a function that will be called when the system has more data for you to work with. The signature of this function is always: an error object first, if there was an error. Otherwise the first argument is null. The second argument is always the result data.

The callback function arises naturally from the event handling loop that Node.js uses to dispatch data as it comes in and out of the system. JavaScript, the language, designed for handling user interface events in the browser, turns out to be well-suited to handling data events on the server-side as a result.

The first thing you end up doing with Node.js when you start to use it is to to create callback spaghetti. You end up with massively indented code, with callback within callback. After some practice you quickly learn to structure your code using well-named functions, chaining, and libraries like the async module. In practice, callbacks, while they take some getting used to, do not cause any real problems.

What you do get is a common interface structure for almost all module APIs. This is in stark contrast to the number of different ways you can interact with object-oriented APIs. The learning surface is greatly reduced.

The other great pattern in Node.js is streams. These are baked into the core APIs, and they let you manipulate and transform data easily and succinctly. Data flows are such a common requirement that you will find the stream concept used all over the place. As with callbacks, the basic structure is very simple. You pipe data from one stream to another. You compose data manipulations by building up sets of streams connected by pipes. You can even have duplex streams that can read and write data in both directions. This abstraction leads to very clean code.

Because JavaScript is a semi-functional language, and because it does not provide all the trappings of traditional object-oriented code, you end up with a much smaller set of core patterns. Once you learn them, you can read and understand most of the code you see. It is not uncommon in Node.js projects to review the code of third party modules to gain a greater understanding of how they work. The effort you need to expend to do this is substantially less than for other platforms.

Thinking at the Right Level

Our programming languages should let us think at the right level, the level of the problems we are trying to solve. Most languages fail miserably at this. To use an analogy, we’d like to think in terms of beer, but we end up thinking in terms of the grains that were used to brew the beer.

Our abstractions are at too low a level, or end up being inappropriate. Our languages do not enable us to easily compose their low level elements into things at the right level. The complexity in doing so trips us up, and we end up with broken, leaky abstractions.

Most of the time, when we build things with software, we are trying to model use cases. We are trying to model things that happen in the world. The underlying entities are less important. There is an important data point in the observation that beginning programmers write naïve procedural code, and only later learn to create appropriate data structures. This is telling us something about the human mind. We are able to get things done by using our intelligence to accommodate differences in the entities that make up our world.

A bean-bag chair is still a chair. Every human knows how to sit in one. It has no back, and no legs, but you can still perform the use-case: sitting. If you’ve modeled a chair as a object with well-defined properties, such as assuming it has legs, you fail in cases like these.

We know that the code to send an email should not be tightly coupled to the API of the particular email sending service we are using. And yet if you create an abstract email sending API layer, it inevitably breaks when you change the implementation because you can’t anticipate all the variants needed. It’s much better to be able to say, “send this email, here’s everything I’ve got, you figure it out!”

To build large-scale systems you need to represent this action-oriented way of looking at the world. This is why design patterns fail. They are all about static representations of perfect ontologies. The world does not work like that. Our brains do not work like that.

How does this play out in practice? What are the new “design patterns”? In our client projects, we use two main tools: micro-services, and pattern matching.

Micro-Services Scale

We can use biological cells as an inspiration for building robust scalable systems. Biological cells have a number of interesting properties. They are small and single-purpose. There are many of them. They communicate using messages. Death is expected and natural.

Let’s apply this to our software systems. Instead of building a monolithic 100 000 line codebase, build 100 small services, each 100 lines long. Fred George, (the inventor of programmer anarchy) one of the biggest proponents of this approach, calls these small programs micro-services.

The micro-services approach is a radically different way of building systems. The services each perform a very limited task. This has the nice effect that they are easy to verify. You can eye-ball them. Testing is much less important.

On the human side, it also means that the code is easy to rewrite, even in a different language. If a junior engineer writes a bad implementation, you can throw it away. They can be written in pretty much any language. If you don’t understand the code, you throw it away and rewrite. Micro-services are easy to replace.

Micro-services communicate with each other by sending messages. You can send these messages directly over internal HTTP, or use a message queue for more scale. In fact, the transport mechanism does not matter all that much. From the perspective of the service, it just deals with whatever messages come it’s way. When you’re building services in Node.js, JSON is the most natural formatting choice. It works well for other languages too.

They are easy to scale. They offer a much finer grained level of scaling then simply adding more servers running a single system. You just scale the parts you need. We’ve not found the management of all these processes to be too onerous either. In general you can use monitoring utilities to ensure that the correct number of services stay running.

Death becomes relatively trivial. You’re going to have more than one instance of important services running, and restarts are quick. If something strange happens, just die and restart. In fact, you can make your system incredibly robust if you build preprogrammed death into the services, so that they die and restart randomly over time. This prevents the build up of all sorts of corruption. Micro-services let you behave very badly. Deployments to live systems are easy. Just start replacing a few services to see what happens. Rolling back is trivial – relaunch the old versions.

Micro-services also let you scale humans, both at the individual and team level. Individual brains find micro-services much easier to work with, because the scope of consideration is so small, and there are few side-effects. You can concentrate on the use-case in hand.

Teams also scale. It’s much easier to break up the work into services, and know that there will be few dependencies and blockages between team members. This is really quite liberating when you experience it. No process required. It flows naturally out of the architecture.

Finally, micro-services let you map your use-cases to independent units of software. They allow you to think in terms of what should happen. This let’s you get beyond the conceptual changes that objects impose.

Pattern Matching Rules

Micro-services can bring you a long way, but we’ve found that you need a way to compose them so that they can be reused and customised. We use pattern matching to do this.

This is once more about trying to think at the right level. The messages that flow between services need to find their way to the right service, in the right form, with the right preparation.

The pattern matching does not need to be complex. In fact, the simpler the better. This is all about making systems workable for human minds. We simply test the values of the properties in the message, and if you can match more properties than some other service, you win.

This simple approach makes it very easy to customise behaviour. If you’ve ever had to implement sales tax rules, you’ll know how tricky they can be. You need to take into account the country, perhaps the state, the type of good, the local bylaws. Patterns make this really easy. Start with the general case, and add any special cases as you need them. The messages may or may not contain all the properties. It’s not a problem, because special properties are only relevant for special cases anyway.

Cross-cutting concerns are also easy to support with pattern matching. For example, to log all the message related to saving data, simply grab those as they appear, make the log entry, and then send the message on its way. You can add permissions, caching, multiple databases. All without affecting the underlying services. Of course, some work is needed to layer up the pattern matching the way you need it, but this is straightforward in practice.

The greatest benefit that we have seen is the ability to compose and customise services. Software components are only reusable to the extent that they can be reused. Pattern matching lets you do this in a very decoupled way. Since all you care about is transforming the message in some way, you won’t break lower services so long as your transformations are additive.

A good example here is user registration. You might have a basic registration service that saves the user to a database. But then you’ll want to do things like send out a welcome email, configure their settings, verify their credit card, or any number of project-specific pieces of business logic. You don’t extend user registration by inheriting from a base class. You extend by watching out for user registration messages. There is very little scope for breakage.

Obviously, while these two strategies, micro-services, and pattern matching, can be implemented and used directly, it’s much easier do so in the context of a toolkit. We have, of course, written one for Node.js, called Seneca.

Galileo’s Moons

We’re building our business on the belief that the language tools that we have used to build large systems in the past are insufficient. They do not deliver. The are troublesome and unproductive.

This is not surprising. Many programming languages, and object-oriented ones in particular, are motivated by ideas of mathematical purity. They have rough edges and conceptual black holes, because they were easier to implement that way. JavaScript is to an extent guilty of all this too. But it is a small language, and it does give us the freedom to work around these mistakes. We’re not in the business of inventing new programming languages, so JavaScript will have to do the job. We are in the business of doing things better. Node.js and JavaScript help us do that, because they make it easy to work with micro-services, and pattern matching, our preferred approach to large-scale systems development.

In 1610, the great italian astronomer, Galileo Galilei, published a small pamphlet describing the discoveries he had made with his new telescope. This document, Sidereus Nuncius (the Starry Messenger) changed our view of the world.

Galileo had observed that four stars near the planet Jupiter behaved in a very strange way. They seemed to move in a straight line backwards and forwards across the planet. The only reasonable explanation is that there were moons orbiting Jupiter, and Galileo was observing them side-on. This simple realisation showed that some celestial bodies did not orbit the earth, and ultimately destroyed the Ptolemaic theory, that the sun orbited the earth.

We need to change the way we think about programming. We need to start from the facts of our own mental abilities. The thought patterns we are good at. If we align our programming languages with our abilities, we can build much larger systems. We can keep one step ahead of technical debt, and we can make programming fun again.

Posted in Uncategorized | 59 Comments

How to Make Simple Node.js Modules Work in the Browser

Node.js is all about writing small, simple modules that do one thing, and do it well. This can be taken to extremes by crazy people. There’s even a module for multi-line strings!

Some people can’t resist writing frameworks either. This post is not for them.

Actually, it is. I wrote the first version of Seneca, my micro-services framework, suffering from Java withdrawal. It’s too big and needs to be refactored into small modules. That’s what I’m doing now. As it happens some of these modules seem like they could be useful on the front-end, so I decided to see if I could make them browser compatible.

Now, the right way to do this is to use browserify. But where there’s a right way, there’s always a wrong way too. This is it. Sometimes, you just need things to be standalone. It’s not every coder that’s lucky enough to use Node.js, you know.

Instead of reinventing the wheel, I went straight to my number one module crush, underscore, the utility belt of JavaScript. I’m so much in love with this module I automatically bang out npm install underscore at the start of a new project. It feels like part of the language. Underscore started out as just a browser library, and then had to support Node.js. What you have here is exactly what you need:

  • Main file has to work in the browser, and be a Node module;
  • Tests have to be run in the browser, and for Node;
  • You need to provide a minified version;
  • Good choices (one assumes!) for third party tools to do all this.

Go take a look at the github repo. You’re going to copy the approach. This post just documents it so you don’t have to figure it out for yourself.

Let’s start with the final product. Here are three small modules, each of which can run in the browser or Node. They can be installed with npm, or bower.

  • jsonic – A JSON parser for Node.js that isn’t strict
  • gex – Glob Expressions for JavaScript
  • patrun – A fast pattern matcher on JavaScript object properties.

You can use any of these as a template. They all follow the same approach. Here’s what to do.

Browserize/Nodize your Code

You may be starting from either a Node module, or a browser library. In both cases, you need to wrap your code up into a single file. Assuming you’re calling your module mymodule, here’s the start of that file, which you’ll call mymodule.js:

"use strict";

(function() {
  var root = this
  var previous_mymodule = root.mymodule

Let’s dissect the code. You should always “use strict”;. No excuses.

Oh. as an aside, I’m an inveterate avoider of semi-colons. Why can’t we all just, …, get along?

The (function() { isolates your code from anything else in the browser, and has no effect on the Node module. You’re going to call this anonymous function with the current context at the end of the module, which in the browser makes this equal to the window object:

}).call(this);

OK, back to the top. The next thing you’re doing is storing the context (this) in a variable name root, and keeping a reference to any previous variable with the same name as your module. This lets you provide the noConflict feature, like jQuery.

Your module is either going to be a JavaScript object or function (not much point otherwise!). Define it like so, inside the isolation function:

var mymodule = function() {
  ...
}

mymodule.noConflict = function() {
  root.mymodule = previous_mymodule
  return mymodule
}

Now users of your module, in the brower, can call noConflict to avoid conflicts over the mymodule name:

var othername = mymodule.noConflict()
// the variable mymodule is back to its old value from here

How do you export your module into the browser? Or for Node.js? By checking for the existence of module.exports, and acting accordingly. Again, this goes inside your isolation function.

  if( typeof exports !== 'undefined' ) {
    if( typeof module !== 'undefined' && module.exports ) {
      exports = module.exports = mymodule
    }
    exports.mymodule = mymodule
  } 
  else {
    root.mymodule = mymodule
  }

If you’re in a Node.js context, you’ll end up exporting in the normal way: module.exports = mymodule. If you’re in the browser, you’ll end up setting the mymodule property on the window object.

There’s one final niggle. What if you depend on other modules, such as, say, underscore? Place this near the top of your isolation function:

  var has_require = typeof require !== 'undefined'

  var _ = root._

  if( typeof _ === 'undefined' ) {
    if( has_require ) {
      _ = require('underscore')
    }
    else throw new Error('mymodule requires underscore, see http://underscorejs.org');
  }

This code checks for the existence of the require function, and uses require if it exists. Otherwise it assumes you’ve loaded the dependency via a script tag (or a build process), and complains with an exception if the dependency can’t be found.

That’s pretty much all you need on the code front. Check out the source code of the jsonic, gex, and patrun modules on github for real-world examples.

Also, don’t forget to do

npm init

if your project needs a package.json file for Node.

Test Your Code

The jasmine framework is the one to use here. It works nicely both for Node and the browser. Create a test folder and write a mymodule.spec.js file containing your tests:

if( typeof mymodule === 'undefined' ) {
  var mymodule = require('..')
}

describe('mymodule', function(){

  it('something must be done', function(){
    expect( mymodule() ).toBe( 'doing something' )
  })

})

The test file should check to see if the module itself needs to be required in. This is for the Node case, where the module package.json is in the parent folder. In the browser case, you’ll load the module using a script tag before you load this test file, so it’s already defined.

It’s a good idea to get your project built automatically by the Travis-CI service. Follow the Travis-CI instructions to get this working. You’ll need to ensure that Travis can run the jasmine tests, so you’ll need to do

$ npm install jasmine-node --save-dev

This makes sure that jasmine get’s installed locally inside the projects node_modules folder. Which means that in your package.json, you can say:

  "scripts": {
    "test": "./node_modules/.bin/jasmine-node ./test",
  }

and then the command line

npm test

will run your jasmine tests for you, and they will be run by Travis-CI for each commit, which is what you really want.

This setup only tests your code in a Node.js context. Now you need to test in a browser context too. Actually, you need two types of browser test. One you can automate, and one you can run manually in browsers on your machine. The automated test comes courtesy of phantomjs, which is a “headless” browser environment (there’s no visible GUI). This runs your tests in a WebKit browser engine, so it’s pretty much like testing in Chrome or Safari (to a first approximation).

Install phantomjs globally. You don’t need a local copy, as you don’t need it for Travis-CI, and you’ll probably use it for other projects too:

$ sudo npm install phantomjs@1.9.1-0 -g

(At the time of writing, Sep 2013, the latest version of phantomjs does not install due to a bug, so I’m explicitly specifying a known-good version.)

To actually run the tests, you’ll need some supporting infrastructure. This consists of the jasmine library code, a “runner” script, and a HTML file to load it all. Your best bet is to copy the files from one of my example repos (see above). The files are:

  • test/jasmine.html
  • test/run-jasmine.js
  • test/jasmine-1.3.1/*

In jasmine.html, you load up everything you need, and execute the test using the jasmine API. It’s all boilerplage, just replace mymodule anywhere it occurs with the name of your module. If you have additional dependencies, add them under the script tag that loads underscore (this assumes you’ve done an npm install underscore –save already):

<!DOCTYPE html>
<html>
<head>
  

  <link href="jasmine-1.3.1/jasmine.css" rel="stylesheet">

  <script src="../node_modules/underscore/underscore.js"></script>

  <script src="../mymodule.js"></script>

  <script src="jasmine-1.3.1/jasmine.js"></script>
  <script src="jasmine-1.3.1/jasmine-html.js"></script>

  <script src="mymodule.spec.js"></script>

  <script type="text/javascript">
    (function() {
      var jasmineEnv = jasmine.getEnv();

      ... jasmine boilerplate ... 

      function execJasmine() {
        jasmineEnv.execute();
      }

    })();
  </script>
</head>
<body></body>
</html>

Now you add a custom script to your package.json:

  "scripts": {
    "test": "./node_modules/.bin/jasmine-node ./test",
    "browser": "phantomjs test/run-jasmine.js test/jasmine.html",
   }

And run it with:

$ npm run-script browser

And your tests should execute as before.

To test in a real browser, just load the test/jasmine.html file directly. This will execute the tests immediately and display some nicely formatted results.

Publish your Module

Time to inflict your code on the world! Muhaha, etc.

First, you’ll want to minify it for use in browsers. Use uglifyjs2 for this. Note: use version 2! Again, no need to install this locally, as it’s a development tool you’ll want to use for other projects:

sudo npm install uglify-js -g

And then in your package.json:

  "scripts": {
    ...
    "build": "uglifyjs mymodule.js -c \"evaluate=false\" -m --source-map mymodule-min.map -o mymodule-min.js"
  },

This generates mymodule-min.js, and also mymodule-min.map, a source map, which makes debugging in the browser easier.

Now you’re ready. Make sure you’ve filled in all the details in package.json, and, in particular, have chosen a version number, say 0.1.0 to start. Then publish!

$ npm publish

If you’re not yet registered with npm, you need to do so first.

Next, you should also publish your module onto bower.io, which is like npm for browser libraries. Bower uses your github repository tags to generate versions.

So, first, generate a tag using the same version number (unless you are evil):

$ git tag 0.1.0
$ git push --tags

Make sure you’ve committed everything before you do this!

And then register with bower:

$ bower register mymodule git://github.com/your-account/mymodule.git

And now you can do bower install mymodule, as well as npm install mymodule.

If you have improvements to this approach, let me know in the comments!

Posted in Node.js | 2 Comments

Introducing NodeZoo.com, a search engine for Node.js modules

I made a bet on a new programming platform 3 years ago, and it paid off. Every line of code that has earned me money since then has been run by Node.js. In case you missed it, Node.js is the evil step-child of Netscape and Gmail that is going to take over software development for the next decade. Starting now.

Brendan Eich invented the JavaScript programming language in 10 days during a death-march in the late 90’s while working at Netscape. It’s both utterly broken and brilliantly productive. Google invented Gmail, and then realized it was too slow. So they wrote a fast web browser, Chrome. Along the way, they wrote an entirely new JavaScript engine, called V8, that is damn fast. Now, V8 can be taken out of Chrome, and used by itself. Throw in some scalable event-driven architecture, and you get the most badass way to build web stuff going. That’s Node.js. Oh, and it does flying robots too…

The best thing about Node.js is the module system, npm. This is separate, and emerged slightly later than Node itself. It comprehensively solves the “versioning problem”. When you break software into modules, you end up with many versions of these modules. As time goes by, different modules depend on different versions of other modules, and it gets messy.

Think of it like this, Rachel invites Joey to a party in her apartment, and Joey in turn invites Phoebe. Rachel also invites Chandler, who in turn invites Ursula, Phoebe’s evil twin sister, who is pretending to be Phoebe. The party ends up with incompatible guests, and fails. Here’s what NPM does. The party is split between two apartments. One apartment gets Joey and Phoebe, the other Chandler and Ursula. Rachel hangs out in the hallway and is none the wiser. Happy days.

When you publish a Node.js module to npm, you specify the other modules you depend on, and the versions that you can work with. npm ensures that all modules you install only see compatible dependencies. No more breakage.

There’s another great thing about the Node.js module system. It emerged organically from the nature of the JavaScript language and the founding personalities (npm was started by @izs). Node.js modules are small. Really small.  Which is great, because that makes Node.js anti-fragile. There are no dependencies on standards, no need for curation, no need for foundations that bless certain modules, no blockage when module authors dig their heels in. The system tolerates bad coders, module abandonment, personality implosions, and breaking changes. With 23,000 plus modules at the time of writing, you’ll always find what you need. You might have to fix a few things, tweak a few things, but that’s better than being completely stuck.

This wonderful anarchy does introduce a new problem. How do you find the right module? There’s a chicken and egg period when you’re new to Node – it seems like there are ten options for almost anything you want to do.

The question is how to solve this problem in a scalable way – not everybody can go to all the conferences or hang out on IRC – although you really should if you can. “Ask the community” doesn’t scale, and the latency is pretty bad too. Also, if your goal is pick one module, the über-helpfulness of the Node community sort of works against that, as you’ll get more than one recommendation! The npm site npmjs.org delegates the search question to Google. The results are less useful than you’d think. Google’s algorithms don’t give us what we need, and the search results, in terms of scannability, are pretty lame. The npm command line has a free text search function. It’s nice, but the results are pre-Google internet quality, and for the same reasons – free text search doesn’t do great finding relevant results. Then there’s the Node Toolbox, which is like a 90’s Yahoo for Node. There’s a human limit to curation and the amount of modules that can be categorized. Ontology building is, frankly, Sisyphusarbeit.

This situation is itchy. Just annoying enough to make you write some code to solve it. Towards the end of last year I randomly ended up reading that wonderful article “The Anatomy of a Large-Scale Hypertextual Web Search Engine” – written back when Larry and Sergey still had Stanford email addresses. The thing that hits you is how simple the idea of PageRank is: if popular web pages point to your web page, your web page must be really good! And the math is just lovely and so … simple! It should have been obvious to everyone.

In a gross misapplication of the underlying mathematical model (random web surfing), the same idea can be applied to Node.js modules to generate a NodeRank – a measure of how awesome your module is. How does it work. Modules depend on other modules. If lots of modules depend on a particular module, then that module must be pretty popular. A good example is express, a web framework. But that’s not enough! The algorithm asks you to look further. The modules that express itself depends on are more popular still. Case in point, connect, a HTTP server framework. The connect module needs to get some NodeRank juice from the express module. That’s what the algorithm does: your module is awesome if it’s used by other awesome modules!

Implementing the algorithm is tricky. But Google to the rescue! (ironic, capital I). I found a great blog post, with python code, that explains how to calculate a fast approximation. Thanks Michael Nielsen! Of course, a little part of me was betting a Node.js port would run even faster (it did, much faster!). So I hacked up an implementation.

Now, You can pull down the entire npm registry, it’s just a CouchDB database. A bit of manipulation with Dominic Tarr’s excellent JSONStream, and out pops a NodeRank for every module.

A ranking by itself does not a search engine make. At the risk of being branded a heretic, I’m using ElasticSearch for the main search engine. Yes, it’s Java. No, it’s not Node. Hey Whadda You Gonna Do! ElasticSearch lets you add a custom score to each document – that’s where the NodeRank goes. I hacked all this together into a little website that lets you search for Node modules: nodezoo.com

You use nodezoo to search for modules in the same way as you use Google: just type in some relevant terms and something reasonable should come back. It’s not necessary for the module description or keywords to contain your search terms. The results still need refinement (big time!), but I need complaints to know where it’s going wrong – tweet me: @rjrodger.

The search results also attempt to provide some additional context for deciding which module to use. They include a link to the github project (if it exists), the stars and forks count, and a few other pieces of meta data.

The nodezoo system itself is still pretty hacky. One key piece that’s missing is updating the search index in real time as people publish modules. At the moment it’s a batch job. And it downloads the entire database each time. That’s probably not a good thing.

I’m going to do a series of blog posts on this little search engine, explaining how it works, and walking through the refactoring. The code is all on github https://github.com/rjrodger/nodezoo if you want to follow along. This is part 1. More soon!

Posted in Node.js | Leave a comment

Why I Have Given Up on Coding Standards

Every developer knows you should have a one, exact, coding standard in your company. Every developer also knows you have to fight to get your rules into the company standard. Every developer secretly despairs when starting a new job, afraid of the crazy coding standard some power-mad architect has dictated.

It’s better to throw coding standards out and allow free expression. The small win you get from increased conformity does not move the needle. Coding standards are technical ass-covering. At nearForm I don’t want one, because I want everyone to think for themselves.

There’s a lot of noise out there. The resurrection of JavaScript is responsible. One “feature” in particular: optional semi-colons. Terabytes of assertion, conjecture and counter-argument are clogging up the intertubes. Please go write some code instead. You know who you are.

Well-meaning, and otherwise fabulous developers are publishing JavaScript coding standards and style guides. You are all wrong. Stop trying to save the world.

Here’s what’s happening: when you started coding you had no idea what you were doing. It was all fun and games until you lost an eye. Once you hurt yourself one too many times with sloppy code, you came to understand that you were a mere apprentice. Starting on the path to master craftsman, you soaked up Code Complete, The Pragmatic Programmer, and of course, Joel.

And then, it happened. On the road to Damascus you gained insight. Your new grab bag of tricks would make you a rock star programmer. Your productivity had already doubled (looking back, that’s hardly surprising). And now you needed to spread the word. What worked for you will save others. You cajoled, you preached, you pestered. You lectured your boss on the need for best practices and standards. And most unforgivable of all, you blogged.

Most developers don’t make noise. Those who make noise, get promoted. You got promoted. You imposed your brilliant ideas on others, certain of victory. You wrote a coding standards document, and you made it law.

And then, nothing. The same old slog, the same death marches, the same bugs, the same misery. No silver bullet.

After a few years, you stopped coding and became a manager. You still know that coding standards, rules and regulations are vital. All it requires is proper implementation. You’ve never quite got there, but you’ll keep trying. Hit ‘em over the head a bit more. Code metrics! In any case, as a manager you get to delegate the pain away.

There is another road. Perhaps you went back to coding, or never left. Over time you came to realize that you know so little, and all your wonderful ideas are sand castles. You’re washed up. This is the next level of insight.

Other people are smarter than you. Not some of them. All of them. The coder writing the user interface? They are smarter than you … about the user interface. You’re not writing the code. Why don’t you trust them? No, that’s not the right question. They will still mess up. Why are you making a bigger mess by telling them what to do?

You get to the point where you understand that people are not machines. You need to push intelligence out to the edges. You need to give up control to get the best results.

So why do most intelligent coders do exactly the opposite? What makes us such ready dictators?

First, you transfer your own experiences onto others. But not everybody thinks like you. Brains are pretty weird.

Second, control feels good. It’s a comfortable hole in the sand. But you can’t tell coders what to do. Cats don’t herd.

Third, you get to duck responsibility. Everybody on the team does. We followed the rules! You failed. Yes, but we followed the rules! Well in that case, here’s another project…

Fourth, good intentions; best practices; professionalism; engineering – the seductions of process. You are chasing the same gold stars you got when you were eight years old. But how is the master craftsman judged? By results, only.

Fifth, idealism, the belief that you can understand the world and bend it to your will. Something we’re pretty good at as a species … after we fail a thousand times, and with repeatable processes. Software projects are always one of a kind.

There are worse sins than these. You only need one of them to end up with a coding standard.

The truly evil thing about coding standards is what they do to your heart, your team’s heart. They are a little message that you are not good enough. You cannot quite be trusted. Without adult supervision, you’ll mess up.

We started nearForm about a year ago, and one thing we really care about is writing great code for our clients. In earlier lives, I’ve tried all the processes and methods and rules of thumb. They all suck. None of them deliver.

Starting with the principle that our coders are really smart. That does work.

I expect everyone to write good clean code. You decide what that means. You decide if you can sleep at night with random code layouts and inconsistent variable names. But you know what, maybe they just don’t matter for a 100 line node.js mini-server that only does one thing. You decide.

It is your responsibility, because you can code.

Posted in Uncategorized | 275 Comments

Introducing the Parambulator module for validating “options” objects in Node.js

If you’ve used any Node.js modules at all, you’ll have noticed a common pattern with configuration. Many modules provide complex functionality that you can control by providing a JavaScript object with a bunch of named options, possibly with sub options. Here’s an example:

var toast = require('toaster').cook({ duration:5, bread:'white', sides:['A','B'] })

The duration, bread, and sides options need to be a number, a string, and an array respectively. It would be nice to validate this, and provide useful error messages if the calling code gets it wrong.

For real world examples of this pattern, check out:

In each of these modules, validation of the options is ad hoc and mixed in with other code. Sometimes you get an error message, if you’re lucky.

A real solution to this problem should be:

  • easy and quick to use, otherwise why bother,
  • declarative (i.e. no code required), that’s just good practice,
  • provide good error messages, which is pretty much the whole point,
  • customizable, in case you have weird and wonderful option structures.

So what are your options for validating the options, and providing decent error messages?

The basic approach is just to do ad hoc validation in code, and set a few defaults while you’re at it:

function handleoptions(opts) {
  if( !opt.foo ) throw new Error(“No foo, bro!”)
  opt.bar = opt.bar || “default value”
  var port = parseInt(opt.port,10)
  ...
}

That’s nice. But you need to read the code to get a clear idea of what’s going on. Also, things start out neat, but eventually that validation logic ends up all over the place. And, unless you’re incredibly pedantic, you’ll miss many error conditions. Oh, and don’t forget about the bugs in the validation code itself – it happens! So easy, quick and customizable, but not declarative, and error messages are a bit take it or leave it.

A more sophisticated approach is to recognize that this problem can be generalized. So use JSONSchema. As with XML schema, this way lets you specify exactly what you want the object to conform to. But it’s hard work (you have to be pretty exact), and JSON Schema are hard to read, and it’s a lot of overhead. You do get to tick the declarative box, but it’s not easy or quick, the error messages are nasty, and customization is “here be dragons” territory.

So, in the best Node.js tradition, I decided to solve this little problem with a new module: Parambulator.  Actually this module was also written with another purpose in mind: as example code for the nodejsdublin developer meetup to provide a small example module for those looking to write their own.

Parambulator let’s you define a quick and easy schema for your options. You don’t have to be exhaustive. Unlike JSONSchema, you can use the same object structure as your expected input, so there’s fewer mental gymnastics. And you get nice error messages by default, which you can customize. Here’s an example:

var toastcheck = parambulator({
  duration: {type$:'integer'},
  bread:    {enum$:['white','brown']},
  sides:    {type$:'array'}
})

toastcheck.validate( options, function(err) { … } )

This gives you “good enough” validation – about as much as you bother writing with ad hoc code. And you can be stricter if you like. It’s declarative, so you can see at a glance what the validation checks are. It’s still a schema, of sorts, but it’s a lot simpler than a full JSONSchema. And you can customize the validation rules and error messages (see the docs for details).

The next time you’re writing a module, and you’d like to make your options handling a little more robust, try it out and let me know. Also if you write any cool validation rules, send me a pull request!

Parambulator: https://github.com/rjrodger/parambulator

npm install parambulator

 

Posted in Node.js | Leave a comment

Node.js – How to Write a For Loop With Callbacks

Let’s say you have 10 files that you need to upload to your web server. 10 very large files. You need to write an upload script because it needs to be an automated process that happens every day.

You’ve decided you’re going to use Node.js for the script because, hey, it’s cool.

Let’s also say you have a magical upload function that can do the upload:

upload('myfile.ext', function(err){
  if( err ) {
    console.log('yeah, that upload didn't work: '+err)
  }
})

This upload function uses the callback pattern you’d expect from Node. You give it the name of the file you want to upload and then it goes off and does its thing. After while, when it is finished, it calls you back, by calling your callback function. It passes in one argument, an err object. The Node convention is that the first parameter to a callback is an object that describes any errors that happened. If this object is null, then everything was OK. If not, then the object contains a description of the error. This could be a string, or a more complex object.

I’ll write a post on that innards of that upload function – coming soon!

Right, now that you have your magical upload function, let’s get back to writing a for loop.

Are you a refugee from Javaland? Here’s the way you were thinking of doing it:

var filenames = [...]

try {
  for( var i = 0; i < filenames.length; i++ ) {
    upload( filenames[i], function(err) {
      if( err ) throw err
    })
  }
}
catch( err ) {
  console.log('error: '+err)
} 

Here's what you think will happen:
1. upload each file in turn, one after the other
2. if there's an error, halt the entire process, and throw it to the calling code

Here's what you just did:
1. Started shoving all 10 files at your web server all at once
2. If there is an error, good luck catching it outside that for loop – it's gone to the great Event Loop in the sky

Node is asynchronous. The upload function will return before it even starts the upload. It will return back to your for loop. And your for loop will move on to the next file. And the next one.

Is your website a little unresponsive? How about your net connection? Things might be a little slow when you push all those files up at the same time.

So you can't use for loops any more! What's a coder to do? Bite the bullet and recurse. It's the only way to get back to what you actually want to do.

You have to wait for the callback. When it is called, only then do you move on to the next file. That means you need to call another function inside your callback. And this function needs to start uploading the next file. So you need to create a recursive function that does this.

It turns out there's a nice little recursive pattern that you can use for this particular case:

var filenames = [...]

function uploader(i) {
  if( i < filenames.length ) {
    upload( filenames[i], function(err) {
      if( err ) {
        console.log('error: '+err)
      }
      else {
        uploader(i+1)
      }
    })
  }
}
uploader(0)

Do you see the pattern?

repeater(i) {
  if( i < length ) {
     asyncwork( function(){
       repeater( i + 1 )
     })
  }
}
repeater(0)

You can translate this back into a traditional for(var i = 0; i < length; i++) loop quite easily:

repeater(0) is var i = 0,
if( i < length ) is i < length, and
repeater( i + 1 ) is i++

When it comes to Node, the traditional way of doing things can mean you lose control of your code. Use recursion to get control back.

Posted in Node.js | 4 Comments

The JavaScript Disruption

The mainstream programming language for the next ten years will be JavaScript. Once considered a toy language useful only for checking form fields on web pages, JavaScript will come to dominate software development. Why this language and why now?

What is JavaScript? It is the language that web designers use to build web pages. It is not the language the software engineers use to build the business logic for those same web sites. JavaScript is small, and runs on the client, the web browser. It’s easy to write unmaintainable spaghetti code in JavaScript. And yet, for all these flaws, JavaScript is the world’s most misunderstood language. Douglas Crockford, a senior engineer at Yahoo, is almost singlehandedly responsible for rehabilitating the language. In a few short, seminal online essays published shortly after the turn of the century, Crockford explains that JavaScript is really LISP, the language of artificial intelligence. JavaScript borrows heavily from LISP, and is not really object-oriented at all. This curious design was well suited to a simple implementation running in a web browser. As an unintended consequence, these same mutations make JavaScript the perfect language for building cloud computing services.

Here is the prediction then: within ten years, every major cloud service will be implemented in JavaScript. Even the Microsoft ones. JavaScript will be the essential item in every senior software engineer’s skill set. Not only will it be the premier language for corporate systems, JavaScript will also dominate mobile devices. Not just phones, but tablets, and whatever enters that device category. All the while, JavaScript will continue to be the one and only language for developing complex interactive websites, completing drowning out old stalwarts such as Flash, even for games. For the first time in a history, a truly homogeneous programming language infrastructure will develop, with the same toolkits and libraries used from the top to the bottom of the technology stack. JavaScript everywhere.

How can such a prediction be made? How can one make it so confidently? Because it has all happened before, and it will happen again. Right now, we are at a technology inflexion point, yet another paradigm shift is upon us, and the JavaScript wave is starting to break. We have seen this before. Every ten years or so, the
programming world is shaken by a new language, and the vast majority of developers, and the corporations they work for, move en mass to the new playground. Let’s take a look at the two technology shifts that have preceded this one, so that we can better understand what is happening right now.

Prior to Java in the first decade of this century, the C++ language was dominant in the final decade of the last. What drove the adoption of C++? What drove the subsequent adoption of Java? And what is driving the current adoption of JavaScript? In each case, cultural, technological and conceptual movements coalesced into a tipping point that caused a sudden and very fast historical change. Such tipping points are difficult to predict. No such prediction is made here – the shift to JavaScript is not to come, it has already begun. These tipping points are driven by the chaotic feedback channels at the heart any emerging technology. One need only look at the early years of the motor vehicle: steam, electric and oil-powered vehicles all competed for dominance in similar historical waves.

What drove C++? It was the emergence of the object-oriented programming paradigm, the emergence of the PC and Microsoft Windows, and support from academic institutions. With hindsight such large-scale trends are easy to identify. The same can be done for Java. In that case, the idea of the software virtual machine, the introduction of garbage collection – a language feature lacking in C++ that offers far higher programmer productivity, and first wave of internet mania. Java, backed by Sun Microsystems, became the language of the internet, and many large corporate networked systems today run on Java. Microsoft can be included in the “Java” wave, in the sense the Microsoft’s proprietary competitive offering, C#, is really Java with the bad bits taken out.

Despite the easily recognizable nature of these two prior waves, one feature that both share is that neither wave led to a true monoculture. The C++ wave was splintered by operating systems, the Java wave by competing virtual languages such as C#. Nonetheless, the key drivers, the key elements of each paradigm shift, created a decade- long island of stability in the technology storm.

What is happening today? What are the key changes? Cloud computing is one. For the first time, corporations are moving their sensitive data and operations outside of the building. They are placing mission critical systems into the “cloud”. Cloud computing is now an abused term. It means everything and nothing. But one thing that it does mean, is that computing capacity is now metered by usage. Technology challenges are no longer solved by sinking capital into big iron servers. Instead, operating expenses dominate, driving the need for highly efficient solutions. The momentum for green energy only exacerbates this trend. Needless to say, Java/C# are not up to the job. We shall see shortly that JavaScript is uniquely placed to benefit from the move to cloud
computing.

Mobile computing represents the other side of the coin. The increasing capabilities of mobile devices drive a virtuous circle of cloud-based support services leading to better devices that access more of the cloud, leading to ever more cloud services. The problem with mobile devices is severe platform fragmentation. Many different platforms, technologies and form factors vie for dominance, without a clear leader in all categories. The cost of supporting more than one or two platforms is prohibitive. And yet there is a quick and easy solution: the new HTML5 standard for websites. This standard offers a range of new features such as offline apps and video and audio capabilities that give mobile websites almost the same abilities as native device applications. As HTML5 adoption grows, more and more mobile applications will be developed using HTML5, and of course, made interactive using JavaScript, the language of websites.

While it is clear that the ubiquity of HTML5 will drive JavaScript on the client, it is less clear why JavaScript will also be driven by the emergence of cloud computing. To see this, we have to understand something of the way in which network services are built, and the challenges that the cloud brings to traditional approaches. This challenge is made concrete by what is known as the C10K problem, first posed by Dan Kegel in 2003.

The C10K problem is this: how can you service 10000 concurrent clients on one machine? The idea is that you have 10000 web browsers, or 10000 mobile phones, all asking the same single machine to provide a bank balance or process an e-commerce transaction. That’s quite a heavy load. Java solves this by using threads, which are way to simulate parallel processing on a single physical machine. Threads have been the workhorse of high capacity web servers for the last ten years, and a technique known as “thread pooling” is considered to be industry best practice. But threads are not suitable for high capacity servers. Each thread consumes memory and processing power, and there’s only so much of that to go round. Further threads introduce complex programming programs, including a particularly nasty one known as “deadlock”. Deadlock happens when two threads wait for each other. They are both jammed and cannot move forward, like Dr. Seuss’s South-going Zax and North-going Zax. When this happens, the client is caught in the middle and waits, forever. The website, or cloud service, is effectively down.

There is a solution to the this problem – event-based programming. Unlike threads, events are light-weight constructs. Instead of assigning resources in advance, the system triggers code to execute only when there is data available. This is much more efficient. It is a different style of programming, one that has not been quite as fashionable as threads. The event-based approach is well suited to the cost structure of cloud computing – it is resource efficient, and enables one to build C10K-capable
systems on cheap commodity hardware.

Threads also lead to a style of programming that is known as synchronous blocking code. For example, when a thread has to get data from a database, it hangs around (blocks) waiting for the data to be returned. If multiple database queries have to run to build a web page (to get the user’s cart, and then the product details, and finally the current special offers), then these have to happen one after other, in other words in a synchronous fashion. You can see that this leads to a lot of threads alive at the same time in one machine, which eventually runs out of resources.

The event based model is different. In this case, the code does not wait for the database. Instead it asks to be notified when the database responds, hence it is known as non-blocking code. Multiple activities do not need to wait on each other, so the code can be asynchronous, and not one step after another (synchronous). This leads to highly efficient code that can meet the C10K challenge.

JavaScript is uniquely suited to event-based programming because it was designed to handle events. Originally these events were mouse clicks, but now they can be database results. There is no difference at an architectural level inside the “event loop”, the place where events are doled out. As a result of its early design choices to solve a seemingly unrelated problem, JavaScript as a language turns out to be perfectly designed for building efficient cloud services.

The one missing piece of the JavaScript puzzle is a high performance implementation. Java overcame it’s early sloth, and was progressively optimized by Sun. JavaScript needed a serious corporate sponsor to really get the final raw performance boost that it needed. Google has stepped up. Google needed fast JavaScript so that its services like Gmail and Google Calendar would work well and be fast for end-users. To do this, Google developed the V8 JavaScript engine, which compiles JavaScript into highly optimized machine code on the fly. Google open-sourced the V8 engine, and it was adapted by the open source community for cloud computing. The cloud computing version of V8 is known as Node.js, a high performance JavaScript environment for server.

All the pieces are now in place. The industry momentum from cloud and mobile computing. The conceptual movement towards event-based systems, and the cultural movement towards accepting JavaScript as a serious language. All these drive towards a tipping point that has begun to accelerate: JavaScript is the language of the next wave.

Posted in Uncategorized | 68 Comments

Node.js – Dealing with submitted HTTP request data when you have to make a database call first

Node’s asynchronous events are fantastic, but they can have a sting in the tail. Here’s a solution to something that you’ll probably run into at some point.

If you have a HTTP endpoint that accepts JSON, XML, or even a streaming upload, you normally read the data in using the data and end events on the request object:

var bodyarr = []
request.on('data', function(chunk){
  bodyarr.push(chunk);
})
request.on('end', function(){
  console.log( bodyarr.join('') )
})

This works in most situations. But when you start building out your app, adding in production features like user authentication, then you run in trouble.

Let’s say you’re using connect, and you write a little middleware function to do user authentication. Don’t worry if you are not familiar with connect – it’s not essential to this example. Your authentication middleware function gets called before your data handler, to make sure that the user is allowed to make the request and send you data. If the user is logged in, all is well, and your data handler gets called. If the user is not logged in, you send back a 401 Unauthorized.

Here’s the catch: your authentication function needs to talk to the database to get the user’s details. Or load them from memcache. Or from some other external system. (Don’t tell me you’re still using sessions in this day and age!)

So here’s what happens. Node will happily start accepting inbound data on the HTTP request, but before you’ve had a chance to bind your handler functions to the data and end events. Your even set up code only gets called after the authentication middleware is finished its thing. This is just the way that Node’s asynchronous event loop works. In this scenario, by the time Node gets to your data handler, the data is long gone, and you’ll stall waiting for events that never come. If your response handler depends on that end event, it will never get called, and Node will never send a HTTP response. Bad.

Here’s the rule of thumb: you need to attach your handlers to the HTTP request events before you make any asynchronous calls. Then you cache the data until you’re ready to deal with it.

Luckily for you, I’ve written a little StreamBuffer object to do the dirty work. Here’s how you use it. In that authentication function, or maybe before it, attach the request events:

new StreamBuffer(request)

This adds a special streambuffer property to the request object. Once you reach your handler set up code, just attach your handlers like this:

request.streambuffer.ondata(function(chunk) {
  // your funky stuff with data
})
req.streambuffer.onend(function() {
  // all done!
})

In the meantime, you can make as many asynchronous calls as you like, and your data will be waiting for you when you get to it.

Here’s the code for the StreamBuffer itself. (Also as a Node.js StreamBuffer github gist).

function StreamBuffer(req) {
  var self = this

  var buffer = []
  var ended  = false
  var ondata = null
  var onend  = null

  self.ondata = function(f) {
    for(var i = 0; i < buffer.length; i++ ) {
      f(buffer[i])
    }
    ondata = f
  }

  self.onend = function(f) {
    onend = f
    if( ended ) {
      onend()
    }
  }

  req.on('data', function(chunk) {
    if( ondata ) {
      ondata(chunk)
    }
    else {
      buffer.push(chunk)
    }
  })

  req.on('end', function() {
    ended = true
    if( onend ) {
      onend()
    }
  })        
 
  req.streambuffer = self
}

This originally came up when I was trying to solve the problem discussed in this question in the Node mailing list.

Posted in Node.js | Leave a comment

The Six Key Mobile App Metrics you Need to be Tracking.

Mobile applications are web sites, and traditional web analytics are not appropriate for mobile applications. What you need is insight that will make your app more effective. You will not find this insight by tracking downloads and installs, phone platforms and versions, screen sizes, new users per day, frequency of use, or any of the traditional metrics. Many of these have been dragged over, kicking and screaming, from the world of web analytics. Yes, these numbers will give you surface measures of the effectiveness of your app. Yes, they are important to know. Yes, you can use them to make pretty charts. But they are all output measures. They measure the results of your app design, interaction model and service level. They do not tell you what to change to achieve your business goals.

To gain real insight into your app and its users, insight that you can use to make your app more effective, you need to measure inputs. There are the six key input metrics that we cover in this article. Funnel analysis tells you why users are failing to complete your desired user actions, such as in-app purchases, or ad clicks. Measuring social sharing tells you what aspects of your app are capturing the hearts and minds of your users. Correlating demographic data with user behaviour will tell you why your user base does what it does. Tracking time and location, together, gives you insights into the contexts in which your app is used. Mobile apps design naturally tends toward deeply hierarchical interfaces – how optimised is yours? Finally, the real business opportunity may be something you never even thought of, so capturing the emergent behaviours of your user base is critical. Let’s tale a closer look at each of these metrics, and then take a look at how you can get this data with today’s services.

Funnel analysis allows you to determine the parts of your application that are preventing your users from reaching your business goals. Let’s take a simple unit converter app as a example. The canonical unit converter app lets you convert between kilograms and pounds, or inches and centimeters, and so on. Let’s say one of your business goals is to get your users to sign up to a mailing list from within the app. If you look at the user journey this requires, you might have a call-to-action button on the main screen, followed by a form to capture the email, followed by an acknowledgment page telling people to check their email accounts to verify their subscription. Funnel analysis breaks this user journey down into discrete steps: the tap on the button, typing in the email address, submitting the email address, reading the acknowledgement page. You need to know the percentage of users you are losing at each stage. Probably more than 50%. Understanding this activity funnel to your desired business goal is critical to building an effective app. Perhaps the next version should drop the call-to-action button, or use better copy text. Use funnel analysis to measure this.

Social media are a key element in the promotion of your app. When you leverage these media, you need to track the viral spread of your app. This is more than simple counting the number of tweets or facebook likes. You need to understand the structure of the social network you are attempting to permeate. You need to find the highly interconnected individuals, those who recommendations are actively followed by their friends and acquaintances. In any social network there are always a small set of key individuals who know everybody. You need to identify these people and engage with them. This might be as simple as special promotions, or even making them employees! Your mobile analytics solution should be telling you who these people are.

Do you understand the demographic constitution of your users, and can you correlate these demographics with user behaviour? This is the classic diapers and beer effect. A major UK supermarket chain found, through mining their purchase data, that increased beer purchases were correlated with increased diaper purchases. Cross-referencing this with the demographic data they have collected via a loyalty card scheme, the supermarket chain was able to figure out that parents with new babies were staying at home having a homemade meal and a beer, rather then going out to restaurants. This allowed for far more effective targeted advertising. Demographic data are more difficult to capture in the mobile app space, but carriers such as Sprint are now beginning to offer this information.

Location is an important element of the mobile user experience, and many mobile analytics services will offer location analysis. However this is not enough. Again, simply counting the number of users in various geographies does not tell you very much. It validates a business goal, but does not give you insight. You actually need to track the temporal dimension as well. Time and space must be analyzed together. Take our unit converter app. Usage of the app on Sunday afternoons within DIY store differs from at-home usage at mealtime during the week. In the first case you might like to show ads for power tools, in the second ads for food products. Mobile analytics offerings have yet to reach this level of capability, so you may need to consider custom solutions for this type of analysis.

Mobile application interfaces are very hierarchical in nature. This means that there are lots of screens with small amounts of information that the user has to navigate through. There simple isn’t enough screen space to show too much information at once. As result, the careful design of the screen hierarchy is critical to effective use of the app. If a particular function, such as in-app purchases, is too deeply buried, you will not achieve your goals for the app. Therefore it is very important to measure of the navigation pathways within the app. Berkeley University in California determined the layout of their campus walkways by not laying any paths at first. After the students had trampled the lawns for a year, they then build the pathways where the students had walked. This is what you need to do. (Actually, the Berkeley story is an urban legend, but it’s still a great one)

The final metric is something that requires a certain open mindedness. It can be measured using some heavy mathematics, but it can also be noticed intuitively. When you put a product on the market, it may well be the case that your customers start using it in weird and wonderful ways, that you never imagined. Hashtags (#thesethings) on twitter are a good example. Twitter did not invent them, but noticed that their users had come up with this interesting convention for marked content themes. They embraced this emergent behavior and were handed a core product feature on a plate. Of all the metrics in this article, this one, emergent behaviour, is the most precious. It could turn you into the next facebook (relationship status? What a feature!), or you could kill the golden goose without even knowing it by ignoring your users (Iridium satellite phones anyone?). Detecting emergent behavior is both and art and a science – keep your eyes open.

First published in GoMoNews Nov 2010.

Posted in Uncategorized | Leave a comment

Debug PhoneGap Mobile Apps Five Times Faster

PhoneGap is a fantastic open source project. It lets you build native mobile apps for iPhone, Android and others using only HTML, CSS and JavaScript. It’s a real pleasure to work with. It makes developing mobile apps a lot faster.

Still, you might find that your debug cycle is still too slow. After all, you still have to deploy your app to your phone for proper testing, and this can chew up precious time. The faster you can wash, rinse and repeat, the faster you can debug, and the faster you can deliver.

One way to speed things up is to use Safari on your desktop. There’s an even faster technique, but we’ll get to that in a minute. Using a WebKit-based desktop browser like Safari means that your development cycle is almost as fast as building a static website. Edit, Save, Reload. Just point Safari at the www/index.html file in your PhoneGap project and away you go.

Well almost.

Desktop browsers don’t offer exactly the same API, nor do they work in exactly the same way. Some mobile functions, like beeping or vibrating the phone are not really testable. The biggest issue though is that desktop browsers are too fast. Don’t forget that your runtime target is a mobile version of WebKit, such as Mobile Safari. Another issue is that touch gestures are tricky to handle, and have to be simulated with click events. It is worth it though for the fast development turnaround for certain kinds of functionality.

The obvious next step is to compile up your app in XCode and deploy to the simulator. Again, this works pretty well, but even the simulator has differences from the actual device, and again, it is just too fast. So what else can you do?

Why not install your native app as a web app? Sounds weird I know. The whole point of using PhoneGap is so that your apps can be native! But, if you install your app as a web app, guess what? No more installs! You just reload the app directly on your device every time you make a change.

Setting this up requires a little configuration. You need to run a web server to serve up the files in the www folder of the PhoneGap project. nginx is a good choice – here’s a simple configuration snippet:

[gist id="604802"]

You can then point your browser at http://your-ip/myapp/index.html and there’s your app! Do this using mobile Safari on your device, hit the + button and select “Add to Home Screen” to install as a web app, and away you go.

The big advantage to this approach is that you can test your app pretty much as it will appear and behave. You can even access the mobile safari debug log. Just remember to use the special meta tags to get rid of the browser chrome.

[gist id="604807"]

One further advantage is that the API environment will now be slightly closer to the full PhoneGap mobile API. Of course, you won’t be able to do things that can only be done using PhoneGap, but this gets you quite far along the road.

One final trick. Do the same thing on the desktop iPhone emulator and speed up your testing there as well!

Posted in Uncategorized | 1 Comment