Sync: synchronizing objects

Introduction

This document is still science fiction. The implementation is lagging behind this documentation. So sync’s not ready yet.

Obviel Sync is a JavaScript library that keeps track of client-side objects and can keep them in sync with a backend and a UI. It supports different kinds of backends: HTTP (Rest_), localstorage, and websockets, and abstracts the application away from the particular backend used.

We can divide a dynamic web application in a frontend and backend. The frontend takes care of the UI the user sees in the browser. It runs on the client; in the web browser. The UI is constructed from models which are JavaScript objects on the client – the Ob in Obviel. How are these models maintained?

The simplest case is a web application that maintains its models itself and that’s it. The models disappear as soon as the web application is closed. The only software that touches the models is the web application itself and the only storage is main memory. Simple single-player games on the web are often like this, though often there is an exception to this when they need to save and share high scores.

In many cases the information represented by an application’s models needs to persist even when the web application is not active. A todo-list application, for instance, needs to have its todos stored so when you close the application and open it again later, the todos are there again.

In many applications, at least some information needs to be shared between different users. Forum software is like this, for example, as forum messages need to be visible to multiple users. Another case we already mentioned is sharing high-score information for a web game.

For many applications therefore a backend is needed for storage or persistence. This is usually the only responsibility of the backend; other responsibilities include coordinating information between different users, controlling access to information, deriving information, receiving new information from external sources, and making sure the data stored is consistent.

When frontend models change this can affect the backend. For instance, when a todo item is added to the todo list, the backend is informed of this. Similarly when a todo item is changed, the backend needs to be informed about the update.

The direction can also be reversed: the web application models need to be updated when the state of the backend changes. For example, when you post an item on a forum, the server backend is informed about it, and the backend then informs other people who may be viewing the forum about the new item as well. So, there is a bi-directional exchange of information between frontend and backend:

  • The backend needs to be informed about changes to the object structure in the frontend (typically triggered by the UI).
  • The frontend needs to be informed about changes done to the object structure on the server (which often results in a UI update).

You can write manual code to keep frontend models in sync with a backend. For HTTP Obviel core offers some facilities to help you with this – you can render a URL directly, causing the object to be loaded from the server and (re)rendered. But even with that convenience you need to write quite a bit of custom code.

Obviel Sync helps with this. It is a declarative system that lets you state how your frontend models relate to a backend. If you use a special way to maintain your frontend models, Obviel Sync takes care of informing the backend.

Since Obviel Sync keeps track of changes to frontend models, it can not only be used to update the backend, but to keep the UI up to date as well - when a frontend model is changed, the UI can be updated automatically.

Keeping Track of Objects

Obviel Sync needs to keep track of when frontend objects have been changed, new ones are created, and existing ones removed. We can then use this information to update the backend as well as the UI.

Obviel Sync uses a special session object to keep track of changes, additions and removals. You then commit the session when you want Obviel Sync to take action. This is also know as the Unit of Work pattern.

In order to establish a session, you first need a connection:

var conn = new obviel.sync.HttpConnection();

XXX explain HttpConnection

We’ll see what you can do with the connections later.

Now you can create a session:

var session = conn.session();

The connection keeps track of the current session in use.

Now let’s imagine you have an object obj and that you have changed it by modifying a property foo on it:

obj.foo = "This is a new value";

To inform Obviel Sync that the property foo has changed on obj, use update:

session.update(obj, 'foo');

Now let’s consider collections of objects. A collection of objects is represented as an array in JavaScript. It must always be part of an object; we call this object the container object. A container object is just an ordinary JavaScript object that can contain one or more items. Here is a container object that has a collection property items:

var container = {
  iface: 'mycontainer',
  items: []
}

We now add an item to the items collection:

var item = {iface: 'item', value: 'A'];

container.items.push(item);

Here’s how to tell the session what we’ve done:

session.add(container, 'items', item);

Similarly we can remove an object from a collection. Obviel Sync has a special helper function that lets you do this easily:

obviel.sync.removeFromArray(container.items, item);

Here’s how we tell the session what we’ve done:

session.remove(container, 'items', item);

Automatically Keep Track of Changes

It can be a bit cumbersome to have to tell the session separately what changes we made to objects. The session provides a special mutator object that makes this easier. We can wrap it around the object we want to mutate:

var m = session.mutator(obj);

You can also create the mutator directly from the connection; this will create a hidden session for you automatically:

var m = conn.mutator(obj);

We now use the mutator to change obj:

m.set('foo', 'This is a new value');

This sets the property foo and informs the session about it (session.update(obj, 'foo')) all at the same time.

We can also modify collections in container objects. First let’s wrap the container object in a mutator:

var m = session.mutator(container);

Now we can get to the items collection and push an item into it:

m.get('items').push(item);

This adds the item to the array items and tells the session about it (session.add(container, 'items', item)).

We can also use this to remove items from an array and inform the session about it:

m.get('items').remove(item);

You can also use get to get a mutator for nested subobjects, not just arrays:

var m = conn.mutator({
  iface: 'outer',
  inner: {
    iface: 'inner'
  }
});
m.get('inner')

Connections

In order to communicate with a backend, Obviel Sync needs a connection.

XXX Obviel Sync at present provides three kinds of connection:

  • obviel.sync.HttpConnection - communicate with a (RESTful) HTTP backend that serves up JSON objects.
  • obviel.sync.LocalStorageConnection - store data in a local storage backend on the browser. XXX somewhat broken
  • obviel.sync.SocketIoConnection - communicate with a websocket backend using Socket.io. XXX immature

To set up a connection you need to create it:

conn = new obviel.sync.HttpConnection();

Initialization

When your application frontend starts up, it typically needs to be filled with some starting content. The frontend can get this from the connection.

You do this by calling init() on the connection. init() takes an object argument – it specifies the root object of the frontend that you want to keep in sync with the backend. Obviel will load initial content for the application from the connection. init returns a jQuery deferred that is completed when this loading is done.

The typical pattern of loading up a connection looks like this:

conn.init(root).done(function() {
   $('#app').render(root);
});

I.e. initialize root from the connection, and then render it somewhere into the DOM tree.

Declarations

You now know how Obviel Sync can be used to track changes, and how to start up loading content for your root object, but not yet how to explain to Obviel Sync how to communicate with the backend. What should be done when the frontend or backend changes? Obviel Sync needs to be told. You declare this on the connection. You need to be done with the declarations before you call init() on the connection.

Declaration is per iface. Per iface you tell Obviel Sync what to do when the frontend is modified, and also what to do when the backend reports an object has been modified.

Let’s examine the operation of init() first. When you initialize an object using init(), the backend is asked to provide the contents of this object; this is called a refresh action. We need to tell Obviel Sync what to do when we ask the frontend to refresh the root object:

conn.frontend('root').refresh({
   method: 'GET',
   url: 'http://example.com/refreshRoot',
   response: obviel.sync.backendUpdate
});

Here we tell Obviel Sync that when a refresh is requested for root, we want to issue a GET HTTP request to the server, using URL http://example.com/refreshRoot, and to take the resulting JSON object as an update from the backend, so pass it to backendUpdate.

So now we need to teach Obviel Sync what to do when a backendUpdate for a root object comes in:

conn.backend('root').update({
  find: function(frontendObj) { return root; },
  action: obviel.sync.updateObject
});

We’ve specified two things here:

  • how to find the root object given an incoming frontendObj. In this case we simply return the global root object, but this could be looking up something by id, as we will see later.
  • what to do with the found object (root in this case). obviel.sync.updateObject combines frontendObj with root and updates its attributes.

That takes care of basic communication from backend to frontend; we can now initialize an object and get its content from the backend. Now the other way around: when we change an object on the frontend, we want to inform the backend:

conn.frontend('root').update({
  method: 'POST',
  url: 'http://example.com/updateRoot'
});

Here we tell Obviel Sync that when an object is updated in the frontend, we want to send a HTTP POST to the URL http://example.com/updateRoot. The content is a JSON respresentation of the backendObj that changed.

Chaining

You can chain declarations. All together now:

conn.frontend('root').refresh({
   method: 'GET',
   url: 'http://example.com/refreshRoot',
   response: obviel.sync.backendUpdate
}).update({
  method: 'POST',
  url: 'http://example.com/updateRoot'
});

conn.backend('root').update({
  find: function(frontendObj) { return root; },
  action: obviel.sync.updateObject
});

Working with collections

Let’s maintain an collection of objects. We want to inform the backend whenever an item is added or removed:

conn.frontend('item').add('item', {
  method: 'POST',
  url: 'http://example.com/addItem'
  response: obviel.sync.backendUpdate
});

Here we say that when we add an item to a collection, we want to POST the contents as JSON to the url http://example.com/addItem, and the response we receive we want to interpret as a backendUpdate action - this is useful as often the backend updates the just added object with some extra properties such as an id to uniquely identify it.

We can also listen to remove actions:

conn.frontend('item').remove({
  method: 'POST',
  url: 'http://example.com/removeItem'
  combine: true,
  transform: obviel.sync.getIds
});

Here we’ve done a few new things. When we remove items from a collection it’s common to remove more than one of them at the same time, and send the list of the items we want to remove to the backend. In addition this list doesn’t need to be the full items at all, but instead a list of ids that uniquely identify each item to remove to the backend (the backend needs to be in charge of id generation in this case). The combine setting indicates we want to combine multiple remove actions into one, and the transform setting indicates we want to only pass the id property values to the backend instead of the full objects.