Project

General

Profile

The “JeeBus” utility package

JeeBus is a messaging and data storage infrastructure for low-end hardware. JeeBus is all about plumbing. It does nothing useful on its own, but can help with creating real-time applications.

The following building blocks are included in JeeBus:

  • a dataflow engine, based on a new Flow package that implements Flow-based Programming (FBP)
  • a static web server plus a websocket server to handle all “live” browser activity
  • a PubSub messaging broker, based on MQTT
  • a general-purpose key/value data store, based on a variant of LevelDB
  • a mechanism to load and execute scripts dynamically, based on JavaScript
  • a number of conventions to make all the pieces work together in a nice & useful way

JeeBus can be combined with glue logic and custom code in a number of ways:

  • by connecting basic components (“gadgets”) through the built-in dataflow mechanism
  • using JavaScript, which run sandboxed and “in-process” as part of the JeeBus server
  • using external processes, hooked into JeeBus as MQTT subscribers and publishers
  • in the browser, running client-side code written specifically for the application

For an example of how this is intended to be used, see the HouseMon homepage.

Status

Currently under active development. Release 0.9.1 is scheduled for the end of March.

Tested on Mac OSX and various Linux’es, including the BeagleBone Black, Raspberry Pi, and Odroid U2.
Windows will require some further work, since the serial driver in JeeBus currently doesn’t support it.

Installation

> See also the [[jeebus:setting up a new JeeBus application]] page, now somewhat outdated.

The Go tools must be installed and GOPATH has to be set, then:

  • install this project with: go get github.com/jcw/jeebus
  • go to the source directory using: cd $GOPATH/github.com/jcw/jeebus
  • launch the server as: jeebus and leave it running in a terminal
  • to see messages from MQTT, run jeebus see ?topic? from a separate console topic defaults to ‘#’ if not specified, i.e. all topics
  • to dump key/value pairs in the database, run jeebus dump ?prefix?

To tinker and try out code changes, use go run main.go to compile-and-run.

Binaries

To bypass any build issues, you can try out binary builds.

There is a read-only area which is available via BitTorrent Sync, under the name madder @ JeeLabs

  • the public read-only access key for this area is “BIYHT7LS64PT4M65NUTXSIK5J3CZTUA5M”

The JeeBus binary and other apps related to it can be found in the builds/ folder. Only a few platform builds will be available initially, but it’s a start, eh?

Mac OSX

Running JeeBus on Mac OSX requires three steps:

  1. get a binary copy of the jeebus executable, and put it in your search path
  2. get a copy of the app/, base/, and common/ folders from GitHub
  3. launch in the proper directory (with a setup.json file in it): jeebus

If you don’t have access to a binary build, or prefer to compile your own:

  • install Go, set up GOPATH, and then type: go get github.com/jcw/jeebus

Source code

The Go source can be found at http://github.com/jcw/jeebus

Commands

The jeebus binary supports a number of built-in commands, other than running it as is for normal server use:

> Note: these don’t work, as of mid-March 2014.

  • jb pub <topic> ?payload? - publish given topic and (JSON) payload to the server
  • jb run ?:port? - start the JeeBus server on the specified port (default “:3333”)
  • jb see - connect to the server and show all MQTT messages as they are published
  • jb serial <dev> ?baud? ?tag? - connect the specified serial port device to the server
  • jb tick ?topic? - send 1-second tick messages to topic (default is “/admin/tick”)

There are also some database utility commands, these can only be used while the server is not running:

  • jb dump ?from? ?to? - show the contents of the database (server must not be running)
  • jb export prefix - export all entries with given prefix as JSON to standard output
  • jb import file - import data from specified file, replacing prefix(es) mentioned in it

The export / import commands can be used to manually make bulk changes to stored data.

Architecture

Messages

Messages are published to MQTT and have a “topic” and a “payload”. The topic is a “/”delimited string, to which others can subscribe, with “+ and ”#" wildcards, as defined in MQTT. The payload is always sent as JSON (and can therefore not contain cycles).
In JeeBus, there are a couple of conventions for topic names:
* topic names starting with up to 2 characters before the slash are reserved for JeeBus
* topic names starting with a slash are associated with the database, see below
Some reserved topics are:
* if/...
connected to a hardware interface
* rd/... - raw data messages, candidate for logging
* sv/... - service requests
* ws/... - connected to a websocket

And as already mentioned, all messages published to /... are picked up by the database.

Database

The database is based on LevelDB and is structured as one large key/value collection. Again, in JeeBus there are some fixed conventions:

  • all keys of the form /... are stored in the database, just like any others
  • but such keys are also stored as hist/.../<timestamp>, i.e. as historical archive
  • the timestamp is in JavaScript format, i.e. milliseconds since Jan 1, 1970 UTC
  • all stored values must be valid JSON
  • don’t mix leaves and branches, i.e. if key blah/... is in use, then blah should be avoided
  • this is somewhat analogous to file system path names: can’t use the same name as file and as folder

Those last conventions are fairly easy to adhere to. If you have a collection and need to store something about that collection as well, then simply pick slightly different names, e.g.

  • use doc1/name/ids with doc1/name/id/..., or doc2/name-info with doc2/name/...

Web server

The web server in JeeBus is based on the net/http and code.google.com/p/go.net/websocket packages.

All HTTP requests are served as static files, since this setup is intended for use with AngularJS, i.e. as SPA and RIA. The web socket connection is used for all real time data, and handles requests and triggers in both directions.

All data sent through the web socket is formatted either as JSON or as binary data. There are some conventions to allow sending MQTT messages, console debug messages, RPC calls, and RPC replies over the same connection, the logic for this is implemented as an Angular “service” (not to be confused with services for MQTT in JeeBus) in the file app/jeebus/jeebus.coffee.

Browser clients connected via a web socket can send MQTT messages to be published for anyone’s use, as well as direct RPC requests, avoiding an extra round trip through MQTT. In the other direction, every new web socket connection is registered as ws/... in MQTT, to allow sending specific messages to it.

The following RPC requests are available from web sockets:

  • attach(prefix) - start tracking changes to all database entries with given prefix
  • echo(...) returns the args as a list, this is just a quick test that everything works
  • db-keys(prefix) returns a list of sub-keys of the specified prefix in the database
  • db-get(prefix) returns the value in the database for the specified key
  • detach(prefix) - stop tracking changes to all database entries with given prefix
  • lua("foo/bar","blah",...) calls “blah” in “scripts/foo/bar.lua”, passing in the remaining args

There is no “db-store”, since this can be done using an MQTT publish for “/…” topics and keys. Other keys cannot be written directly (only RPC code in the server and Lua scripts are allowed to modify other keys in the database).

When setting up a web socket, the client specifies a subprotocol to identify the application running in the browser (i.e. “housemon” or “tosqa”). This is used to dispatch MQTT requests sent from a browser, and to decide which browsers to broadcast to when an MQTT message to “ws/…” is picked up.

Attach and detach

A web client can indicate that it wants to track changes to the database (limited to keys starting with “/”). The wrappers in app/jeebus.coffee should make it fairly convenient to use this mechanism from AngularJS:

\$scope.admin = jeebus.attach ‘/admin/’

The content will start out as an empty map, with existing entries initially appearing as fresh additions. After that, all changes will be tracked, i.e. added, modified, or deleted to match the database on the server.

When no longer needed, i.e. when a controller is destroyed, the model should be detached:

\$scope.\$on ‘\$destroy’, ~~> jeebus.detach ’/admin/’
In the view.jade file, it’s then a matter of referring to live values in that model:
Time of last server start: {{ admin.started }}
Note: attach + detach have not yet been optimised, but the intention is to make these calls quite efficient.
h3. Keys vs objects
When attaching to the database, there is an important distinction between keys and values. For example:
<pre>
/a/b/c/d1 = { e1: { f1: g1 } }
/a/b/c/d2 = { e2: { f2: g2 } }
</pre>
If you attach to /a/b/c/, then you will see two entries in the model: d1 and d2, with values as expected.
If, on the other hand, you attach to /a/b/, then you’ll also see two keys, but now called c/d1 and c/d2. In other words, the path nesting which the database offers as its key/value store is only nested in one way, and what attach does is splice up that path (e.g. “/a/b/c/d1”) in two: a prefix, and a residual key.
These residual keys may not be as convenient if they contain slashes, since you cannot write model.c/d1 to access the stored value, of course. You have to write model['c/d1'] instead. So the easiest way to use the attach mechanism, is to attach as deeply as possible, at the point where residual keys are still usable as identifiers in CoffeeScript and JavaScript. For residual keys which are not identifiers, such as time stamps, there is no other way to access the values than through the model[residualkey] notation.
h1. Older versions
h2. JB 0.3
Adding node.js back in for live-reloads during development.

JB 0.2

Early version of the new Go-based server design.