Home »

How to collect data

This example assumes you have some JeeNodes and wish to collect readings from them.

The simplest case is to plug one JeeNode running the RF12demo sketch into a USB port, and set the rest up as room nodes - because all the drivers already exist for this setup.

Setting up the JeeNodes

See this page for some related info about setting up a network of wireless nodes.

Installing JeeMon + JeeRev

See the setup and hello world pages for details and getting-started examples.

Decoding incoming data

To decode incoming data, you need to create a hub, i.e. an application based on JeeRev which is permanently running. As the hello world examples show, setting up a hub is easy.

For decoding, we need two bits of information: how the JeeNode can be reached via USB, and what the RF12 node ID is of the sending room node(s).

To find out what the name of the USB interface is, enter this command on the command line:

JeeMon tcl SysDep listSerialPorts

(or launch the TkCon console with JeeMon con and then type in SysDep listSerialPorts)

You will get a list of the serial FTDI interfaces (such as a USB BUB) currently plugged into your USB ports. If more than one is plugged in, you'll need to figure out which one is the JeeNode you want to use.

Now create a configuration file called hub-config.txt with a text editor, containing something like this:

features { interfaces }

devices: {
  usb-A600dVI3: { location here driver autosketch }
  RF12-868.6.12: { location there driver roomNode }
}

Note the absence and presence of colons - the format of the configuration file uses a very specific format.

Be sure to replace the USB port (mine is usb-A600dVI3), and the RF12 settings with what you're using as frequency, group, and node ID for the room node (mine is set to 868 MHz, group 6, node 12).

The only feature we need here is interfaces. It scans all USB ports and uses the information listed in the devices: section to decide what to do with each interface.

The "autosketch" driver tells JeeRev that a JeeNode is attached to the USB port, and that it will identify itself when opened (see the drivers page for details). Since it's running the RF12demo sketch, JeeRev automatically loads a driver with that same name. Similarly, a roomNode driver will automatically be loaded as soon as packets from the specified node start coming in. As it so happens, the RF12demo.tcl and roomNode.tcl drivers are included in the RF12 library and automatically found by JeeRev, so all we need to do now, is launch JeeMon:

JeeMon hub

If all is well, this will start up a hub running the JeeRev code, and load the above configuration settings. Here is some sample output:

$ JeeMon hub
15:30:18.480      ep+ ::oo::Obj8 sock7 -server
15:30:18.522   hooks+ HUB.BOOT 
15:30:18.523   hooks+ HUB.INIT 
15:30:18.524   hooks+ HUB.READY 
15:30:18.533    iface add usb-A600dVI3
15:30:18.552   hooks- HUB.READY ::interfaces
15:30:20.012   autosk driver usb-A600dVI3 [RF12demo.7] A i1 g6 @ 868 MHz 
15:30:20.014 RF12demo config usb-A600dVI3 RF12-868.6
15:30:55.854   hooks+ DRIVER.REPORTS there.light 94 {when 1303824655 driver roo>
15:30:55.855   hooks+ DRIVER.REPORTS there.moved 0 {when 1303824655 driver room>
15:30:55.856   hooks+ DRIVER.REPORTS there.humi 40 {when 1303824655 driver room>
15:30:55.857   hooks+ DRIVER.REPORTS there.temp 238 {when 1303824655 driver roo>
15:30:55.858   hooks+ DRIVER.REPORTS there.lobat 0 {when 1303824655 driver room>

Note that this very simple hub doesn't require any custom code - just a few configuration settings.

Real-time web viewer

Let's extend this so we can view the readings in a web browser. Change the features entry in the config file to:

features { interfaces welcome webdemo }

Then restart JeeMon and point your browser to http://127.0.0.1:8181/. There is a "DataTables" link which presents a table with incoming data - and if you're running Chrome, Opera, or Safari, that table will update itself whenever new readings come in (see Web server events for details):

Here's how it works: the welcome feature pulls in the Webrig feature, which is JeeRev's embedded webserver, and sets up the home page. And the webdemo feature pulls in the DataTables example and everything else needed to present a jQuery-/Ajax-driven table with readings.

That's all - still no custom code. Note that all readings are tracked on the web page, but they are not saved anywhere.

Logging readings to file

Another scenario is to add a simple file log with all readings saved as text in CSV (i.e. comma-separated value) format. This one is even simpler, just change config as follows:

features { interfaces logdata }

(I've left out the welcome and webdemo features, but feel free to mix and match features at will)

The logdata feature has default settings, so your hub will now start saving files named ./logdata/YYYYMMDD.csv - one file per day, with automatic rollover at midnight.

To change these settings, add an extra section to the config file - here's the unmodified (i.e. default) configuration:

logdata: {
  logdir ./logdata
  format csv
  fields {time param scaled unit}
  gmt 0
}

The currently supported file formats are: txt, csv, and tab (these are also used as file extension).

The gmt setting specifies whether you want time stamps and file rollover to be based on GMT or on local time.

Logging raw data to file

(not available in JeeRev 0.9)

The lograw feature saves all serial data received via the interfaces module to file. Similar to logdata, files are saved in a directory and rotated on a daily basis:

features { interfaces lograw }

The default configuration is as follows:

lograw: {
  logdir ./lograw
  gmt 0
}

The output format is fixed, and files are saved with a ".txt" extension. Example output:

L 13:25:28.438 usb-A900ad5m {[RF12demo.7] A i1 g5 @ 868 MHz }
L 13:25:28.716 usb-A900ad5m {DF I 2111 1152}
L 13:25:28.717 usb-A900ad5m {Available commands:}
[...]
L 13:25:28.732 usb-A900ad5m {Current configuration:}
L 13:25:28.733 usb-A900ad5m { A i1 g5 @ 868 MHz }
L 13:25:33.515 usb-A900ad5m {OK 19 115 49 194 41 33 166 189 0}
L 13:25:42.506 usb-A900ad5m { ? 158 0 255}
L 13:25:47.689 usb-A900ad5m {OK 3 165 247 4 0}
L 13:26:09.809 usb-A900ad5m {OK 38 57 57 229 0}
L 13:26:09.825 usb-A900ad5m { -> ack}
L 13:26:18.336 usb-A900ad5m {OK 19 163 55 86 40 171 180 19 70 88 47 12}

It's essentially a transcript of all incoming serial data, formatted in such a way that it can be loaded as a Tcl script (as used by the replay feature).

Interfacing with Pachube

This feature is enabled by including it in the config file (again, you can add whatever other features you want):

features { interfaces pachube }

In addition, you need to specify which parameters to send to Pachube, the feed number to use, and your private API key:

pachube: {
  send { house.*.rate lipotest.ping }
  feed 5942
  apikey ...
  throttle 5s
}

The throttle option defines the maximum rate for sending data and defaults to 5 seconds. Only the last reading in each period will be kept and sent.

Datastreams are automatically created if they don't exists yet (as of April 2011, the free Pachube account is limited to 5 streams).

That's it - all matching readings will end up on your Pachube feed.

There's also a mechanism to collect data from Pachube, i.e. the reverse process. This is enabled with a collect option specifying which streams to subscribe to (as with send, wildcards are allowed):

pachube: {
  ...
  collect { house.* lipo* }
}

For this to work, the datastreams should be tagged with JeeRev-style parameter names, i.e. a.b.c.

One usage scenario is to send values from one hub and collect them on other hubs anywhere in the world (note that this requires your private API key on both sides, even though Pachube data is always public).

Calling other programs

Similar to the pachube feature, the callout feature supports launching external apps whenever certain parameters are reported. To activate it, use:

features { interfaces callout }

The details are specified in the config file, for example:

callout: {
  match { *.rate *.temp }
  exe {echo >>test.txt}
  args {time param scaled unit}
}

This will run commands such as:

echo >>test.txt "2011/04/28 21:30:08" roof.temp 16.5 °C
echo >>test.txt "2011/04/28 21:30:14" upstairs.myra.temp 20.0 °C
echo >>test.txt "2011/04/28 21:30:31" house.elec.rate 353 W
echo >>test.txt "2011/04/28 21:30:59" house.elec.rate 355 W
echo >>test.txt "2011/04/28 21:31:07" living.temp 20.8 °C
echo >>test.txt "2011/04/28 21:31:41" jc.books.temp 20.5 °C
echo >>test.txt "2011/04/28 21:31:53" house.elec.rate 353 W
echo >>test.txt "2011/04/28 21:32:00" bathroom.temp 20.2 °C

The options are fairly limited, but you can launch a shell script to adjust args and do more complex things.

Collect data via the web

(not available in JeeRev 0.9)

The webfeed feature takes incoming web requests and turns then into readings:

features { welcome webdemo webfeed }

Here's an example request:

The returned page displays the time stamp associated with this entry as confirmation.

And here's what the hub will do with such readings, in the same way as any others:

Nothing fancy, and in particular very little validity checking. You have to enter a parameter name and value as part of the url, the rest is optional and can contain arbitrary additional details about the submitted reading. It took a whopping 12 lines of code to implement this webfeed.tcl feature.

Export via Web Hooks

(not available in JeeRev 0.9)

This is the opposite of web feeds. There is sort of a "standard" way to pass information to other web servers, using a mechanism called Web Hooks. A very crude variant of this is now available as webhook feature:

features { replay webhook }

It requires some configuration info specifying where to send the request to - here's an example for feeding a ThinkSpeak test channel:

webhook: {
  url {http://api.thingspeak.com/update?key=...&field1=$value}
  match { house-erate }
}

This will send a web request to the specified web server whenever new readings are reported for any of the matching parameters (just one in this case). Note that combining web hooks and web feeds is one way to pass readings from one hub to another.

This feature is just proof-of-concept for now, it will need more work to become practical: POST support, proper URL quoting, configurable headers, cookies, etc.

Other types of nodes

The above examples are based on the RF12demo and roomNode drivers, both very small: under two dozen lines of code.

But what if you want to tie your own sketches and nodes into your hub? To make this happen, you need to add suitable drivers to your hub application. Writing a driver in Tcl is fairly straightforward (especially if you copy an existing one and start modifying it), see the custom drivers page.

webdemo.png (103.4 KB) jcw, 2011-04-30 11:56

webfeed-out.png (75.5 KB) jcw, 2011-05-07 12:18

webfeed-in.png (37.8 KB) jcw, 2011-05-07 12:19