Project

General

Profile

HouseMon + JeeBus + Flow

Added by jcw about 6 years ago

I’ve just released Flow 0.1 (docs) - it’s been a deep dive into Go and FBP for me.

So time seems to be going backwards: from HouseMon 0.9 to JeeBus 0.3 to Flow 0.1 … what gives?

Here’s where I’m headed with all this:

  • Flow is the foundation for it all: dataflow programming is – p e r f e c t – for real-time apps such as HouseMon and Tosqa. The fact that it’s called 0.1 just reflects my inexperience with it all. But there’s no way back. There are lots of “worker” components in the 0.1 release already. They don’t really belong there, but it was easy to develop them that way. These will be moved to JeeBus soon - due to the registry, this won’t affect the program code (just a few “import” lines).

  • JeeBus will collect all the functionality which is generally useful for HouseMon and Tosqa: a HTTP web server, MQTT messaging (client and server), websockets, the LevelDB database, a JavaScript engine for implementing logic not built into the core, and AngularJS with CoffeeScript on the browser side. I’ve been repeating this for some time now, and it won’t change. Lua is out (but it could easily be tied in as external MQTT client, same for any other language). With Flow now in the picture, JeeBus will also include lots of fairly general “workers”, tying into the serial port, websockets, MQTT, etc.

  • HouseMon is simply a small “main” application built on top of the JeeBus package. It needs less than a dozen lines of code to make a usable application. In addition, HouseMon is where additional home-monitoring and automation specific stuff can be implemented: in Go, as dataflow Workers and Groups, as JavaScript / CoffeeScript definitions, and as external processes: shell, Lua, Python, Ruby, Java, you name it. I see HouseMon as the proving ground for everything else, with the option to migrate whatever makes sense to JeeBus or Flow.
    * The other thing which hasn’t changed is “developer mode”. This is a setup now in JeeBus, which adds node.js and “gin” to the mix, to support live reload while making changes. I’ve briefly stepped away from it while hacking on Flow, but will definitely bring it back to the foreground once HM/JB and client-side development enter the picture again.
    An excellent example of the sort of gradual evolution I have in mind is the JeeBoot server, needed to make JeeBoot usable, i.e. over-the-air uploading for JeeNodes and other m328-based boards. I’ve started rewriting the server in Go, first as a separate process, importing the JeeBus and Flow packages. The server side is actually the easy bit
    developing a working client-side user interface, i.e. with Angular in the browser, is more work - and not yet finished. Once it’s usable, I intend to move the code into JeeBus. If all goes well, it can simply become another tab in the browser, also in HouseMon.

Soooo. All in all, it’s been a bit of a roller-coaster ride, but now all the key pieces are in place, IMO. Early days, some hacks, many fatal errors when used improperly, but all the corners of this application domain are now starting to solidify. It’s all there and I can’t stop being amazed at the sheer number and depth of open source packages this is all based on. It’s “giants all the way down”, to paraphrase a common saying :)

If you have been patient enough to read all of the above, and the recent posts in this forum, then I applaud you. You know how many trials and dead ends have happened in the past few years. If you are interested in joining in with any aspect of this design, then I applaud you even more. It only takes a few brave souls to push this project immensely forward, and to make it truly practical, flexible, and beautiful. I’ll be spending most of my time in this project on Flow and JeeBus, making sure that it not only works correctly and handles all the important scenario’s, but also that “the way in” for co-developers, co-extenders, co-deployers, co-admins, and co-users becomes an exciting, enjoyable, and fruitful experience.

The path ahead is clearly visible now, and looks - at least to me - better defined than ever. I hope you’ll like it as well!


Replies (26)

RE: HouseMon + JeeBus + Flow - Added by Remcokortekaas about 6 years ago

Although not contributing, I’ve been following you guys all the way in the forum. I can only say it looks like this is developing in a very clear, clean and high-end product. If you get it on the shelves I think nothing, even commercial, will compare to it. Write a book about it and sell the components in a webshop and you will have a good global business I think…. ;-) Not to mention lifting the IoT maker world to a new level.

Very impressed!

RE: HouseMon + JeeBus + Flow - Added by martynj about 6 years ago

As a mere tinkerer with the more tangible bits of Physical Computing, this tide has, at times, been hard to follow. But fascinating to observe. For the main architect and driver (and those co-contributors plus general sounding boards involved) all I can say is …   chapeau!

Now that the meta flows are pinned down, I’ll just stick with soldering up the plumbing ….

>

RE: HouseMon + JeeBus + Flow - Added by BeagleNode about 6 years ago

I too have been following this amazing journey. The coding discussions are way over my head but I have (barely) enough knowledge to appreciate the difficulties encountered along the way.

Inspired by those here and the general Arduino community I have about 10 JeeNodes collecting various sensor data. They send their data to a central node and from there to a MySQL database. As the number of nodes and my knowledge have increased change is inevitable. The JeeBoot concept is of great interest to me. Managing an ever increasing number of sensor nodes is becoming more difficult.

I’d like to contribute but as I said the development is way over my head. I might be useful for some testing as the development advances.

John

RE: HouseMon + JeeBus + Flow - Added by Mars about 6 years ago

Nice work indeed.

Although these programming languages are way beyond my skills, the Flow techniques look very familiar.

If I compare the described techniques with OSC (Open Sound Control), there are many similarities. A human musician interacts with sensor(s) that detect physical activity such as motion, acceleration, pressure, displacement, flexion, keypresses, switch closures, etc. The data from the sensor(s) are processed in real time and mapped to control of electronic sound synthesis and processing:
- OSC is fully ‘flow’ based;
- The flow can be processed using ‘transforms’;
- A ‘transform’ can map or even change the data/music, send them to other ‘transforms’, or be the end of the ‘flow’ (speakers for instance if you’r talking about music);
- The order of the ‘transforms’, ie the ‘flow’ can be changed giving different end results.
- It is VERY flexible!

Very happy to see such nice development!

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

I’m having a ball with Flow these days. Writing more and more highly independent and reusable “workers” for it. This is the list so far:

$ go run main.go -v
Flow 0.1.0

  BinaryFill CalcCrc Clock Counter Dispatcher FanOut HttpServer IntelHexToBin
  JavaScript LevelDB LogReader LogReplayer Logger MqttPub MqttServer MqttSub
  Node-homePower Node-ookDcf Node-ookRelay Node-p1scanner Node-radioBlip
  Node-roomNode Node-smaRelay NodeMap Pipe Printer ReadTextFile Repeater
  SerialIn Sink Sketch-RF12demo SketchType TimeStamp Timer

Documentation at http://godoc.org/github.com/jcw/flow
$

That’s just a beginning, but already this stuff is latching together like crazy…

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

More progress on the HM/JB/FW front.

JeeBus can now replay a log in simulated time, look up the packets in a nodemap, launch and run the corresponding decoders, recombine the results, save them to the database, publish on MQTT, act as HTTP server, accept WebSocket connections (not tied into MQTT yet), and handle a full-fledged browser app based on AngularJS. Several utility commands are now included for cmd-line use: dump/export/import the database, get/put individual items in the DB, publish/subscribe to MQTT.

Almost everything is tied together with simple “gadgets” combined into dataflow circuits and can therefore be reconfigured at will, either in code or from a JSON setup file.

[[housemon:devmode|Development mode]] works again, with automatic recompilation of the Go application as well as live-reload for any browser-side changes. All it requires is a 1-line “index.js” file and having node.js installed - everything else is now taken care of under the hood. There is no dependency on “gin” anymore, nor the need to run “npm” manually. Node.js now does all that, in a way acting as both an installer and a supervisor.

The entire JeeBus app compiles to a ≈ 17 MB executable, and all the Mac and Linux versions can be cross-compiled (I’m now using “goxc” for that). The Linux version for ARM runs as is on BBB, RasPi, and Odroid, without any installation dependencies.

On Odroid, after leaving the simulator running for over 12 hours, I see it using 12 MB of RAM and having consumed 38s of total processor time, i.e. under 0.1% (that’s after some 40,000 messages stored in LevelDB and published to MQTT).

In short: this is exactly what I was after. This design can easily scale and perform as back end on low-end Linux boards, while serving a highly responsive browser-based front end. It might even work on boards considerably smaller than a RasPi.

With JeeBus and Flow stabilising, I can get back to HouseMon again at last. The plan is to re-use JeeBus as a Go package, with a minimal “main” in HouseMon and an easy way to define more gadgets + circuits in Flow’s registry. So the “housemon” executable will become a replacement for jeebus with more features, decoders, etc.

By the end of next week, getting started with HouseMon from scratch will require only these steps: 1) install “Go” and “Node.js”, 2) type the command “go get github.com/jcw/housemon”, 3) launch HouseMon with “node .” from inside the HM directory, and 4) go to http://localhost:3333/.

RE: HouseMon + JeeBus + Flow - Added by tve about 6 years ago

It looks like you deleted the 0.3.0 and 0.2.0 branches from the jeebus repo? What’s the incantation to make the jeebus master branch do something? It looks like it doesn’t contain an mqtt broker anymore?

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

I’ve kept tags for both those branches, to mark the state.

All the HM-specific gadgets have been moved “forward” to HouseMon. JB no longer launches MQTT by default (but all MQTT-related gadgets are in there). To run a setup with MQTT loaded by default, as well as the replay simulator, I suggest using the jcw/housemon checkout instead.

Sorry for these massive rearrangements, but I really need to extract generic stuff for use with Tosqa and other projects. W.r.t. what goes where: 1) flow engine + generic gadgets in jcw/flow, 2) all gadgets more general than just HM go in jcw/jeebus (things like http, mqtt, serial, leveldb), 3) leaving just the HM-specific gadgets in jcw/housemon (currently gadgets/decoders and gadgets/rfdata). IOW: HW >= JB >= FW.

I hope this makes a usable setup and split. Flow has been stable for some time, and JeeBus is where I’ll be adding whatever other new functionality makes sense.

Is this workable? I’m trying to get all these big structural changes out of the way asap, so that this stuff stops being a roller-coaster ;)

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

jcw,tve,

Just to chip in…..

I was using the 3-day old repo stuff (without the move) and all was fine.

I just migrated to the new repo’s and all went fine also, but I did my setup a little differently as I want to ‘develop’ and thats difficult with the import paths unless you add some symbolics.

Here is what I did just 10mins ago….

Archived my old github forks {jeebus,housemon} and split them away from githubs stupid ‘only one fork via GUI’ policy.
Reforked {flow,jeebus,housemon}
cloned MY forks to by local filesystem (adding upstream back to JCW’s for push)
created a /src/github.com/jcw folder with symbolics for {housemon,flow,jeebus} pointing back to MY local repo’s @ /src/github.com/TheDistractor/{housemon,flow,jeebus}
(The above now allows me to launch jeebus or housemon in my local repo structure, make changes push/fetch etc and it all thinks its comming from github.com/jcw/….)
Warning: Just DONT do a “go get…” otherwise you’ll find yourself reverting to last commit - use git for all updates etc, NOT go.

I then just updated housemon/setup.coffee with my new gadgets and nodemaps , circuits etc and I’m back to where I was before the move. Only change that stumped for for 30sec was sensorSave to PutReadings but thats all sorted now.

So in general following @jcws install guide in this thread and using the above to setup your environment so you can change code and contribute back without messing up imports and your good to go.

—lightbulb

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

and…I’ll report back if I find anything broken.

Whilst I am here, jcw - I now see we have gadgets with homes in the appropriate places (flow,jeebus,housemon).
How should we set things for gadget re-use?

I proposed that any core (generic) JEE gadgets/decoder get submitted back to either jeebus or housemon (difficult to determine what is where)
I consider a JEELIB gadget/decoder as something that is directly related to the jeelabs shop components.
A simplistic example to hand would be BMP085 decoder using JEELIB.

For other gadgets/decoders we should initially publish awareness on the HM forum I think, with perhaps a single sticky topic with gadget/decoder name and link to somewhere explaining in more detail, perhaps a forum thread or a blog entry etc.

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

> I did my setup a little differently as I want to ‘develop’ and thats difficult with the import paths unless you add some symbolics

Wait. If you have a config.txt in the current dir, HM (and JB) will pick it up. You can add lines such as “APP_DIR=./app”, etc. See main.go for the known variables. You can also set environment variable with the same name, and they will override everything else. There should never be a need to put stuff in specific places or play tricks with symlinks - just adjust the paths in the config.txt where you’re launching the app.

As for gadget repositories: anywhere works, as long as Go can get it. Just include the proper imports, see the JB and HM main.go sources again for examples. Here too, there is no need to put stuff in a specific place: every gadget is self-registering, and all you need to use it is its name. You could have a lightbulb/blah on github, with .go files in it full of gadgets, and all I’d need is to include this line in my main:

import _ “github.com/lightbulb/blah”

When I do “go get -u”, it’ll automatically be downloaded (and updated, due to the “-u”).

My vote would be to keep a very strict split between HM and JB repo’s: everything in HM, until solid and stable enough to include in JB - if it’s not HM-specific, of course. I now realise that I need to move my circuit diagram editor back out of JB for now - it’s waaay too early to do that work on the “main” JeeBus repo. IOW, JB needs to be declared beta real soon now, instead of acting as an alpha playground.

We can easily change to a better strategy for gadget repositories later. The main requirement is that the JB API solidifies very quickly, so that we don’t end up with breakage across all the gadgets created from now on.

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

jcw,

Thats not the issue, I understand config.txt etc, what I am talking about is modifying my copy of ‘jeebus’ and ‘flow’ (and HM) whilst I work things out.

Say I make a modification to ‘jeebus’ and want to submit a push to you, then I am best off keeping a pure jcw/jeebus fascade, so I dont have my own jeebus with different imports etc.
Then, my change can simply be pushed to my repo, and then I make a pull request, no imports need to be adjusted.

If I did this in a standalone fork (say TheDistractor/jeebus, I would have to adjust hm imports, or if I modified TheDistractor/flow, I’d have to adjust jeebus ‘imports’ etc.

Using a symlink allows me to code against MY fork, whilst still keeping code (and imports) pretty much inline with jcw/xxx, so a pull request is simple.

—lightbulb

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

> I need to move my circuit diagram editor back out of JB for now - it’s waaay too early to do that work on the “main” JeeBus repo

While I did move it out, and into the Tosqa project for now, it’s actually progressing quite nicely:

Add/delete of gadgets and wires works, all in scalable SVG, with CSS styling and cursor hints. Drawing and dragging appears to be very snappy so far. The editor takes a few JSON data structures, which are similar but not yet identical to what’s needed in Flow. Changes are reflected back in those same structures, so this is actually quite close to a first practical version. The circuit editor is an Angular directive, meaning that showing “current node” details in Angular should be quite easy to do.

Still missing: feeds, as I haven’t yet figured out how to visualise them. Also missing, but planned: an undo/redo mechanism.

The (fairly dense) code is here, if you’re interested: https://github.com/tosqa/tosqa-host/blob/master/app/circedit/circedit.coffee

All in glorious MIT-licensed form ;)

cedit.png (83.9 KB) cedit.png
1965

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

jcw,

Looking good!.

Are feeds not simply visualised as an alternative color box (1 per feed) with 2 input controls, one (dropdown) for ‘type’ that shows ‘tag’ or ‘message’ and another text input as ‘text’ with no ‘In’ specifically, but its single Out going to a gadget/circuits InX pin.

i.e almost based upon what you have already with the ‘arrow’ replaced with the input/tag controls.
Right click canvas, , up pops a box, fill in data, connect feed.Out to an In pin that has nothing else connected.

Course, its easy for me to say, I’m not yet in a position to ‘do’ as I’ve broken my DB Rpc :(

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

The colours and icons are configurable per-type right now.

Indeed, feeds are a bit like extra boxes, but they are a separate list in the circuit description (see github). Adding feeds in the diagram might require dealing with different things which are not represented in the same way in the data structure. Not sure - JSON structures are of quite easy to re-org and re-shape.

Maybe a feed needs to become an attribute of the gadget, also in the JSON definition of Flow. Then again, I do like to see all the feeds together, as quick overview of all the startup conditions for that circuit.

I tend to view these feeds as “config settings”, which merely happen to go through the flow mechanism when a circuit is run. So it might indeed be possible to treat them as the attributes of a gadget. Perhaps even completely in Angular, i.e. click on a gadget to select it, then show details somewhere else on the page with Angular, including “settings”, being a list of thing which end up being the feeds when processed. The presence of a feed could perhaps be indicated with a different “pin” marker in the diagram.

One advantage I see is that keyboard entry then becomes a standard form-based mechanism, with easier styling, tabbing, copy-and-paste, etc. I’ll explore things a bit - I’m a bit more at ease with NG than with D3 at the moment.

But either way, it sure is a potent combination, that NG + D3 stuff!

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

before I broke my db rpc, I checked a few of my devices, and I think the keyboard/gesture approach should figure highly, especially feeds like you say, as they will be the most active input focus imo.

Having a data entry panel would work better than editing the canvas objects (for feeds specifically), especially with a ScrollTo.

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

It sure helps to talk about this stuff - simple solutions tend to emerge more easily that way :)

I think the following might work quite nicely: two NG tables under the diagram, or to the side. One with details about the currently-selected gadget, the other a list of all the feeds. Or perhaps even two tabs on the page: one with the diagram full-width, the other perhaps with a mini-diagram rendering, plus those two or three panes with tables, text, etc.

Another use case for feeds is JavaScript, in the case of the JS gadget. So there’s actually a use case to support editing non-trivial amounts of text in these feeds.

It doesn’t have to be perfect, of course :) - just enough to support all the minimal editing requirements for now.

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

My own view of HM (0.7.x + heavy mods) , over the last 3 months, is that I use my phone for 70% interaction, 20% tablet, 10% pc.
The pc for initial setup/config/troubleshoot, the phone for status checks etc. My HM 0.7 also uses a twitter feed to notify updates which is invariably by phone.
The tablet in in the kitchen used mainly for overriding HW on/off or specific light control, and setting the underfloor etc.

So - mobile is high on my agenda for day to day use.
In HM 0.7.x I used 1 page per widget, with shortcuts on my phones home screens direct to each page. Works very good like that.

Oh, and I use nginx for front facing, but with 0.9.1 I think I’ll go direct to HM with custom SSL even tho nginx websocket support is great.

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

Forgot to add,

So, as the default case (for me) is ‘visualising’ status and toggling/setting stuff on/off etc, my pref for a default view (of the diagram tool) would be the gadget/feed view, with the full screen diag as an additional navigation (either tab or keyboard/gesture) as I would see this only being used during circuit construction/modification, and hence ‘very seldom’ used, but an absolute requirement none the less, not even the 80/20 rule here I think, more like 99.9/0.1 unfortunately.

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

Good point - I like the idea of putting the feeds first. And yes, they do indeed belong on the web page, not in the diagram.
Even though that means all my blood, sweat, and tears spent on the circuit editor won’t take centre stage :)

Zurb Foundation 5 has good support for responsive layouts to make these pages work well across phone, tablet, and pc.

Great, thanks. This points to a practical way forward.

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

jcw wrote:
> Even though that means all my blood, sweat, and tears spent on the circuit editor won’t take centre stage :)

But, the editor is absolutely critical for those many users who want to get cracking easily (and are wary of code changes), so that will be the first port of call for them.
When major milestone is close, we could add ‘intro’ page that could take the user through a walkthrough tour of HM and plonk them into the full diagram view at the end, then on next use it reverts to normal behaviour once a ‘don’t show this tour dialog on startup’ checkbox is clicked.

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

Ok, I think I’ll go for something like this, current gadget details on the left, all feeds on the right:

In this example, “g1.In” has three feed messages. The actual feed data is a contenteditable='true' HTML element - which is really convenient, since they automatically expand as needed.

(still fighting a bit with Foundation’s fat table layouts - no idea why there is so much white space under each row, oh well…)

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

jcw,

OK, so in your screengrab, (right) - this would translate to circuit data:

feeds: [
{ data: “some data”, to: “g1.In” }
{ Tag: “blah”, Msg: “tagged data”, to: “g1.In”}
{ data: 123, to: “g1.In” }
{ data: “some input text\nwith multile lines\ncould be JavaScript source code”, to: “g7.In” }
]

Sooooo…perhaps I need to refetch flow, but I seem to be using feeds like this:

feeds: [
{ data: “some data”, to: “g1.In” }
{ tag: “blah”, data: “tagged data”, to: “g1.In”}
{ data: 123, to: “g1.In” }
{ data: “some input text\nwith multile lines\ncould be JavaScript source code”, to: “g7.In” }
]

Have the Monikers changed now Tag vs tag Msg vs data? My existing stuff seems to be fine using second approach - not got any code in front of me so cant remember exactly (and too lazy to check github :) )

Also is g7.In just a typo/wip, as I would have expected g1.In1..7 etc. Or at least another g1.Input pin name.

And also I now notice that because its json (marshal) I probably could have used 123(int) whereas I stupidly have used “123”(string) and parsed to int in my gadgets (although I actually need int64 mostly so I’d have to cast anyhow).

Observations:

left (Printer) panel would be better ‘rolled up’ [more…./+] on subsequent edit (after initial create/save), and titled as “Printer (g1)” imo, because after its created subsequent edits would almost never be to the right panel, and almost always to left panel (feeds). An if you want to edit that you would ‘roll down’. This would save precious real estate on responsive devices, and rolled-up would act as the feeds ’header’ in a top/bottom responsive manner.

RE: HouseMon + JeeBus + Flow - Added by lightbulb about 6 years ago

Ah, my bad:

“CURRENT” gadget on left, ALL feeds on right……..scrub the g7 sentence then, all other Q’s remain valid (to me).

RE: HouseMon + JeeBus + Flow - Added by jcw about 6 years ago

Whoops, you’re right on the Tag/Msg thing. This stuff hasn’t been hooked up yet, so I didn’t catch the error.

Note that the design of the data structures is still a bit in flux. Wires, for example, could be better modelled as a map of “frompin/topin”, with the capacity as value, e.g. { "g1.Out/g2.In": 0 }

Feeds are undecided. Maybe they should be stored alongside the input pins. And maybe the list of input and outputs in the gadget definition should be separate.

Such changes would prevent storing inconsistent data (not all cases, but more than now).

I won’t change things without a good reason at this stage. Ideally, I’d like to have the cleanest data structure design possible, regardless of whether that fits the current code and tools (since data lives forever and code always evolves). But in practice, it’s simply yet another set of trade-offs.

(1-25/26)