Project

General

Profile

HM 0.9 development

Added by jcw over 3 years ago

There was some initial discussion on Github (see https://github.com/jcw/housemon/issues/114), but this forum is probably a better spot for that.

See that link for context. In short: the server side of HouseMon is being rewritten to use MQTT as core messaging system. This allows more flexibility, both in the way messages flow through the application and in offering more ways to tie into the server, using any programming language. I’m also evaluating the use of an embedded Lua sandbox for server-side decoding and writing custom rules for automation.

The second change is that I’m splitting off “The Core” into a separate project, to be used by HouseMon, Tosqa, and other projects. This core will be called JeeBus (loosely related to an old JeeLabs project by that same name). It has its own new repository at https://github.com/jcw/jeebus.

Lastly, the change with the most impact probably, is that JeeBus is written in Go - a language well-suited for writing servers in. It uses coroutines (called “goroutines” in Go) and lightweight channels for synchronisation and to avoid the Node.js callback “pyramid of doom”. The memory use of JeeBus is considerably lower and the performance is higher than with Node.js, as used in HM 0.8 and before. I’m developing on Mac, while regularly making sure that the result works well on the BeagleBone Black.

To summarise: HM 0.9 will be lighter on the server side, while keeping the good parts on the client side as before, i.e. AngularJS, WebSockets, and the CoffeeScript/Jade/Stylus dialects.


Replies (188)

RE: HM 0.9 development - Added by SevenW over 3 years ago

I added a new mqtt client to the repository for demonstration purposes to make connection to the mqtt server from a different process (executable). Maybe this can evolve into a test client, which could be moved into a test folder.

I like to do some further experiments with more HM like functionality, which eventually may not belong to the core of jeebus. For example rf12demo and logger like modules. Should we add those into a new HM-branch, or for the moment keep them with jeebus?

SevenW

RE: HM 0.9 development - Added by jcw over 3 years ago

Yes, a new HM “0.9.x” branch would be good. JeeBus is a generic core for several projects (not so generic yet, but that’s the plan).

I suggest experimenting in the HM repository unless it’s really obvious that it will end up in JB.
We can always move stuff to JB later once it solidifies a bit and turns out to be of more general use.

RE: HM 0.9 development - Added by jcw over 3 years ago

Here’s the basic idea for HouseMon 0.9 w/ JeeBus:

RE: HM 0.9 development - Added by SevenW over 3 years ago

multiple executables in a go project are not as easy as a hoped for. For future reference:
http://stackoverflow.com/questions/9985559/organizing-a-multiple-file-go-project

RE: HM 0.9 development - Added by jcw over 3 years ago

Good link, thx - I was just wondering about that…

How about creating one app with the 1st cmd-line arg to select the proper code? I.e. like “go run”, “go build”, etc. does.

It’s too early still, but I’d like to keep the “jeebus” exe running at all times, even during development. That way the serial ports, WSN reception, database, and logging don’t have to be disrupted while tinkering with services or the web browser front end. By writing things as a “secondary” Go app, everyone can try stuff out, and then we move it into the JB exe when ready.

RE: HM 0.9 development - Added by SevenW over 3 years ago

link to discussion on mixed language golang projects:
https://groups.google.com/forum/\#!topic/golang-nuts/6orXabMNivE

This might be useful for a HM 0.9.x branch.
By far the easiest way is conform to the go way of work, by having the root of the git repo containing the go files. This makes the project go get-able. And as remarked in the thread: “the rest of the code doesn’t really care where it lives”, where code refers to other languages.

RE: HM 0.9 development - Added by lightbulb over 3 years ago

Hi Guys,

OK, I have read the initial github posts, and the posts moved over here in redmine.

I am going to try to get up to speed in next few days.

( Just as an aside, I ran my 3 year old ‘go’ app that writes my meter reads into google spreadsheet (via AppEngine), through ‘gofix’ and it has brought it bang up to date, (changed quite a few bits of old go code) - and it seems to run great. )
so - big hand clap for gofix….

As @jcw is using OS/X, I’m going to be taking the Ubuntu track (but I do have VMWare workstation with Arch & win7/8 VM’s to check cross compiles when we get to that stage (rs232 issues excepting etc).

I was thinking ….. that it would be pretty easy to add something like /if/vserial/ipaddress:port to create a socket listener for socat/ser2net style serial access from remotes?? This way I’ll also be able to leverage remote serial devices with a little trickery (i’ll experiment).

I’ll try my first build tomorrow.

nb: I’ll still be contributing to 0.8 until 0.9 replaces it.

—lightbulb

RE: HM 0.9 development - Added by SevenW over 3 years ago

Another interesting code organization is in te lua package that is used by jeebus. The go code is not in the root of the repo, but a go get clones the complete repo, and enables the normal use of git, while all go stuff works from the lua subfolder.
https://github.com/aarzilli/golua
# go get -u github.com/aarzilli/golua/lua

RE: HM 0.9 development - Python - Added by SevenW over 3 years ago

I made a Python prototype that connects to the jeebus server and publishes and receives messages. It is based on the python mosquitto bindings that can be installed with apt-get install python-mosquitto on Ubuntu. In case of sufficient interest I can publish it somewhere.

RE: HM 0.9 development - Added by jcw over 3 years ago

Neat. We could create an area in HM with language bindings - how about “lang/python”?

RE: HM 0.9 development - Added by jcw over 3 years ago

I’ve checked in a couple of new “sub-commands” for JB. The idea is to launch it as server, and extra processes for other tasks:

  • first of all, JB must now be started as jeebus, no args, to launch it as server
  • jeebus see - display all MQTT messages on stdout
  • jeebus serial <dev> <baud> ?tag? - connect a serial port to the server

The “tag” can be left off with the RF12demo and blinker sketches, and in general with any sketch which starts by sending out a [...] line. It is used to tag the messages sent to MQTT.

This code doesn’t do proper outbound routing yet, i.e. it can’t send out stuff to a specific serial port. Also, the MQTT messages are not yet sent as JSON.

RE: HM 0.9 development - Added by SevenW over 3 years ago

I see where you are going to. So this is the “one executable”, but you intend to run multiple instances of this executable then.
I have been doing some thinking about the connection to serial ports and jeenode sketches. We first need to establish a generic design for this type of building blocks, and then make specialized instances for something like the serial/[blinker].

Naming is also important here. You have been talking about interfaces and serial. Lightbulb proposes a virtual serial to connect through sockets. I think such an entity that connects (in)directly to the real world can be called a device, device proxy, device driver or something like that. A typical property is that it can be controlled, or produce data, just like the rf12demo. We also better make distinction between the communication channel, and the payload. For example, the raw data from the rf12 network is the payload, while /if/serial/dev.. is the communication channel. Similar to MQTT. Such a modeling enables us to connect all kind of devices to the jeebus. And when we would just connect to the mqtt port, remote connections is a piece of cake.

Devices (for short for now) can be the rf12demo sketch or SerailCmd() just added to jeebus, A python script control plugwise hardware, a webscraper, a cpu/mem/net monitor on the linux box itself, a file reader of rf12demo data (replay), etc, etc.

And having this perspective on it, I ask myself whether those devices should be integrated in the jeebus process?

I try to make some drawings with those and more thoughts tonight.

RE: HM 0.9 development - Added by jcw over 3 years ago

I’ve reorganised JeeBus into a package “jeebus” and a “jb” main. The server is now started as “jb run”.

For HM 0.9, we can create a “hm/main.go” file, then the install becomes “go get github.com/jcw/housemon/hm” (or “go run hm/main.go” during development). We don’t even need to install JeeBus separately, if HM does at least everything now in jb/main.go starting the server becomes “hm run”, watching messages is “hm see”, and starting a serial port is “hm serial …”. This way, we can add whatever sub-commands we like to HM, such as logger, replay, etc.

Still a few niggles to work out w.r.t Lua, so maybe a dual hm + jb setup will be simpler after all. We’ll see.

I think I’ve also figured out a way to integrate this with the existing HM 0.8 client-side stuff - by keeping node.js and primus-live in the mix for development only. This is required anyway, to compile CoffeeScript to JavaScript, etc. The workflow could be extended to check these generated .js etc files into github as well. That way, using HM 0.9 won’t require node.js, while still remaining extensible via MQTT.

More on this soon.

RE: HM 0.9 development - Added by jcw over 3 years ago

The README on GitHub documents the new installation and startup process. The new main.go file contains a “logger” command as example of how to add more functionality to HM 0.9 as external command. JeeBus is now integrated into HouseMon, in that both are now running in parallel. All uses of the database need to be migrated, this will require a somewhat different RPC and event mechanism over the websocket.

The design of topic names will require some more thought. It’s definitely not correct yet.

RE: HM 0.9 development - Added by lightbulb over 3 years ago

@jcw,

did you ‘push’ your latest mods referred above?

I cannot see “logger” in jeebus.go:SubCommand

I’ve literally just built my dev environment, so maybe missed something.

—lightbulb

RE: HM 0.9 development - Added by lightbulb over 3 years ago

Still - it seems to have built and is running.
I had to ‘dig’ my blinkboard out from an old project (hotglued)

still, i have just pressed a few buttons and smiling :)

—lightbulb

RE: HM 0.9 development - Added by jcw over 3 years ago

> I cannot see “logger” in jeebus.go:SubCommand

Ah, wait. JeeBus does not have logger, HouseMon has. Go to the 0.9.x branch of HM and do “go run main.go logger”. The HM app (currently) “inherits” all the sub-commands of JB, and can add its own.

So right now: forget JB, build HM instead and use that (“go run main.go run” iso “jb run”, etc).

This will probably change: I’d like to have JB build rarely but with all the MQTT, LevelDB, Lua etc stuff in it, with a “secondary app” such as HM adding more functionality to the server vua MQTT pubsub, but not including the server stuff and avoiding JB’s build hassles such as Lua. Apps written in pure Go can easily be cross-compiled for a dozen platforms at once, see https://github.com/mitchellh/gox.

RE: HM 0.9 development - Added by lightbulb over 3 years ago

Ah - ok - doh.

So we are back in the HM0.9 branch for HM stuff, however, the guts ‘jeebus’ will be coming from github ‘jeebus’.

I have not installed yet - just a quick scan over the readme, but can see already how they interact.

When I get a moment, I’ll do a slightly more involved setup/install for a few platforms that can be kept up to date as features get migrated.

I have already come a cropper with a GOPATH to a set of older projects (another workspace) that I forgot was in my profile. Switched to another terminal and all falls apart….sorted now :{

That brings me to an issue with running this as ‘server’ (daemon). This looks improbable in go as go apps terminate once main() terminates…..
Are you currently advocating lots of jb instances all doing their own little interactions with the main mqtt broker in jb run??

ps: on second thoughts - don’t answer that now - its early days..

—lightbulb

RE: HM 0.9 development - Added by jcw over 3 years ago

Ok, I’ve rearranged things further (sorry for the wild gyrations). JeeBus, i.e. the “jb” command, now contains the server and various utility subcommands. The HouseMon build now contains only extensions (only logger, so far).

The thing is that this way HM can be pure Go, i.e. no C included, and all platform binaries can quickly be generated. Watch this:

 housemon (0.9.x *) ➤ time gox
Number of parallel builds: 8

-->      darwin/386: github.com/jcw/housemon
-->    darwin/amd64: github.com/jcw/housemon
-->       linux/386: github.com/jcw/housemon
-->       linux/arm: github.com/jcw/housemon
-->     freebsd/386: github.com/jcw/housemon
-->   freebsd/amd64: github.com/jcw/housemon
-->     openbsd/386: github.com/jcw/housemon
-->     linux/amd64: github.com/jcw/housemon
-->   openbsd/amd64: github.com/jcw/housemon
-->     windows/386: github.com/jcw/housemon
-->   windows/amd64: github.com/jcw/housemon
-->     freebsd/arm: github.com/jcw/housemon
-->      netbsd/386: github.com/jcw/housemon
-->    netbsd/amd64: github.com/jcw/housemon
-->      netbsd/arm: github.com/jcw/housemon
-->       plan9/386: github.com/jcw/housemon

real    0m2.328s
user    0m11.810s
sys 0m2.491s
 housemon (0.9.x *) ➤ ls -l housemon_*
-rwxr-xr-x  1 jcw  staff  3766076 22 Jan 17:32 housemon_darwin_386
-rwxr-xr-x  1 jcw  staff  4595856 22 Jan 17:32 housemon_darwin_amd64
-rwxr-xr-x  1 jcw  staff  3812152 22 Jan 17:32 housemon_freebsd_386
-rwxr-xr-x  1 jcw  staff  4639448 22 Jan 17:32 housemon_freebsd_amd64
-rwxr-xr-x  1 jcw  staff  3899824 22 Jan 17:32 housemon_freebsd_arm
-rwxr-xr-x  1 jcw  staff  3815816 22 Jan 17:32 housemon_linux_386
-rwxr-xr-x  1 jcw  staff  4645016 22 Jan 17:32 housemon_linux_amd64
-rwxr-xr-x  1 jcw  staff  3899184 22 Jan 17:32 housemon_linux_arm
-rwxr-xr-x  1 jcw  staff  3802032 22 Jan 17:32 housemon_netbsd_386
-rwxr-xr-x  1 jcw  staff  4629048 22 Jan 17:32 housemon_netbsd_amd64
-rwxr-xr-x  1 jcw  staff  3889136 22 Jan 17:32 housemon_netbsd_arm
-rwxr-xr-x  1 jcw  staff  3854552 22 Jan 17:32 housemon_openbsd_386
-rwxr-xr-x  1 jcw  staff  4702288 22 Jan 17:32 housemon_openbsd_amd64
-rwxr-xr-x  1 jcw  staff  3112855 22 Jan 17:32 housemon_plan9_386
-rwxr-xr-x  1 jcw  staff  3505152 22 Jan 17:32 housemon_windows_386.exe
-rwxr-xr-x  1 jcw  staff  4282368 22 Jan 17:32 housemon_windows_amd64.exe
 housemon (0.9.x *) ➤

RE: HM 0.9 development - Added by jcw over 3 years ago

> Are you currently advocating lots of jb instances all doing their own little interactions with the main mqtt broker in jb run??

Only for development (and boy is it convenient in that context - Go compiles about as quickly as Node.js launches!).

One step further along would be to have some configuration info somewhere, and activate all the services on startup. This could be part of HM:

  • HM could start the JB server in the background if not yet running
  • HM sets up its services by subscribing to MQTT on that JB server
  • HM gets the config, and starts everything up (connect to serial ports, logging, etc)

This could be placed behind “housemon start” (“go run main.go start”), with other sub-commands in HM available for development and command-line maintenance. IOW, I see the CLI as the “under the hood” stuff, for us developers to extend and tinker with.

At the end of the day (ehm, maybe a bit later…), I would hope that you only need something like this to get HM up and running from scratch:

  • grab the HM and JB binaries for your platform
  • launch it as “housemon start”
  • do everything else in the browser

(HM start could even download the JB binary from JeeLabs, to remove one more step)

No package installs. No Go install if you’re not planning on changing HM or JB. No dependency hell. Nothing.

RE: HM 0.9 development - Added by jcw over 3 years ago

Just to clarify: lots of processes are ok during development (multiple instances of an executable share all their code memory).

For real use, all services in JB and HM can run inside those two processes, plus a few processes for what you attach to MQTT yourself.

PS. The performance of all this is flat-out staggering, if you’re a long-time user of scripting languages like me…

RE: HM 0.9 development - Added by lightbulb over 3 years ago

> Only for development (and boy is it convenient in that context…

I agree (with my developer hat on)

As for everything else….I’m still getting up to speed.

As for ‘consumer’ experience, this has to be very simple/slick, without lots of command line incantations. But thats down the line a little and by that time we should have ironed out any platform differences. At least a ‘go’ stub can work cross platform and has the relevant os checking builtin.

I do agree the x platform builds are very nifty, I have not looked, but I wonder if there is anything like nodemon etc. As far as the binary’s go
, they appear to be very efficient, but what I like best is the goroutines concept coupled with channels, a real logic trimmer.

I do think that I will miss the ‘dynamic’ evaluation, but hey….we have lua right…

back to my attempt at a REPL ( “ADD serial /dev/…. ” / “REMOVE serial /dev/…” )
(purely a learning process for me - not likely to be submitted as push).

—lightbulb

RE: HM 0.9 development - the last portability hurdle - Added by jcw over 3 years ago

Hm. The only component giving me trouble with builds is Lua (and rs232’s current lack of Windows support).

Maybe Agora would be a better choice: https://github.com/PuerkitoBio/agora/wiki/Getting-started

It’s meant to be used for exactly the case needed in JeeBus: running small snippets of code dynamically. The use cases are: decoding and encoding small messages to / from text for the various hardware devices out there, and massaging MQTT or LevelDB a bit by subscribing to certain topics, picking up messages, and then passing them on to MQTT and/or LevelDB again.

Anything non-trivial, or non-instant, shouldn’t be running embedded in the server anyway. This would be just to side-step the fact that Go is a compiled environment and statically typed. The fact that it actually resembles the Go syntax could be a pre…

If I throw out Lua and bring in Agora, all the build issues are gone. No more install/build/mess-with Lua, no more tags arguments to think about.
By making the RS232 serial package optional, even Windows builds could be cross-compiled with gox. Then someone would simply have to set up a basic MQTT <
> COM port bridge and all the functionality in the core would be available across all platforms.

RE: HM 0.9 development - Added by lightbulb over 3 years ago

Hi,

Before you decide to ditch Lua, whats your problem with lua build specifically? (proviso, I have not tried to cross-compile yet).
and yes….I could see the ’tags luaa / llua ’ etc causing a build issue, especially in M\$W.
As for rs232, x-platform, I had planned to look into that specifically tomorrow, backed-up with a ’virtual serial’ concept.
Go can amicably ’LoadLibrary’…….to get round a few of these gotchas.
still playing catchup though I’m afraid
will be a few days before I can be more productive.

—lightbulb

RE: HM 0.9 development - Added by lightbulb over 3 years ago

btw: I like to look of Agora - seems nice fit, however, lets not shut the door on Lua for a few days…

—lightbulb

(1-25/188)