Starting on new Flow engine

Added by jcw over 2 years ago

I’ve started work on a new Flow engine in the dev branch on GitHub (this is a follow-up for issue #526):

  • I’m starting from scratch, trying to avoid “weak” decisions this time
  • switching back to TDD for real now, with the “Ginkgo” and “Gomega” test frameworks
  • you need to do go get -t to make Go download these test package dependencies
  • here’s a great way to run tests continuously: alias gwatch='ginkgo watch -succinct'
  • Output pins are now exposed as channels as well, i.e. full symmetry with Input pins
  • I’ve also hooked this up to Travis CI, using the new Go 1.3 release as test environment

Due to TDD, anyone interested can now easily follow along by reading the tests. As you’ll see, there’s almost nothing so far - it’s all baby steps, as I reset my mind to use the TDD approach. I’m not going to chase complete compatibility with the Flow engine we already have, but will try to stick as much as possible to the same terminology and functionality - as long as it will lead to a more dynamic design than the current one.

If you do follow along, your prompt comments and criticism (if anything looks like a dubious idea) would help me a lot while moving along and would be very much appreciated. And of course: feel free to fork and try alternatives at any point. My goal is to come up with a truly clean, modular, and robust core engine - the more we bang on that inner core, the better our chances of getting it right in Go.

IMO, Flow can become a truly natural way to harness Go’s channels and goroutines.

Replies (14)

flow.Tag replaced by NewTag(), TagOf(), and MsgOf() - Added by jcw over 2 years ago

As the subject title says - flow.Tag is being replaced by NewTag(), TagOf(), and MsgOf(), instead of exposing a flow.Tag type.
This should make it easier to deal with a mix of tags and non-tags in gadgets.

Now on to the big challenge, i.e. dynamically modifying circuits in a running setup:

For circuits, one idea would be to allow pausing it to make changes to the circuit. This could be implemented with (hidden) gadgets which sit between each circuit label and the gadgets inside the circuit. Or to put it differently: each label in a circuit becomes a special-purpose pipe-like gadget, instead of the current approach of treating labels as aliases. With this extra indirection, the flow of messages can be suspended to allow making changes to the circuit.

This resembles the way FBP addresses this problem in its compound nodes (i.e. circuits): make all the inputs look as if they have been closed to the rest of the circuit, which then causes all the connections (i.e. wires / channels) to drain. Once there are no more messages on (in?) a wire, it can be changed, connected differently, and therefore also allow adding or removing gadgets in the circuit. Once modified, all labels are then set up to act as pass-through for incoming messages again, and the flow processing resumes.

I’m going to try adding “func (c *Circuit) Pause() {...}” to Flow (note that only circuits can be paused, not gadgets).

Starting on new Flow engine - Added by tve over 2 years ago

WRT test frameworks, did you take a look at goconvey? It’s very cool and has a ton of helpers to test
expectations. I haven’t compared it with Ginkgo/Gomega yet.

RE: Starting on new Flow engine - Added by jcw over 2 years ago

I’m happy with ginkgo so far, see also for a comparison. On the Mac, I can get pass/fail notifications in the top right corner of the screen, which works well for me. It’d be interesting to compare the different approaches when developing remotely - something which will become more important once development on the BBB starts being useful.

No big investment so far, so if there are good reasons to adopt another system, we should decide soon.

RE: Starting on new Flow engine - Added by jcw over 2 years ago

I’m struggling a bit to find a good convention for messages-as-maps, so that it becomes easier to send JSON-like objects around. Right now, flow.Message is an interface{}, not a type, so it can’t have member functions of its own.

Maybe the solution is to define something like “type Message struct { interface{} }” - then it’ll probably be possible to write myMsg.GetString("field") iso the current flow.GetString(myMsg, "field"), which looks messier. I’m quite hesitant to abandon the generic form it is now, which supports sending any data item across channels.

Haven’t found the magic potion yet to become a Go master overnight, unfortunately! ;)

Update: I’ve switched back to a simpler design, using type Map map[string]Message. Not as automatic, but still clean, IMO.

RE: Starting on new Flow engine - Added by tve over 2 years ago

Ginkgo/Gomega actually look very good too. Maybe even better than Goconvey. Great to have choices! But imagine if they could pool forces….

WRT Message type, I really like the map[string]interface{}, but that’s the structured messages vs. not debate. In the end, to me the map is better defined and easier to reverse engineer when looking at someone else’s code and the message ordering and boundary issues with the other scheme drive me insane… The PacketMap definition does allow you to write myMsg.GetString(“field”) too. So that’s my vote ;-)

RE: Starting on new Flow engine - Added by jcw over 2 years ago

With the new design, you can turn any message into a map, regardless of whether it was one or not:

myMap := flow.AsMap(msg)

If msg wasn’t a map, it’ll be converted into one, with msg stored in key “”.
Not sure how tags should be treated, maybe they should stay, maybe not.

The concept of a stream of messages with tags inserted in between to mark boundaries is quite central in the original FBP design. Let’s see whether and when we need them, or if they can be jettisoned altogether at some point. One alternative would be to use a flow.Map with a specific structure to play the same role as tagged messages. I’m not too worried about that part.

I’ll ponder a bit on how to get the circuit mechanism going again. It’ll probably take some time to make that work.

RE: Starting on new Flow engine - Added by jcw over 2 years ago

Is anyone familiar with Pure Data? It’s an open source visual system, originally made for audio processing (similar to the commercial Max/MSP system), with lots of interesting concepts. Very nice intros and embedded help examples, when you download the “Pd Extended” version. See

Tons of design concepts which might be relevant in the context of HouseMon and Tosqa as well…

RE: Starting on new Flow engine - Added by jcw over 2 years ago

Here’s a screenshot of a live PureData control panel, demonstrating all the settings available in the serial port object:

It probably won’t make much sense without some background on Pd’s basics, and its visual representation, but this does contain a huge amount of configurability in a single screen, perfect for exploring all the settings. In real use, only a few of these settings would be used.

If you’re interested in thinking about / participating in the development of HouseMon, I really recommend downloading Pd-extended and trying out its embedded manual + examples. The concepts won’t map literally to HouseMon, but there is a lot to learn from how it deals with what are also hardware sensors and actuators, including the screen interface. The “network” examples are very nice, hiding a lot of complexity behind a simple visual appearance. OSC is also well integrated, and a good example how MQTT-like functionality can work with such an approach.

pd-serial.png (82.1 KB)

RE: Starting on new Flow engine - Added by SevenW over 2 years ago

@jcw: I am reading along, although I come from far.
The thing that I do not really understand is the need to modify circuits. I would say inserting and removing a circuit is needed, but runtime modifying it? Or is this just in the bigger picture that eventually all circuits are part of samoe master circuit?

The discussion on messages is probably on trying to get all parts of a message such as “id”, “tod”, “typ” not in seperate messgaes to a gadget, but in a compound messages so there is no debate on which belongs together? Very helpful, as I want to replace the timestamps with the timestamps of the remote logger.

RE: Starting on new Flow engine - Added by jcw over 2 years ago

Yes, modifying comes from seeing the whole picture: I want to be able to keep the system running at all times, and make changes to some part of the processing while the rest remains live. Also, it would be nice to see new remote nodes simply appear as new gadgets once packets come in (this is more important in Tosqa, which needs to deal with live insertion of new hardware).

After studying Pure Data in a bit more detail, I’m considering using a new type as basic datatype in Flow: a vector of arbitrary values, where the first one would usually be a string identifying either the data type or representing a message call to perform. This would help with structuring things: either as “positional args” convention in a pre-arranged order, or with a map as one of the entries to deal with named/keyed structures (like JSON objects). Haven’t decided whether flow.Message should be redefined as []interface{} for this, or whether such vectors would become merely another type alongside others.

I agree that having all the pieces arrive sequentially is inconvenient most of the time, but the moment you add structure, you also create a new type of potential impedance mismatch, i.e. when the data arrives in one “shape” whereas some gadget expects it in a different shape. With some simple primitives, converting shapes would be simpler, and vectors seem like a useful compromise between no structure at all and too much structure (which could require constant shape conversions, depending on the gadget).

RE: Starting on new Flow engine - Added by lightbulb over 2 years ago


I am still on hols, (back wed eve), and have tried to keep up with posts as
best I am able. Internet is rubbish and only got 2 vpns established in
whole time….grrrr

Anyhow, please keep mobile devices in mind.
I am aware of pd as my daughter uses it as for pre-amp shaping (she plays
The gui concept used within the version of pd she has would not translate
well to mobile devices at all if I remember it properly.

That said it may well have changed since I last helped her.

As for circuits, I have yet to need a realtime edit. I only seem to
currently need a edit-restart workflow, but I am sure there are domains
that need the pause-buffer-edit-restart workflow, so as to not loose serial
data for instance.

Fwiw, I currently have a traditional “property panel” like u find in ide’s
that is described via a meta data stream over a socket. Implemented as a
directive, is collapsible and can popout. Works great on my mobile devices
and desktop alike. The meta is also exposed as a rest api, so it can be
‘curled’. The point is, updates by mobile is really important for me, and a
minimalist easy typing interface is important for my fat fingers.. :)

Look fwd to picking things up again soon.

On 30 Jun 2014 07:36, wrote:

RE: Starting on new Flow engine - Added by jcw over 2 years ago

HM/TQ/JB was recently switched from Zurb Foundation to Pure CSS, which nicely supports responsive layouts. Try it with the latest version: make the window very narrow, and you can see it in action. It should be all set for mobile (touch will need some care for the circuit editor, but that’s not a top priority).

As for real-time edits: this becomes unavoidable once all system config is done through the browser. The point is that even if you don’t use dev-mode, any sort of editing of circuits and scripts running on the server will require some level of dynamism. I also really want to get rid of the current dispatcher hack.

RE: Starting on new Flow engine - Added by lightbulb over 2 years ago

I’m all good with my responsive gui. I use mobile 90% of time now as I said
a while back. Point I was making was for gadget parameter edits (as u would
need in circuit editor) and that the pd concept would be a bit clunky on
mobile (from what I remember) as you edit little ‘input’ boxes in the patch
editor (think that was what it was called). A tabular property editor is
what I have settled upon, and I tried a few concepts believe me…
I have also biased my widgets for “touch/gesture” support as I am now
mobile biased for my Monitoring/automation. In short, yes you guessed it, I
am now mobile biased for all but original setups. Even to point of shaking
my phone can turn all my electric appliances off last thing in evening -
sick I know, but u can get carried away with this stuff…:-D. Will check
what’s changed when I get back.

On 30 Jun 2014 09:47, wrote:

RE: Starting on new Flow engine - Added by jcw over 2 years ago

Just saw this note in the Pd manual:

> Pd runs on GNU/Linux, Windows, and Mac OS X, as well as mobile platforms like Maemo, iPhoneOS, and Android.

Not that this of any use to HM ;)