Hybrid Dataflow – Convergence of Flow-Based Programming with Synchronous Dataflow

In my previous post I discussed the differences between two visual programming paradigms; synchronous dataflow programming and asynchronous flow-based programming. Although the two programming paradigms are approaching the visual programming from little different perspectives, both approaches converge at some subset of the visual programming domain. Let’s look at a very simple example of multiplication of two constant numbers.

Flow diagram for multiplying two constants

Flow diagram for multiplying two constants

The above diagram would multiply two constant numbers 7 and 5 and return 35 when evaluated. This diagram would be exactly the same in synchronous dataflow and in asynchronous flow-based programming; in both approaches the diagram needs to be evaluated only once. Flow-based programming and synchronous dataflow can be seen as converging into a single approach when only constant values are being used. Of course one cannot solve any real computation problems with only constant values.

What if one of the inputs to the multiplication primitive was an asynchronous stream of numeric values instead. For each numeric value in the stream, the primitive would multiply the stream value with a provided constant value and stream the output values out as they are evaluated. This is the approach that the flow-based programming paradigm takes, with dialects that support replacing streams with constant values. This approach is illustrated in the image below.

Flow diagram for multiplying a constant with a stream

Flow diagram for multiplying a constant with a stream

I have used thick connector lines to illustrate streams and thin connector lines to illustrate one-time values. In the above example, one of the inputs (x) is treated as a stream the other input (y) is treated as a value.

Indeed there is no reason why a hybrid of the two approaches to dataflow programming couldn’t coexist in the same flow-based programming language. We are already using a hybrid approach in the above diagram by using both stream and constant value inputs to the same node. In a general convergent hybrid approach, each node with a stream input would continue executing as long as the stream is valid (i.e. the stream is not closed by the runtime). The node would process each element in the stream one at a time, in the conventional flow-based programming manner. Non-stream one-time inputs would be treated in a synchronous dataflow approach. One-time value input to a flow-based programming node would be treated as an infinite stream that would always return the same (not necessarily constant) value.

Consider the following example. Synchronously multiply two input values and use the result of the multiplication as an input for an asynchronous operation as one of the multipliers. This example is illustrated in the image below. The operation multiplies the result of the left most operation together with each value in the stream x’. The result is evaluated when ever values arrive at the stream x’ just like in our past examples.

Hybrid approach combining flow-based programming with synchronous dataflow programming.

Hybrid dataflow approach combining flow-based programming paradigm with synchronous dataflow programming.

Combining the synchronous dataflow and asynchronous flow-based programming as first class citizens in the same visual programming language has some clear advantages. Some problems are naturally easier to both comprehend and solve in a synchronous approach and some flow-based programming design patterns would become simpler if a synchronous dataflow was a first-class citizen of the language. This is especially true for low-level operations where synchronous dataflow is often the way programmers naturally approach problems. Asynchronous approach naturally works very well on orchestrating the system and is very natural in designing system architectures and defining dependencies.

Related Posts

2 Comments

Mark

about 10 months ago

Great article! I am looking for an article that seems to not be on this site anymore. It is the queue article with all the great visuals and explanation. Could you please upload it? Thank you!

Tomi Maila

about 10 months ago

Mark, unfortunately the old content is not currently available. I am figuring out the best way to bring it available for the interested readers.

Leave a Comment



Heads up! You are attempting to upload an invalid image. If saved, this image will not display with your comment.