Embedding synchronous dataflow into a flow-based programming diagram

In my previous post I discussed about combining flow-based programming and syncrhonous dataflow both in a single visual programming language. The goal of such language would be to combine the best of  both of the flow-based approaches allowing the developer to choose the paradigm that best suits to the problem in hand. Today I want to take a deeper look into this hybrid dataflow model and introduce a specific synchronous structure that, from outside the structure appears and acts like a normal asynchronous flow-based programming node but from within encloses a synchronous dataflow diagram.

Consider an image processing task of smoothing an image by replacing each pixel value with an average of rectangular 3×3 neighborhood of the original pixel. We want to implement such an algorithm in our hybrid flow-based programming application. The input and output of such diagram needs to be a stream of images. Perhaps we have a flow-based programming diagram where we read images from a folder, pass them to the smoothing algorithm that then passes them to a node saving them back to the disk.

A Flow-Based Programming application that iterates over all of the images in a folder and smooths them using a Smooth Image subdiagram.

A Flow-Based Programming application that iterates over all of the images in a folder and smooths them using a Smooth Image subdiagram.

Now let’s take a look at the implementation of the Smooth Image subdiagram in the above flow-based programming example. The Smooth Image subdiagram is illustrated in the image below that takes advantage of the proposed new  synchronous structure. The synchronous structure in the subdiagram processes a stream of images, each of which is represented by a single information package (or an object in a OOP flavored language). The images arrive in an asynchronous manner from the previous Read Image step. In the examples, the asynchronous wires are represented with dotted lines and the synchronous wire with solid lines.

Embedding synchronous dataflow diagram into a flow-based programming diagram using a specific synchronous structure

Embedding synchronous dataflow diagram into a flow-based programming diagram using a specific synchronous structure

The synchronous structure starts executing upon startup of the subdiagram and remains executing until the input image stream is closed. The diagram waits for any incoming images (information packages) in the image stream. Each image in the example is expected to be a two dimensional array containing numeric values representing gray values for each pixel.

Every time an image arrives at the left connector of the synchronous structure, the content of the synchronous structure is executed exactly once in a synchronous dataflow manner. At the start of each execution of the content the synchronous structure, the left hand side image stream input gets the value of the received grayscale image. The image is then iterated with two for loop structures, outer for loop structure iterating each row in the image and inner for loop iterating each column in the image. Within the inner for loop, a 3×3 subarea around each pixel is averaged and the averaged value for each pixel is returned. The right hand edges of the for loops reconstruct the averaged pixels into the form of the original image and the smoothed image is passed to the right edge of the synchronous structure. The synchronous structure returns the image via its left hand side connector to the asynchronous surrounding diagram when its content has completed executing.

This hybrid dataflow approach using the synchronous structure allows easily enhancing the flow-based programming with synchronous dataflow subdiagrams. This greatly enhances the expressive power of the visual dataflow and flow-based programming languages by making them well equipped for both the complicated low-level operations requiring synchronization and high level orchestration well expressed with flow-based programming diagrams.

Related Posts

13 Comments

Herman Van Goolen

about 1 year ago

Are you trying to present a visual programming language that not only supports flow-related elements, but also structured programming elements? I believe that such an approach would result in a complexity that is hardly to show in a diagram. How would you express the flow of all exceptions? E.g. how do you handle the pixels on the corners and the edges? Going back to the level of the flowcharts of 50 years ago seems to require too much effort, even with the advanced graphical assistance of today. In FBP the flow diagrams are limited to a high level. The iterations and similar elements of structured programming are part of the components, without any specific graphic support.

Tomi Maila

about 1 year ago

Herman, I am playing a thought experiments of merging synchronous dataflow programming languages with those of flow based programming languages. Both have proven to work independently in a visual environment without issues. When you asked about pixels on corners and edges, what exactly did you mean. I didn't quite understand.

Herman Van Goolen

about 1 year ago

Tomi, in your diagram you are always assuming to use a 3x3 rectangular, but at the corners this would be limited to 2x2 and at the edges to 2x3.

Tomi Maila

about 12 months ago

There are various ways to treat the border and corner pixels. If we assume that the Sub Array node returns a smaller subarray on the extremas i.e. 2x3 on the borders and 2x2 on the corners, then the outcome of the sample application is identical to calculating the convolution of the image with a kernel of a 3x3 matrix with all elements having a value of 1/9 and assuming a zero filled border of one pixel surrounding the image. This is completely valid approach to image smoothing in the field of machine vision, but naturally there are others. One could for example divide the resulting sum array in the example with the size of such subarray instead of dividing with number nine. That way the extrema cases would contain the average of the 2x3 or 2x2 region for borders and corners respectively.

Herman Van Goolen

about 12 months ago

Tomi, it was not my intention to specify how the border and corner problem has to be tackled. I only got the impression that a diagram that shows all complexity (e.g. how to handle borders and corners) would be much more extensive, and that the use of such a diagram could become inefficient.

Tomi Maila

about 12 months ago

I think the paradigm of language for describing low-level algorithms is a matter of personal preference. I myself am a person who find myself much more fluent using a visual representation of all programming. I agree that for some problems, especially for algebraic problems, the text-based approach is more concise. The same way as Clojure application can call libraries written in Java code, I feel a visual programming language should in general be able to call libraries written with text-based programming languages. However I do not see needs to limit the visual programming language is such a way that certain problems become hard to solve, even if the visual solution is more verbose than a corresponding text-based solution.

Herman Van Goolen

about 12 months ago

Tomi, as a software architect I have always been using lots of graphical representations, in the event of structured programming maybe even rather close to what you are proposing. At the low level I only used it to explain details to individual programmers. Attempts to enforce the 40-50 programmers to draw similar low level flows failed. One of the reasons was that, whereas a graphically represented idea is often much easier to grasp and understand, it is much more difficult to compose a (2 dimensional) graph than to describe it in a (1 dimensional) sequential way. In addition it is much easier to modify (1d) text than (2d) graphics. The choice is mainly a matter of efficiency.

Yair

about 12 months ago

Herman, here's some LabVIEW code which does this (LV is presumably Tomi's main inspiration for the appearance of the dataflow code in these examples). It generates an NxN array of values and then smooths it based on the description here and it includes edge handling (although it does cheat a bit in that it does rely on LV's behavior for indexing part the end of the array). The top shows the code and the bottom shows the values after execution: I intentionally did not comment, prettify, verify or made it efficient. I didn't even put the duplicated code in a subroutine. The point was just to show that the basic algorithm can be relatively concise.

Yair

about 12 months ago

Apparently Tomi doesn't like us embedding HTML ;) Here's a link to the image (which would actually be executable code, if you had LV and if I paid attention to what I was doing): http://forums.ni.com/ni/attachments/ni/130/11086/1/Smoothed.png

Tomi Maila

about 12 months ago

Yair, thanks for your input. I added an ability to upload and attach images to new comments. Regarding your NI forums attachment link, is there a discussion going on at NI forums I am unaware of?

Yair

about 12 months ago

No, it was just a convenient place to host the image.

Yair

about 12 months ago

And for completeness, here's the image:

Tomi Maila

about 12 months ago

Thanks Yair for uploading the image here :)

Leave a Comment



Heads up! You are attempting to upload an invalid image. If saved, this image will not display with your comment.