Since the summer I have been building dataflows. A dataflow takes a data source and puts an ordered list of steps on top of it: filter, rename, change a type, join, look something up. What comes out is a new list that behaves like any other data source, and nobody had to write a line of Lua to get there.
Data never arrives in the shape a board needs. A machine reports one column too many. A database gives you the article number but not the article name, and the name sits in a different table. Two sources have to be brought together on a key. Until now the answer to all of that was a script: read the list, loop over it, build a new table. That works, and nobody enjoys doing it.
A step is one small named operation on a table. The steps are ordered, and each one can be renamed, disabled, moved or deleted after the fact. The preview shows the data as of the step you have selected, so you see what a step did the moment you add it. That feedback is what makes the whole thing usable.
What exists so far: remove, rename and reorder columns, change a type, filter, sort, remove duplicates, take a range of rows, switch rows and columns, format numbers and dates, replace text, add a column, aggregate, union, and join with the four usual types (inner, full, left and right).
The one people asked for most is the lookup column. The machine reports an ID, the readable text lives somewhere else, and a lookup column fills it in.
A dataflow can also build on another dataflow, so a long chain can be split into parts that each make sense on their own.
That’s it.
If you have questions feel free to contact me.
Links:
Your first steps with Dataflows →
Link table-like datasets using joins →