A Unified Table Data Type to specify data communications

[heyong4725]

A Unified Table Data Type to specify data communications

In this discussion, I would suggest we unify the data type between operators thru Table schema abstraction.

Basically we will have three main abstractions in DORA dataflow:

• Declarative dataflow specification thru YAML
• A Table data type to specify data communications between operators (including source/sink)
• Operators that compute over Tables

A Table has s schema, which is a list of column descriptors, each consisting of a name and an associated data type, which could be structured, semi-structured, or unstructured (e.g., image, etc.).
For example:

• [(timerID, int)], [(cameraID, int), (cameraName, string), (cameraData, image)]
• [(lidarID, int), (lidarData, byteStream)]

Thru this table schema, core relational operators can be provided by DORA, such as map, filter, join etc

User provided operators to implement user defined functions (e.g., .py, .so, .wasm), which are often ML inference workloads

In this way, it may also enable us to enforce type verification and encoding.