New command: shell

[anatoly-scherbakov]

columns:
  uuid:
    shell:
      command: uuid -v4
      pipe: false

ysv should leverage the ecosystem of UNIX command line tools. It should permit the user to process the values of a given column through an external program.

There are multiple use cases to this.

1. As we have seen in practice, sometimes ysv's built in filters are not enough. We have to write custom code in another language to do some sort of complex processing for particular columns.

With shell command, we could teach ysv to call our Python script in a separate process and feed the values, line by line, to that script. It will read the output from stdout of the script and incorporate the resulting values into the output CSV dataset.

This would make ysv enormously extensible. Moreover, we could allow it to run multiple instances of the external program and thus facilitate the multiprocessing capabilities of modern hardware (which, say, Python alone cannot easily do).

2. Even without custom code, the communication using UNIX pipes allows to use standard command line tools, for example awk.

In both of these cases, we will get substantial expansion in functionality by leveraging tools that already exist out there, – and we can do that with great efficiency.

[anatoly-scherbakov]

More examples.

columns:
  number:
    input: number_plus_five
    shell: awk { $1 + 5 }

or

columns:
  phone_number:
    input: Phone
    shell: python run.py validate_united_states_phonenumbers

In each case, ysv runs the provided shell command as another process (or processes) and feeds the input values to the stdin of that command. It then reads the processed values from stdout and inserts them into the output CSV dataset.

[anatoly-scherbakov]

It seems jq team is working on something similar: jqlang/jq#147