Skip to content

cleichner/pipeline

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

33 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Pipeline

This library defines a decorator that turns any function into a zmq pipeline stage.

Example time

This is a boring function.

def echo(data):
    return data

And this is a boring part of a zmq pipeline that takes input from ports 5555, 5556, and 5557 over tcp on localhost and echos everything it reads to port 5558.

from pipeline import pipeline

@pipeline(in_ports=[5555,5556,5557], out_port=5558)
def echo(data):
    return data

echo.run()

What do the examples do?

There are four example programs that cover every part of a pipeline.

  • source -- continuously prints numbers on --out-port
  • read -- reads from stdin and pushes to --out-port
  • work -- reads from --in-ports, doubles the input, and puts it on --out-port
  • sink -- reads from --in-ports and prints what it reads to stdout

How do I run them?

To set up a pipeline like this:

source (5555) -->
                  work (5556) --> sink
read   (5558) -->

In one shell start this (so you can see the output):

python source.py --out-port 5555 &
python work.py --in-ports 5555 5558 --out-port 5556 &
python sink.py --in-ports 5556

In another shell start this (so you can interact with it):

python read.py --out-port 5558

About

Python decorator library for easy ZMQ pipeline creation

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages