Printing progress on add significantly slows down add

[Stebalien]

With #4296 applied, redirecting stdout to /dev/null speeds up add by ~17%. Something is waiting on this progress output that shouldn't be (we should be using a "lossy" channel to return progress updates, dropping any progress updates the client isn't ready for).

[Stebalien]

Hm. Actually, we can't do this because we send back all blocks added, not just progress updates. A CBOR API may help a bit but...

[kevina]

@Stebalien I have not even looked at the code yet, but would increasing the buffer size help?

[Stebalien]

Probably not (it'll just fill up).

After looking at it, we can, for larger files, skip progress updates if the buffer is full (I believe). Also, we can avoid blocking on printing updates (I'm working on that now).

[Stebalien]

Also, we don't buffer stdout/stderr. We should consider using double-buffering (with separate IO go routines for flushing).

[kevina]

@Stebalien see requirement (6) in #3856 (comment):

6. Ideally, the command code should not have to worry about connection errors, unless it needs to abort. That basically means that it should be able to just blindly send data, and errors, and only have to check if it should abort.

[Stebalien]

@kevina I don't see the connection. This is just an issue about perf (and dealing with the fact that stderr/stdout is slow).

[kevina]

@Stebalien part of the idea behind (6) is that the client should never block the server just because it is not ready to receive more output

[Stebalien]

Ah, I read that as an issue in programmer friendliness, not performance. Also, I kind of disagree. We generally want to block the server when the client isn't ready. Otherwise, we'd have to buffer all content we aren't ready to handle (could be gigabytes). This is a special case because dropping a few progress updates isn't a problem.