client-api: Rewrite websocket loop

[kim]

Split the websocket stream into send and receive halves and spawns a
new tokio task to handle the sending. Also move message serialization +
compression to a blocking task if the message appears to be large.

This addresses two issues:

1. The select! loop is not blocked on sending messages, and can thus
   react to auxiliary events. Namely, when a module exits, we want to
   terminate the connection as soon as possible in order to release any
   database handles.

2. Large outgoing messages should not occupy tokio worker threads, in
   particular when there are a large number of clients receiving large
   intial updates.

Expected complexity level and risk

4 - The state transitions remain hard to follow.

Testing

• Ran a stress test with many clients and large initial updates,
  and observed no hangs / delays (which I did before this patch).
  In reconnection scenarios, all clients where disconnected timely, but
  could reconnect almost immediately.

[gefjon]

I'd like to figure out what's going on with the SerializeBuffer and fix it before merging, but otherwise this looks good to me.

[kim]

Updated to:

• Reclaim the serialize buffer
• Not send any more data after sending a Close frame (as mandated by the RFC)

I think that we should also clear the message queue and cancel outstanding execution futures in the latter case, but that can be left to a future change.

[jsdt]

I looked through this for a while, and I'm still not very confident that I understand the error cases. I think we should do some bot testing with this to see what effect it has, but I think I'd like to try writing some tests for this, so we can trigger some of these tricky cases.

[kim]

Update to:

• split into smaller functions that mainly transform Streams. For readability and testability.
• actually terminate the actor loop when recv from the socket returns an error

[kim]

Updated to:

• consider that buffer reclamation can fail if the socket is already closed

• re-introduce spawning the send loop

  This seems to be necessary in order to guarantee timely release of the database.
  I'm considering to spawn the receive end, too, so that we can get rid of the unbounded buffer + apply backpressure to clients instead.

[kim]

Updated to:

• spawn the receive end, too