Topic delete reconcillation

[sehz]

This fix issue where topic delete is not completed when SPU goes offline.

Steps to reproduce and test:

Build and Set up Cluster

$ cargo build --bin fluvio
$ ./target/debug/fluvio start --local --develop
$ ./target/debug/fluvio topic create test
    topic "test" created

Kill SPU in the local cluster

$ ps -ef | grep fluvioubuntu
   ubuntu   3733903       1  5 04:34 pts/0    00:00:00 /home/ubuntu/fluvio/target/debug/fluvio cluster run sc
  ubuntu   3733913       1  0 04:34 pts/0    00:00:00 /home/ubuntu/fluvio/target/debug/fluvio cluster run spu -i 5001 -p     0.0.0.0:9010 -v 0.0.0.0:9011 --log-base-dir /tmp/fluvio
   ubuntu   3733968 3622789  0 04:34 pts/2    00:00:00 grep --color=auto fluvio
$ kill -9 3733913

Delete topic with SPU offline.

This will hang, so you have to press ^c

$ flvd topic delete  test
^c
Error: 
   0: Fluvio client error
   1: Fluvio socket error
   2: time out in serial: 1002 request: 1

The topic test will be delete mode but not removed because of finalizer

$  kubectl get topic test -o yaml
apiVersion: fluvio.infinyon.com/v1
kind: Topic
metadata:
  creationTimestamp: "2021-04-02T04:36:24Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2021-04-02T04:36:50Z"
  finalizers:
  - foregroundDeletion

SPU logs

$ ls /tmp/fluvio/spu-logs-5001
 test-0

Restarting SPU will complete delete

$ ./target/debug/fluvio cluster run spu -i 5001 -p 0.0.0.0:9010 -v 0.0.0.0:9011 --log-base-dir /tmp/fluvio starting spu server (id:5001)
SPU Version: 0.5.0 started successfully

Topics are gone:

$ ./target/debug/fluvio topic list
No topics found

And topic log directories are gone as well:

$ ls /tmp/fluvio/spu-logs-5001