gRPC client retries

[JamesNK]

Fixes #1189

This PR adds support for gRPC retries. When retries are enabled, a wrapping call (either RetryCall or HedgingCall) is created that will manage making HTTP requests to the server. The wrapping call will create GrpcCall instances that will perform the HTTP request to the server.

• All of the gRPC retries spec is supported.
• GrpcCall is mostly unchanged.
• RetryCall and HedgingCall (and their base type) handle retry logic, message buffering, throttling, etc. Once a gRPC call is commited then reposibility is handled back to it for returning a response.
• Strongly typed ServiceConfig type for client configuration. Currently it is only ever manually created by the client, but in the future deserializing from JSON/Protobuf will be supported for populating it by a naming resolver.
• Pay-for-play. gRPC calls that don't have retries enabled will use GrpcCall without performance impact of features required by retries.
• Clear separate between retry logic and standard HTTP gRPC calls.

---

Other bugs fixed during this PR:

• Fix - An error from the server to the client will now correctly update the status of in-progress server streaming and bidirectional streaming calls. This matches Grpc.Core behavior. Fix response stream errors not updating call status #1198
• Change - A completed client streaming and bidirectional streaming call in the client will throw RpcException from RequestStream.WriteAsync. This matches Grpc.Core behavior. Change some RequestStream.WriteAsync errors to match Grpc.Core #1199
• Change - Don't log cancellation exception from HttpClient on deadline exceeded in client. Fix deadline timer rescheduling because of timer precision #1201

---

TODO

• Hedging with exceeded deadline
• Hedging with throttling
• Retry limit configuration
• Buffer limit configuration
• Refactor configuration objects to improve intellisense
• Handle ResponseStream.MoveNext cancellation

---

Work in progress on https://github.com/grpc/proposal/blob/master/A6-client-retries.md

var serviceConfig = new ServiceConfig
{
    MethodConfigs =
    {
        new MethodConfig
        {
            Names = { Name.AllServices },
            RetryPolicy = new RetryThrottlingPolicy
            {
                MaxAttempts = 5,
                InitialBackoff = TimeSpan.FromSeconds(1),
                RetryableStatusCodes = { StatusCode.Unavailable }
            }
        }
    }
};

var channel = GrpcChannel.Create("https://localhost:5001", new GrpcChannelOptions
{
    ServiceConfig = serviceConfig
});
var client = new Greet.GreeterClient(channel);

// Make gRPC call with retries
var reply = await client.SayHelloAsync(new HelloRequest { Name = ".NET" });
Console.WriteLine(reply.Message);

[Tratcher]

The lazy config model doesn't seem like a good fit here. It results in a lot of per-request validation and parsing (sometimes repeatedly per request) that could have been done once up front when creating the channel.

[JamesNK]

@halter73 @JunTaoLuo @jkotalik @wtgodbe @davidfowl

[JunTaoLuo]

Some minor questions

[JunTaoLuo]

Changes look good though I wouldn't be surprised if there are some synchronization corner cases that we missed. Given that the features added are marked with experimental status, I'm okay with merging the changes and fix issues that come up.

Let's wait for @jtattermusch to take a look as well.

[jtattermusch]

Left a few comments, but overall LGTM. Thanks for implementing this!

[JamesNK]

🍰 🥳 🎈 🎉