GRPC on AWS Application Load Balancer

This is an example GRPC application that runs as an ECS service on a custom ECS cluster with traffic routed via the internet through an Application Load Balancer.

This is my attempt at re-creating the demo from the following blog post

Like the post, I adopted the route guide grpc example from the official grpc repo and adapted it as follows:

• Added an option to the route_guide_client.py file to allow for secure channel communication. Default is False

• Added an option to the route_guide_server.py to allow for mutual TLS with the client. Default is False.

When running locally, use the Makefile to generate the libs from the protobuf and to generate the certs and build the docker container:

# create a .env file with an aws profile name and ecr repo link

source .env

make build-protobufs

make cert

make build-docker

To run locally:

# to use TLS auth
python route_guide_client.py --secure


python route_guide_server.py --secure

Deploying to AWS

The terraform scripts provision the following resources:

• ECS Cluster
• Application Load Balancer
• Fargate ECS Service
• Custom VPC with 2 public subnets and 2 private subnets ( ALB need at least 2 subnets )

To provision:

terraform -chdir=terraform init

terraform -chdir=terraform plan -var-file=config.tfvars -out myplan

terraform -chdir=terraform apply myplan

terraform -chdir=terraform output

Get the load_balancer output and export it as HOST env var which will be read by the client.

If all goes well with running against AWS, one should see the following output:

export HOST=<DNS A record of ALB from terraform output>

python route_guide_client.py --secure


REMOTE HOST >  route-guide-1954688975.us-east-1.elb.amazonaws.com

-------------- GetFeature --------------
Feature called Berkshire Valley Management Area Trail, Jefferson, NJ, USA at latitude: 409146138
longitude: -746188906

Found no feature at 
-------------- ListFeatures --------------
Looking for features between 40, -75 and 42, -73
Feature called Patriots Path, Mendham, NJ 07945, USA at latitude: 407838351
longitude: -746143763

Feature called 101 New Jersey 10, Whippany, NJ 07981, USA at latitude: 408122808
longitude: -743999179

Feature called U.S. 6, Shohola, PA 18458, USA at latitude: 413628156
longitude: -749015468

Feature called 5 Conners Road, Kingston, NY 12401, USA at latitude: 419999544
longitude: -740371136

To remove all AWS resources:

terraform -chdir=terraform destroy -var-file=config.tfvars

gRPC TLS Notes

• ALB has end-to-end support for HTTP/2 TLS so there's no need to enable HTTPS as protocol in the Target Group between the ALB and containers

  In this example we are using full HTTPS connections from ALB > gRPC service and back. The blog post shows a version where the protocol from ALB > gRPC is HTTP only, even though it uses HTTPS as a listener.

  If one tries to establish a client connection as per the blog post, it will fail with "SSL:Verify" errors.

  I suspect the reason is the blog post also uses a custom domain registered with Route53 and the same domain is in the certificates registered via the certificate manager.

  So perhaps thats why the client works without specifying any root certificates for verification ...

• To use a self signed cert on ALB, we need to sign the cert with subjectAltName set to both localhost and *.us-east-1.elb.amazonaws.com. This can be adjusted to suit...

  Note that this is by no means a secure way of running a production service. The server keys are unencrypted and really should use a KMS service to provision the certificates dynamically at run time...

• Also the certificate should be issued to a domain, and we can create an Route 53 Alias record that points to the DNS A record of the ALB ...

• If running with mutual client / server TLS on, we need to enable HTTPS for healthcheck else ALB fails...

• To inspect the generated TLS certs we can use openssl:

openssl x509 -in server.crt --text

References

GRPC Guide

Source

Running on ECS

Generate Certificate for ACM

SSL WITH GRPC

Section on TLS

TFX Serving on EKS example

Using TLS with GRPC

Article on GRPC over REST

ALB example using Terraform

Example of ECS Fargate service on private subnet with ALB

Example code for above

Replace listeners for elb using cli

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-update-certificates.html

https://docs.aws.amazon.com/cli/latest/reference/elbv2/modify-listener.html

aws --profile <myprofile> elbv2 modify-listener --listener-arn <listener_arn> --certificates CertificateArn=<certificate_arn>