Explore possibility to enable serving etcd health check on a different port so we can have different tls requirement for etcd server and etcd health

[wenjiaswe]

Problem description:
If we enable TLS between etcd server and its client, for example, kube-apiserver, etcd healthness check will require tls credential as well. For use cases where we do not want anyone to access etcd data, but still want to expose healthness check with no credential required, it won't work.

Proposed solution:
How about enabling serving the healthcheck on a separate port so if needed, we can access etcd health without TLS?

xref: kubernetes/kubernetes#74690 (comment)

[jingyih]

The metrics and health are served separately from the clients. Looks like we can use "--listen-metrics-urls" to provide a different URL for metrics and health. The problem is that we use clientTLS when serving metrics and health, but (with some code change) it is possible to have different TLS configurations for metrics and health.

Reference:

1. Metrics and health are served separately from clients:
   

   etcd/embed/etcd.go

   Lines 231 to 236 in e1ca3b4

   	if err = e.serveClients(); err != nil {
   	return e, err
   	}
   	if err = e.serveMetrics(); err != nil {
   	return e, err
   	}

2. Their endpoints can be set using this flag:
   

   etcd/etcdmain/config.go

   Line 313 in 4cd0bf8

   cfg.ec.ListenMetricsUrls = flags.UniqueURLsFromFlag(cfg.cf.flagSet, "listen-metrics-urls")

3. However, clientTLS is used when serving metrics and health (line 765):
   

   etcd/embed/etcd.go

   Lines 755 to 765 in e1ca3b4

   	func (e *Etcd) serveMetrics() (err error) {
   	if e.cfg.Metrics == "extensive" {
   	grpc_prometheus.EnableHandlingTimeHistogram()
   	}
   	if len(e.cfg.ListenMetricsUrls) > 0 {
   	metricsMux := http.NewServeMux()
   	etcdhttp.HandleMetricsHealth(metricsMux, e.Server)
   	for _, murl := range e.cfg.ListenMetricsUrls {
   	tlsInfo := &e.cfg.ClientTLSInfo

4. But it is possible to have a separate TLS for metrics and health, with some code change:
   Allow to use own certificate for metrics endpoint #10504 Allow to use own certificate for metrics endpoint

[wenjiaswe]

Thanks @jingyih ! I think --listen-metrics-urls (available in 3.3) is probably gonna serve my need. I will try it out.

[wenjiaswe]

/close
--listen-metrics-urls works. Thanks!