Failed to connect D-Tale process in hosted notebook

[felihong]

Hi there,

I installed dtale v1.7.11 in my notebook, which is hosted in Google K8S engine as a pod. The DTale process can be assigned properly to a reference:

import dtale
import pandas as pd

df = pd.DataFrame([dict(a=1,b=2,c=3)])
d = dtale.show(df)

However I failed to connect to it as the the kernel keeps busy when running print(d).

I checked the pod logs and realized this has something to do with the port:

textPayload: 
"- - -" 0 - "-" 158 0 129 - "-" "-" "-" "-" "<POD_IP>:40000" 
PassthroughCluster <POD_IP>:52390 <POD_IP>:40000 <POD_IP>:52388 -

I've read that for hosted notebooks, the port 40000 has to be allowed but I'm not quite sure where should I start to configure this. My notebook container is running inside a pod, together with an istio sidecar proxy. The pod is managed by a StatefulSet which exposes a Service with a cluster ip.

I'm not quite sure if this is the right place to file this issue, plz let me know if I should rather head to the k8s side. And many thanks in advance!

[aschonfeld]

Oooh, kubernetes is a tough one. Let me talk to our core team and get back to you. We were able to get it running but it did take some work

[aschonfeld]

@felihong I just added this documentation on how we have D-Tale interacting with Kubernetes when using JupyterHub

Hope this helps. Let me know if you're still encountering errors. You'll probably want to get whomever manages jupyterhub or kubernetes involved as well.

[felihong]

Hi @aschonfeld , thanks for the detailed explanation!

Below a snippet of the original configuration of my notebook pod's service:

spec:
  clusterIP: <cluster-ip>
  ports:
  - name: http-notebook
    port: 80
    protocol: TCP
    targetPort: 8888
  selector:
    statefulset: notebook
  sessionAffinity: None
  type: ClusterIP

My intuition was modifying the service with multiple target ports, which should be supported by K8S as multi-port services

ports:
  - name: http-notebook
    port: 80
    protocol: TCP
    targetPort: 8888
  - name: http-notebook-dtale
    port: 80
    protocol: TCP
    targetPort: 40000

However this is immediately reconciled by my notebook controller and restored to the previous value (Has something to do with the default config). So I created a new service targeting to the notebook pod, with port value 80 and targetPort value 40000. Thus my pod has two services being exposed. But nothing has changed.

Interestingly, I can access to the DTale page (by visiting the ip directly) if I changed the service's type to LoadBalancer, which allocates an external ip endpoint to the dashboard page. Something is still going wrong with the internal traffic...

[javefang]

@felihong I think the problem here is that you can't map both port 8888 (notebook) and 40000 (d-tale) to port 80. If you pick a different port it should work

e.g.

ports:
  - name: http-notebook
    port: 80
    protocol: TCP
    targetPort: 8888
  - name: http-notebook-dtale
    port: 40000
    protocol: TCP
    targetPort: 40000

Also how are you exposing notebook? I see from the snippet that the k8s service it is a ClusterIP service. Do you also have an Ingress? If so, you also need to update that. Posting your Ingress config if you need more help.

[felihong]

Hi @javefang , sorry for the late reply.

You are right about the mapping, I specified the port as 40000 and the service yaml can be now updated properly. Thanks for pointing out.

My cluster is configured with an Ingress, which serves the main endpoint using istio-ingressgateway service:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-cert: <SSL_CERT>
    ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.global-static-ip-name: <IP_NAME>
    networking.gke.io/managed-certificates: gke-certificate
  name: envoy-ingress
  namespace: istio-system
  selfLink: /apis/extensions/v1beta1/namespaces/istio-system/ingresses/envoy-ingress
spec:
  rules:
  - host: <HOST_NAME>
    http:
      paths:
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
status:
  loadBalancer:
    ingress:
    - ip: <IP_NAME>

Should I add another backend to pointing to servicePort 40000? Thanks for your help!

[javefang]

Hi @felihong , you can map d-tale (port 4000) to a different path (e.g. /dtale) of the same ingress, given there is not to be an conflict with jupyterhub. Let me know how it works for you :)

[aschonfeld]

@javefang if you're still having issues I just recently released v.1.8.13 with support for a jupyterhub extension jupyter-server-proxy. Here's the documentation

Hopefully this might help with using D-Tale in jupyterhub on top of kubernetes