bug: Connection refused (os error 111)

[alexanderGalushka]

I run deno in the docker container in the k8s pod for as a REST API service. Every N seconds this deno service sends a POST request with metrics data over to another service in k8s, prometheus pushgateway

here is the code snippet of how the POST request is being made with fetch:

// function to push metrics to prometheus "pushgateway"
export async function pushMetrics(
  id: string,
  metrics: string,
) {
  const pgURL = getPushgatewayHostName(pushgatewayJobName, id);
  try {
    const cleanedUpFileContent = unescape(encodeURIComponent(metrics));
    const out = new TextEncoder().encode(cleanedUpFileContent);  
    const response = await fetch(
      pgURL,
      {
        method: "POST",
        body: out,
      },
    );
    if (response.status != 202 && response.status != 200) {
      log.error(`failed to POST to pushgateway URL ${pgURL}, status ${response.status}`);
    }
  } catch (e) {
    log.error(`unexpected error: failed to POST to pushgateway URL ${pgURL}, ${e}`);
  }
}

function getPushgatewayHostName(pushgatewayJobName: string, id: string): string {
  const pushgatewayURI = `metrics/job/${pushgatewayJobName}/instance/${id}`;
  if (ENV == undefined || ENV == "") {
    return `http://pushgateway:9091/${pushgatewayURI}`;
  }
  return `http://${ENV}-${basePushgatewayK8sServiceHost}/${pushgatewayURI}`; 
}

permissions for the network calls are wide open:

CMD ["deno", "run", "--cached-only", "--unstable", **"--allow-net"**, "--allow-read=./workers", "--allow-env", "--v8-flags=--disallow-code-generation-from-strings", "main.ts"]

request comes through OK if I run it outside of k8s (not using k8s dns service host name e.g. pgURL=http://pushgateway.mynamespace.svc.cluster.local:9091): bring up 2 docker containers attached to the same network, so the deno service POST to http://pushgateway:9091, pushgateway is the name of the second container deno service is POSTing to

if I run it in the k8s I get the following error:

unexpected error: failed to POST to pushgateway URL http://myenv-pushgateway.mynamespace.svc.cluster.local:9091/metrics/job/myjob/instance/1234567890, Http: error sending request to url (http://myenv-pushgateway.mynamespace.svc.cluster.local:9091/metrics/job/myjob/instance/1234567890): error trying to connect: tcp connect error: Connection refused (os error 111)

looks like a Rust error message:

error trying to connect: tcp connect error: Connection refused (os error 111)

I can exec into the deno container in k8s and curl pushgateway, so it's not related to k8s network policies

not sure if these 2 issues are related:
#7660
#6751

using:

• latest deno v1.4.2
• k8s cluster v1.18.6

@bartlomieju @kitsonk @hayd can you please take a look

[kitsonk]

You don't need to @ people to have a potential issue looked at.

[hayd]

This certainly reads like #6751.

[alexanderGalushka]

You don't need to @ people to have a potential issue looked at.

true

[alexanderGalushka]

has anyone looked at this?

[puneetk]

bump +1

[Merulast]

Same thing appeared again on my setup. Suddenly with the recent update of the docker image (denoland/latest) Iam not able to connect to my mariadb nodes anymore. If lucky I get the os error 111, if not the entire thread just get hung.

Also reproductable with the smalest possible mysql connection sample excluding all higher level libs.

[itsahsiao]

Same here @Merulast 😞

We're also having issues with our application on production connecting to our supabase postgres db. No issues when running the application locally or when connecting to the database directly.

[bartlomieju]

@littledivy please take a look

[itsahsiao]

We resolved it by updating our deno postgres driver to the latest version. We were on postgres@v0.16.0 and changing to postgres@v0.17.0 seemed to fix it for some reason.

[hamanuha]

I’m having the same issue. I can’t connect to the local postgres database (using supabase).

[hamanuha]

I’m having the same issue. I can’t connect to the local postgres database (using supabase).

It works when using 172.17.0.1 instead of localhost

[cohlar]

I have the same issue. Thanks for sharing @hamanuha , replacing localhost by 172.17.0.1 works for me as well.

[Peter-Kang]

I’m having the same issue. I can’t connect to the local postgres database (using supabase).

It works when using 172.17.0.1 instead of localhost

I had the same issue connecting from deno to mongodb over docker. This solved my issue. Thank you so much for shareing!

[topperspal]

I’m having the same issue. I can’t connect to the local postgres database (using supabase).

It works when using 172.17.0.1 instead of localhost

Wow this just worked!

I am using supabase and deno edge function. Could you explain how this works?

[anil-zenollect]

it's Docker's gateway for its bridge network.

try running command in your terminal docker network inspect bridge

[SaiHarshaVardhan-ez]

I have the same issue. Thanks for sharing @hamanuha , replacing localhost by 172.17.0.1 works for me as well.

-- hey ! i was getting the same error while working with supabase -- i have tried to change it to 172.17.0.1 it did not work it is giving connection timed out error can you help me with this

[lucacasonato]

This is likely related to localhost resolving to an IPv6 address, but the port not actually listening on IPv6. This would be fixed by us implementing Happy Eyeballs: #25661. I'll close this in favor of that.