Default route not set

[ertimas]

Hi Andy,

I am running into a missing default ip route, which the k3s team solves via a dummy in this issue: k3s-io/k3s#1144.

It seems like you would have run into that? Did you have this configured prior to the script, or did I miss something?

Thanks for your time!

[clemenko]

I have not seen that before. Let's look at the basics.
What OS ? cat /etc/os-release
What nics ? ip a
did you modify the config.yaml?

[ertimas]

Hi Andy,
Thank you for getting back to me.

OS

Oracle Linux 8.10 server

NICS

Config

I didn't modify config.yaml, though I also didn't have the file in /etc/rancher/rke2/. So I symlinked one to the existing /etc/rancher/rke2/rke2.yaml. Which got through that issue. Though I'm left with a new one....

hauler.repo is in fileserver/.

Here's the output from netstat

Any thought on how to troubleshoot fileserver not seeing hauler.repo would be much appreciated. Note: my hauler directory is has global read/write permissions.

[clemenko]

the two issues may be related.
A. routes
B. hauler.

Can you try a clean install of Oracle 8 and see if any routes are there? /proc/net/route specifically?

[ertimas]

Here's /proc/net/route. Note I've got a second interface attached so that I can get the VM setup before "airgapping" it

[clemenko]

I wonder is the error was an isolated issue. Can you try and install it again?

Wait, are you disabling the NIC when installing the stack? Kubernetes needs an NIC at all times.

[ertimas]

Last time I left both NICs on during installation. I've tried it a couple times with each/both NICs, no ip given to hauler_all_the_things.sh, still the same issue

[clemenko]

there is no IP needed for the control function. can you run the control command with bash -x ./hauler_all_the_things.sh control ?

[ertimas]

I found that if I didn't provide an IP then it might look at the second NIC, and it looks like this made it happen.

Here is the result of bash -x ./hauler_all_the_things.sh control

Here's the log where createrepo was run

[clemenko]

Can you see if the hauler.repo file is there : curl -sfL http://10.237.13.41:8080/hauler.repo?

[ertimas]

I can see the file, but it's inaccessible if running the hauler server, see the yellow box. Note: if I run python3 http.server I can access the file from another box, which leads me to believe basic networking and file permissions are fine.

[clemenko]

I wonder is Hauler is not able to server anything on a node with 2 nics.
What does ss -tln show?

[ertimas]

It's shows the same thing as netstat -lnt essentially.

Manually running hauler store serve fileserver <my store directory> did work. It took about a minute for it to come up. Here are the debug logs in case they're helpful. Marking this as closed

[clemenko]

huh. I see you closed the issue. Is it working now?

[ertimas]

TLDR; I couldn't get your script to work.

There were two issues.

1. The fileserver service looks like it's fine, but after several minutes fails to be curlable. So, I ran hauler store serve -l debug fileserver <my hauler directory> and it worked. The command showed the fileserver taking >1minute for the to come up. See the screenshot I posted....
2. The registry fails to load. This is due to the a longhorn-csi component in the hauler store .zst being corrupted during tar/untar. Removing the csi components from airgap_hauler.yaml fixed it.