Link MiqWorker record to a running pod when not created using run_single_worker.rb

[agrare]

Depends on:

• Fix non rails worker starting on podified #23116
• Always create opentofu_worker miq_workers record manageiq-providers-embedded_terraform#68

Fixes ManageIQ/manageiq-providers-embedded_terraform#67

[agrare]

I added some debug logging showing when we find a worker without a system_uid and a pod that matches the worker class:

{"@timestamp":"2024-07-30T17:27:45.709442","hostname":"orchestrator-5cf6b5b749-fxkfw","pid":7,"tid":"9808","service":"evm","level":"info","message":"MIQ(MiqWebServiceWorker#start) Worker started: ID [], PID [], GUID [89ac1fbe-81a3-4cc4-9afe-723ac46bb898]"}
{"@timestamp":"2024-07-30T17:27:45.713792","hostname":"orchestrator-5cf6b5b749-fxkfw","pid":7,"tid":"9808","service":"evm","level":"info","message":"MIQ(OpentofuWorker.sync_workers) Workers are being synchronized: Current #: [0], Desired #: [1]"}
{"@timestamp":"2024-07-30T17:27:45.811334","hostname":"orchestrator-5cf6b5b749-fxkfw","pid":7,"tid":"9808","service":"evm","level":"info","message":"MIQ(ContainerOrchestrator#patch_deployment) deployment_name: 1-opentofu-runner, data: {:spec=>{:replicas=>1}}"}
{"@timestamp":"2024-07-30T17:27:46.182359","hostname":"orchestrator-5cf6b5b749-fxkfw","pid":7,"tid":"9808","service":"evm","level":"info","message":"MIQ(MiqQueue.put) Message id: [1293], Zone: [default], Role: [], Server: [], MiqTask id: [], Handler id: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqEvent.raise_evm_event], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [[\"MiqServer\", 1], \"evm_worker_start\", {:event_details=>\"Worker started: ID [14], PID [], GUID [1d7f7921-aba9-4026-b1ea-d197f34d55cd]\", :type=>\"OpentofuWorker\"}]"}
{"@timestamp":"2024-07-30T17:27:46.182432","hostname":"orchestrator-5cf6b5b749-fxkfw","pid":7,"tid":"9808","service":"evm","level":"info","message":"MIQ(OpentofuWorker#start) Worker started: ID [14], PID [], GUID [1d7f7921-aba9-4026-b1ea-d197f34d55cd]"}
{"@timestamp":"2024-07-30T17:27:46.228408","hostname":"orchestrator-5cf6b5b749-fxkfw","pid":7,"tid":"9808","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#sync_starting_workers) AG: found a worker without a system_uid: Class [OpentofuWorker] Id [14]"}
{"@timestamp":"2024-07-30T17:27:46.228506","hostname":"orchestrator-5cf6b5b749-fxkfw","pid":7,"tid":"9808","service":"evm","level":"info","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#sync_starting_workers) AG: found a pod 1-opentofu-runner-84b54759f6-hhfs4 assigning to worker: Class [OpentofuWorker] Id [14]"}

[jrafanie]

@agrare are these legit test errors? Is they related to the ansible runner specs that Jason added recently?

[agrare]

@jrafanie the test failures look unrelated, I'll rebase and kick the tests

[agrare]

TODO exclude starting workers from cleanup orphaned worker rows and confirm that miq_workers are marked failed after the starting timeout on podified. Handle the case where a miq_worker record is created but the pod hasn't started yet.

[Fryguy]

@agrare and I discussed over video and I am concerned there is a race condition between actually starting the worker and sync_starting_workers. If the worker doesn't actually start before we hit that method, then it's possible that current_pods will not have the record. In that case you'd have a worker record with a blank system_uid that also will not get assigned a system_uid in sync_starting_workers, and then it will be deleted in cleanup_orphaned_workers. Then, on the next pass, current_pods will have the worker, but the database record won't exist, and there won't be anything to sync it to. That is, the pods_without_workers variable will have extra things (which shouldn't happen), but there isn't any code to check that case of extra values in there.

[agrare]

Okay I've done a live test where I manually skip any opentofu-runner pods to force the case where the record has been created but the pod is not created yet. Confirmed that we are not deleting the worker record during this period, pending the check on deleting the worker record after the 10 minute startup time.

[agrare]

Okay confirmed that the "worker starting but no pod is ever created" case behaves correctly:
{"@timestamp":"2024-08-21T20:16:46.619501","hostname":"orchestrator-5754f779b8-xkf42","pid":7,"tid":"83cc","service":"evm","level":"err","message":"MIQ(MiqServer::WorkerManagement::Kubernetes#exceeded_heartbeat_threshold?) Worker [OpentofuWorker] with ID: [40], PID: [], GUID: [2822a83c-7b5a-49d4-a67f-53cc36ff7f09] has not responded in 603.585520507 seconds, restarting worker"}

[agrare]

NOTE it looks a little weird to have .with(worker).and_return(worker) here but that is just the way the matcher works, without this line the worker object passed in to stop_worker does not have the mock expect(worker).to receive(:stop_container) applied and it actually drops into ContainerOrchestrator to try to delete the container. This ensures that stop_worker has the right object with the stubbed methods.

[agrare]

Okay live test forcing the race complete and specs should be green, taking out of WIP

[miq-bot]

Checked commits agrare/manageiq@4f070f7~...d7a2b57 with ruby 3.1.5, rubocop 1.56.3, haml-lint 0.51.0, and yamllint
2 files checked, 0 offenses detected
Everything looks fine. 🏆

[Fryguy]

Backported to radjabov in commit 71e76a6.

commit 71e76a6c1b0173bdf6fcb7e594cc0535fa309ebe
Author: Jason Frey <fryguy9@gmail.com>
Date:   Thu Aug 22 09:41:35 2024 -0400

    Merge pull request #23112 from agrare/miq_worker_worker_management_kubernetes_non_rails_system_uid
    
    Link MiqWorker record to a running pod when not created using run_single_worker.rb
    
    (cherry picked from commit a04c02ddf3452b083fe4e8205f4d41fbfa907927)