Sync waves complete when objects are still "Progressing"

[hintofbasil]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a
question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When using the app of apps pattern with sync waves the sync waves completes when the application is marked as "Synced" but doesn't wait for the application to be "Healthy". If the application contains a deployment this results in the sync wave progressing when the new pods are not in a healthy state.

To Reproduce

• Clone https://github.com/hintofbasil/ArgoCD-sync-waves-test

• Apply app_of_apps.yaml into a cluster with ArgoCD. The app of apps will behave as expected - it will sync the first application, wait for the pod to reach a ready state, sync the second application, wait for the pod to reach a ready state, then mark itself as healthy and synced.

Change the delay parameter in app_of_apps.yaml then apply the resource. This time the pods will be updated at the same time.

Expected behavior

On the second apply of the app of apps I would expect the first pod to reach a ready state before the second sync wave beings.

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v1.6.1+159674e.dirty
  BuildDate: 2020-07-21T14:27:57Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: dirty
  GoVersion: go1.14.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.6.2+3d1f37b
  BuildDate: 2020-07-31T23:45:07Z
  GitCommit: 3d1f37b0c53f4c75864dc7339e2831c6e6a947e0
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

Logs

Paste any relevant application logs here.

[jessesuen]

Because of numerous bugs like this with the app-of-apps pattern, we're actually removing the health status for Application resources in the v1.8 release -- we can't seem to get the sync waves with apps working correctly.

The long term solution is to introduce a AppSync CRD which can deal with coordinating syncing of multiple apps, correctly.

[hintofbasil]

Thanks for the heads up Jesse, we were investigating a design that relied on that status being accurate so very glad to know early.

Looking forward to reading the spec mentioned in #1283.

[ronaldour]

@jessesuen Is there any known workaround to orchestrate the deployment order of applications in the meantime?
We are facing the same issue, the sync-waves work perfectly on the initial deployment, but not when syncing the app after that. It seems that it just updates the Application resource but it doesn't wait for the actual resources of the application to be updated.

[ronaldour]

We created a custom health check for the argoproj.io/Application resource type and seems to be working so far. I'm not 100% sure it works all the time but it's worth trying as a workaround.

argoproj.io/Application:
        health.lua: |
          hs = {}
          if obj.status ~= nil then
            if obj.status.sync.comparedTo.source.targetRevision ~= obj.spec.source.targetRevision then
              hs.status = "Progressing"
              hs.message = "Application Syncing"
              return hs
            else
              if obj.status.health.status == "Healthy" and obj.status.sync.status == "Synced" then
                hs.status = "Healthy"
                hs.message = "Application Ready"
                return hs
              end
            end
          end

          hs.status = "Progressing"
          hs.message = "Waiting for Application"
          return hs

[rajivml]

@ronaldour I have replaced the default application health check logic with yours and all apps got stuck in "Progressing" and "Waiting for Application" status , even though they are synced correctly. So, looks like there is something wrong with your code where as with the below code status is being reported correctly

        hs = {}
        hs.status = "Healthy"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.health ~= nil then
            hs.status = obj.status.health.status
            hs.message = obj.status.health.message
          end
        end
        return hs

And if you notice, all of them are synced,