Restart a task running over PCP if it errors with 'puppetlabs.orchestrator/plan-already-finished'

[lucywyman]

Use Case

When running a task through the Orchestrator, the task gets added to PuppetDB with a plan ID for it to run as part of, in order to work with the Orchestrator API. If the Orchestrator is restarted as part of the task (for example as part of the peadm::pe_install task), it will fail with the error "The plan has already finished". This happens because when Orchestrator restarts it marks any running plans as failed, which also means they are interpreted as "finished". This is correct, however Bolt is using Orchestrator as a transport and doesn't care about whether the associated plan ID still exists or what it's state is, it just needs to use the Orchestrator to connect to targets and run the task.

To solve this, we can catch the plan-finished error and rerun the task with a new plan ID.

Questions

• How many times should we catch the error? Just once? Make it configurable?
• Are there other instances where we shouldn't catch the error and retry? (I can't think of any).