Skip Mode

[wxtim]

Closes #5641 (also fixes #5975, and fixes #5820 )

Skip Mode Proposal Doc

This branch includes (marked against skip mode proposal):

1. A new task run mode called "skip" to sit alongside the existing "live" and "simulation" and "dummy" modes.
2. Skip mode settings in `[runtime][<namespace>][skip].
3. Task-level control of the run mode - allowing run mode of tasks to override that of the workflow using [runtime][<namespace>]run mode.
   a. Broadcast can change run mode for future task job submissions.
   b. Cylc Validate and lint will warn about the setting not being live.
4. cylc set --out skip sets outputs from skip mode.
5. Tests to ensure that run mode = skip respects is_held flag.
6. Tests to ensure that force triggering a task will not override the run mode.

Extras
7. Run Mode is available as an task attribute in the UI
8. When tasks are run in skip mode, the prerequisites which correspond to the outputs they generate should be marked as satisfied by skip mode rather than satisfied naturally for provenance reasons. For the purpose of cylc remove logic, satisfied by skip mode should be treated the same as satisfied naturally.

There are two extensions, which I haven't dealt with yet, because I want to ensure that the basic functionality works, and move to the substantial documentation PR which need follow this.

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• Feature: PR against master

[oliver-sanders]

Code getting close, small things:

[oliver-sanders]

Have started functional testing, looking good so far, a couple of small issues:

1) Cannot broadcast [skip] settings

E.G. This command should configure a task to run in skip mode AND fail when it does so:

$ cylc broadcast -s 'run mode = skip' -s '[skip]outputs = failed'

However, the [skip] part seems to be ignored. Not a biggie from a functionality perspective (why would anyone want to do this!), but if we're allowing users to do this it should work.

2) The started output isn't being produced?

This example stalls for me:

a? & a:started => b

[runtime]
  [[root]]
    run mode  = skip

3) Skip mode messes with the average task runtime

Skip mode tasks are not excluded from the average task runtime calculation which messes it up a fair bit.

Not a biggie, can be moved to a follow-on issue.

4) Skip mode messes with the analysis view

Skip mode tasks are not excluded from the analysis view statistics which is a tad confusing.

Not a biggie, can be punted into a small issue for the data workflows team. Just make sure that it is easy to filter out skip mode tasks in the task_jobs DB table (i.e. by filtering by platform name).

Opened: cylc/cylc-uiserver#626

[wxtim]

1. Cannot broadcast [skip] settings

Fixed by f2fb60e. It turned out to be a bit of a rabbit hole with disable task event handlers. I can see wanting to change this as being a reason to broadcast skip mode settings.

2. The started output isn't being produced?

No, but I don't think it's my change at fault - Since it appears to exist as an entirely separate bug present in Cylc as far back as 8.0.0! I'll attempt to fix anyway!

Proposed fix up at #6351 - since it's a bug. Will rebase and fix after.

3. Skip mode messes with the average task runtime

Fixed 660ca3a

I have noticed that dummy.py looks like

[oliver-sanders]

This integration test is actually running real jobs!

However, I don't think it's actually testing that the run mode is correctly overridden because it doesn't check how the task is submitted.

Also, it's testing live and skip modes, however, the workflow run modes are live and simulation?!

Here's my suggestion, it goes with the small task_job_mgr change I posted above (makes it easier to suppress live jobs from running whilst still testing non-live submission):

from cylc.flow.task_state import TASK_STATUS_SUBMITTED, TASK_STATUS_SUCCEEDED

@pytest.fixture
def capture_live_submissions(capcall):
    """Capture live submission attempts.

    This prevents real jobs from being submitted to the system.

    If you call this fixture from a test, it will return a set of tasks that
    would have been submitted had this fixture not been used.
    """
    def fake_submit(self, _workflow, itasks, *_):
        for itask in itasks:
            for status in (TASK_STATUS_SUBMITTED, TASK_STATUS_SUCCEEDED):
                self.task_events_mgr.process_message(
                    itask,
                    'INFO',
                    status,
                    '2000-01-01T00:00:00Z',
                    '(received)',
                )
        return itasks

    # suppress and capture live submissions
    submit_live_calls = capcall(
        'cylc.flow.task_job_mgr.TaskJobManager._submit_live_task_jobs',
        fake_submit
    )

    def get_submissions():
        nonlocal submit_live_calls
        return {
            itask.identity
            for ((_self, _workflow, itasks, *_), _kwargs) in submit_live_calls
            for itask in itasks
        }

    return get_submissions


@pytest.mark.parametrize('workflow_run_mode', ['live', 'skip'])  # TODO: Should this be 'simulation'?
async def test_run_mode_override_from_config(
    capture_live_submissions, flow, scheduler, run, complete, workflow_run_mode
):
    """Test that `[runtime][<namespace>]run mode` overrides workflow modes."""
    id_ = flow({
        'scheduling': {
            'graph': {
                'R1': 'live & skip',
            },
        },
        'runtime': {
            'live': {'run mode': 'live'},
            'skip': {'run mode': 'skip'},
        }
    })
    schd = scheduler(id_, run_mode=workflow_run_mode, paused_start=False)
    async with run(schd):
        await complete(schd)

    if workflow_run_mode == 'live':
        assert capture_live_submissions() == {'1/live'}
    else:
        assert capture_live_submissions() == set()

Note: This test requires #6326 (because I was too lazy to add an itask.jobs entry, but could do that also)

Could go further by mocking away the get_submission_method method to test which of the dummy submission methods is called if desired.

Might be worth pulling in the workflow run modes in case they change too:

@pytest.mark.parametrize('workflow_run_mode', [mode.value for mode in WORKFLOW_MODES])