WIP: Introduce TrialRunner Abstraction #720

bpkroth · 2024-03-19T21:24:40Z

This is another step in adding support for parallel trial execution #380.

Here we separate out the running of an individual trial to a single class - TrialRunner.

Multiple TrialRunners are instantiated at CLI invocation with the --num-trial-runners argument.
Each TrialRunner associated with a single copy of the root Environment, and made unique by means of a unique trial_runner_id value that's included in that Environment's global_config.

TODO:

tests

In future PRs we will add:

New Scheduler implementations to run TrialRunners in parallel.
Async polling of status results in each TrialRunner independently.

…nit test for bench (not tested)

…e bulk registration (check for is_warm_up)

…ion loop

…eduler

mlos_bench/mlos_bench/launcher.py

bpkroth · 2024-09-23T20:00:11Z

mlos_bench/mlos_bench/storage/sql/trial.py

@@ -46,6 +46,16 @@ def __init__(  # pylint: disable=too-many-arguments
        self._engine = engine
        self._schema = schema

+    def _save_new_config_data(self, new_config_data: Dict[str, Union[int, float, str]]) -> None:


Needs tests

bpkroth · 2024-09-23T20:00:35Z

mlos_bench/mlos_bench/storage/sql/common.py

@@ -145,6 +161,7 @@ def get_results_df(
                "tunable_config_id",
                "tunable_config_trial_group_id",
                "status",
+                "trial_runner_id",


Needs tests

bpkroth · 2024-09-23T20:00:53Z

mlos_bench/mlos_bench/storage/base_trial_data.py

+        """ID of the TrialRunner."""
+        if not self._trial_runner_id:
+            self._trial_runner_id = self.metadata_dict.get("trial_runner_id")
+        return self._trial_runner_id


Needs tests

mlos_bench/mlos_bench/schedulers/base_scheduler.py

motus added 30 commits February 21, 2024 15:34

do not pass the optimizer into _run()

616f44e

mypy fixes

33e332a

start splitting the optimization loop into two

0247259

first complete version of the optimization loop (not tested yet)

483e378

Merge branch 'main' into sergiym/run/2loops

addd5a4

allow running mlos_bench.run._main directly from unit tests + add a u…

e97266f

…nit test for bench (not tested)

move in-process launch to a separate unit test file

64771fd

add is_warm_up flag to the optimization step

bd7c55e

Merge branch 'main' of github.com:microsoft/MLOS into sergiym/run/2loops

387722a

in-process optimizaiton loop invocation works!

9f15aee

add multi-iteration optimization to in-process test; fix the mlos_cor…

65cd072

…e bulk registration (check for is_warm_up)

make in-process launcerh tests pass

c010d95

remove unnecessary local variables to make pylint happy

7cfef3a

move trial_config_repeat_count checks to the launcher

7233180

make experiment.load() return trial_ids and use them in the optimizat…

be7dcec

…ion loop

use proper last_trial_id in the main loop; fix the unit tests

3c52e03

update launcher tests with the new output patterns

0d9dc97

remove unused variable

4e171e0

Merge branch 'main' into sergiym/run/2loops

ab69fa0

better naming for functions in the optimization loop

52adab8

start implementing the scheduler class

df893d9

change the default value for is_warm_up parameter to False

5aca764

Merge branch 'sergiym/run/2loops' into sergiym/run/scheduler

4d183df

started to implement teh start() method of the sync scheduler

309e10c

Merge branch 'main' of github.com:microsoft/MLOS into sergiym/run/sch…

9a72b40

…eduler

implement proper Scheduler constructor

ffe23e1

more clean-ups to the base scheduler

cb863e0

minor pylint fixes

990b019

add _add_trial_to_queue() method

2ac0520

better handling of warm-up phase (no redundant code)

b95100a

bpkroth added 3 commits July 23, 2024 22:02

Merge branch 'main' into trial-runner-abstraction

2b4aaba

apply suggestion

5a327dd

formatting

49087c4

bpkroth commented Jul 23, 2024

View reviewed changes

mlos_bench/mlos_bench/launcher.py Outdated Show resolved Hide resolved

bpkroth added 3 commits July 23, 2024 22:34

wip: assigning new trial runner ids to old trials

0ac79aa

Merge branch 'main' into trial-runner-abstraction

5c772cd

Merge branch 'main' into trial-runner-abstraction

0657ece

bpkroth mentioned this pull request Sep 23, 2024

Move to azure VM managed run commands #853

Open

bpkroth added 4 commits September 23, 2024 19:13

Merge branch 'main' into trial-runner-abstraction

18536e6

format

094cbf6

refactor to allow easier scheduling overrides

6f54b21

reformat

0ac19ae

bpkroth commented Sep 23, 2024

View reviewed changes

comments

eab3c6e

bpkroth commented Sep 23, 2024

View reviewed changes

mlos_bench/mlos_bench/schedulers/base_scheduler.py Outdated Show resolved Hide resolved

bpkroth added 12 commits September 23, 2024 20:24

tweaks

b1fa8c2

wip

a48e831

Adding status() output to MockEnv

6557291

expose root_env_config property

2877df1

refactor to use scheduler

e3005a0

Merge branch 'main' into use-scheduler-in-dummy-runs

14cd3fe

wip

376461a

tweaks to metadata checks

a61678c

comments

fbdf3a1

Merge branch 'use-scheduler-in-dummy-runs' into trial-runner-abstraction

192dd80

comments

cfbe440

Merge branch 'use-scheduler-in-dummy-runs' into trial-runner-abstraction

a5edafc

bpkroth mentioned this pull request Sep 27, 2024

Use scheduler in dummy runs #861

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Introduce TrialRunner Abstraction #720

WIP: Introduce TrialRunner Abstraction #720

bpkroth commented Mar 19, 2024 •

edited

Loading

bpkroth Sep 23, 2024

bpkroth Sep 23, 2024

bpkroth Sep 23, 2024

WIP: Introduce TrialRunner Abstraction #720

Are you sure you want to change the base?

WIP: Introduce TrialRunner Abstraction #720

Conversation

bpkroth commented Mar 19, 2024 • edited Loading

bpkroth Sep 23, 2024

Choose a reason for hiding this comment

bpkroth Sep 23, 2024

Choose a reason for hiding this comment

bpkroth Sep 23, 2024

Choose a reason for hiding this comment

bpkroth commented Mar 19, 2024 •

edited

Loading