Redesign Adapter subsystem #876

gkirgizov · 2022-09-13T11:29:49Z

Adaptation subsystem is refactored -- now it's more high-level and declarative, requiring less attention from developers.

Main changes:

It's possible to adapt functions that must work with Opt Graphs instead of adapting their arguments & return values.
It works like this: adapt_func( f(Pipeline)->Pipeline ) => f'(OptGraph)->OptGraph
That is, from the function that can work with domain graphs (e.g. Pipeline) we get a function that can work with OptGraphs (it automatically adapts its arguments and return type).
Ad-hoc check for whether the function must be adapted (like isinstance(mutation_type, Callable)) is dropped -- it is determined automatically inside BaseAdapter.
Additional functions are introduced for adapting/restoring whole populations: adapt_population
Functions that shouldn't be adapted can be marked with a special decorator @register_native -- these functions are supposed to work with OptGraph directly.
Added separate unit tests for adapters

See doc-comment of the new class AdaptRegistry for more detailed description.

Aimed at #742

pep8speaks · 2022-09-13T11:30:08Z

Hello @gkirgizov! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file examples/advanced/fedot_based_solutions/graph_model_optimization.py:

Line 10:1: F401 'fedot.core.adapter.AdaptRegistry' imported but unused

In the file fedot/core/adapter/__init__.py:

Line 1:1: F401 '.adapter.BaseOptimizationAdapter' imported but unused
Line 1:1: F401 '.adapter.DirectAdapter' imported but unused
Line 2:1: F401 '.adapt_registry.AdaptRegistry' imported but unused
Line 2:1: F401 '.adapt_registry.register_native' imported but unused

In the file fedot/core/composer/gp_composer/gp_composer.py:

Line 3:1: F401 'fedot.core.adapter.AdaptRegistry' imported but unused

In the file fedot/core/composer/gp_composer/specific_operators.py:

Line 2:1: F401 'typing.Any' imported but unused

In the file fedot/core/dag/graph.py:

Line 3:1: F401 'typing.Any' imported but unused
Line 3:1: F401 'typing.Generic' imported but unused

In the file fedot/core/dag/graph_delegate.py:

Line 2:1: F401 'typing.Callable' imported but unused

In the file fedot/core/dag/graph_operator.py:

Line 206:16: E126 continuation line over-indented for hanging indent

In the file fedot/core/optimisers/adapters.py:

Line 6:1: F401 'fedot.core.optimisers.gp_comp.individual.Individual' imported but unused
Line 8:1: F401 'fedot.core.pipelines.node.Node' imported but unused
Line 10:1: F401 'fedot.core.pipelines.template.PipelineTemplate' imported but unused

In the file fedot/core/optimisers/gp_comp/evaluation.py:

Line 2:1: F401 'multiprocessing' imported but unused

In the file fedot/core/optimisers/gp_comp/gp_optimizer.py:

Line 8:1: F401 'fedot.core.optimisers.gp_comp.individual.Individual' imported but unused

In the file fedot/core/optimisers/gp_comp/operators/mutation.py:

Line 4:1: F401 'typing.Sequence' imported but unused

In the file fedot/core/optimisers/gp_comp/operators/regularization.py:

Line 1:1: F401 'copy.deepcopy' imported but unused
Line 4:1: F401 'fedot.core.adapter.AdaptRegistry' imported but unused
Line 48:29: W503 line break before binary operator

In the file fedot/core/optimisers/opt_history.py:

Line 200:35: F541 f-string is missing placeholders

In the file fedot/core/optimisers/optimizer.py:

Line 3:1: F401 'typing.Union' imported but unused

In the file fedot/remote/run_pipeline.py:

Line 9:1: F401 'fedot.core.optimisers.adapters.PipelineAdapter' imported but unused

In the file test/unit/adapter/test_adapt_pipeline.py:

Line 4:1: F403 'from fedot.core.pipelines.verification_rules import *' used; unable to detect undefined names
Line 8:5: F405 'has_correct_operation_positions' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 9:5: F405 'has_primary_nodes' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 10:5: F405 'has_final_operation_as_model' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 11:5: F405 'has_final_operation_as_model' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 12:5: F405 'has_no_conflicts_with_data_flow' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 13:5: F405 'has_correct_data_connections' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules

In the file test/unit/adapter/test_adapt_registry.py:

Line 156:9: E126 continuation line over-indented for hanging indent
Line 182:9: E126 continuation line over-indented for hanging indent
Line 196:9: F841 local variable 'mutated_dom_graph' is assigned to but never used
Line 197:38: F841 local variable 'e' is assigned to but never used
Line 198:9: F841 local variable 'mutated_graph' is assigned to but never used

In the file test/unit/adapter/test_adapt_verification_rules.py:

Line 5:1: F403 'from fedot.core.pipelines.verification_rules import *' used; unable to detect undefined names
Line 9:5: F405 'has_correct_operation_positions' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 10:5: F405 'has_primary_nodes' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 11:5: F405 'has_no_conflicts_with_data_flow' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules
Line 12:5: F405 'has_correct_data_connections' may be undefined, or defined from star imports: fedot.core.pipelines.verification_rules

In the file test/unit/api/test_api_utils.py:

Line 2:1: F401 'pathlib' imported but unused
Line 6:1: F401 'pytest' imported but unused
Line 13:1: F401 'fedot.core.log.Log' imported but unused
Line 18:1: F401 'fedot.core.utils.default_fedot_data_dir' imported but unused

In the file test/unit/dag/test_graph_utils.py:

Line 1:1: F401 'typing.Tuple' imported but unused
Line 1:1: F401 'typing.Iterable' imported but unused

Comment last updated at 2022-09-13 14:29:43 UTC

gkirgizov · 2022-09-15T11:20:08Z

@pep8speaks pls update youself

codecov · 2022-09-19T10:06:17Z

Codecov Report

Merging #876 (66d1232) into master (55e6528) will decrease coverage by 0.02%.
The diff coverage is 96.05%.

@@            Coverage Diff             @@
##           master     #876      +/-   ##
==========================================
- Coverage   88.01%   87.98%   -0.03%     
==========================================
  Files         198      201       +3     
  Lines       13479    13528      +49     
==========================================
+ Hits        11863    11903      +40     
- Misses       1616     1625       +9

Impacted Files	Coverage Δ
...ore/optimisers/gp_comp/operators/regularization.py	`54.76% <0.00%> (-1.06%)`	⬇️
...timisers/gp_comp/pipeline_composer_requirements.py	`95.23% <ø> (ø)`
fedot/core/pipelines/pipeline.py	`96.20% <ø> (-0.05%)`	⬇️
fedot/core/optimisers/opt_history.py	`78.72% <80.00%> (+1.58%)`	⬆️
fedot/core/dag/graph.py	`78.31% <87.50%> (+1.09%)`	⬆️
...dot/core/optimisers/gp_comp/operators/crossover.py	`95.00% <92.30%> (+4.41%)`	⬆️
fedot/core/adapter/adapter.py	`94.28% <94.28%> (ø)`
fedot/core/adapter/adapt_registry.py	`94.59% <94.59%> (ø)`
fedot/api/api_utils/api_composer.py	`98.83% <100.00%> (+0.04%)`	⬆️
fedot/core/adapter/__init__.py	`100.00% <100.00%> (ø)`
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

nicl-nno

Предлагаю сразу на основе описания PR-а сделать заметку в документации, посвященную именно адаптерам. Можно со ссылками на докстринги (т.к. они довольно подробные), которые нужно почитать.

fedot/core/adapter/adapter.py

fedot/api/api_utils/api_composer.py

…en adapting population

nicl-nno · 2022-10-05T10:20:21Z

fedot/core/adapter/adapter.py

+        raise NotImplementedError()
+
+    @abstractmethod
+    def _restore(self, opt_graph: OptGraph, metadata: Optional[Dict[str, Any]] = None) -> DomainStructureType:


Может тогда этот метод назвать restore_graph или restore_single?

Аналогично с _adapt.

oops, не заметил твой коммент перед мерджем.
вообще, это непубличный метод, и его не предполагается вызывать извне. это скорее ядро реализации метода, для определения в потомках.

gkirgizov added composer Related to GP-composition algorithm architecture (re)design of existing or new framework subsystem refactoring labels Sep 13, 2022

gkirgizov changed the title ~~608 adapters~~ Redesign Adapter subsystem Sep 13, 2022

gkirgizov force-pushed the 608-adapters branch 3 times, most recently from 21a0c8c to 32d95ee Compare September 14, 2022 13:54

gkirgizov force-pushed the 608-adapters branch from 32d95ee to a63e414 Compare September 15, 2022 16:35

gkirgizov marked this pull request as ready for review September 19, 2022 09:52

gkirgizov force-pushed the 608-adapters branch 3 times, most recently from 9f5f286 to 25a1ad7 Compare September 20, 2022 15:06

nicl-nno requested changes Sep 20, 2022

View reviewed changes

fedot/core/adapter/adapter.py Outdated Show resolved Hide resolved

fedot/core/adapter/adapter.py Outdated Show resolved Hide resolved

fedot/api/api_utils/api_composer.py Show resolved Hide resolved

gkirgizov force-pushed the 608-adapters branch from 25a1ad7 to 8f5fe7b Compare September 21, 2022 13:51

MorrisNein reviewed Sep 27, 2022

View reviewed changes

fedot/api/api_utils/api_composer.py Show resolved Hide resolved

gkirgizov force-pushed the 608-adapters branch from a1e27f4 to 67980a7 Compare October 3, 2022 12:00

gkirgizov requested review from nicl-nno and MorrisNein October 4, 2022 10:52

gkirgizov added 9 commits October 4, 2022 13:52

WIP adapt registry

8d18637

WIP adapter

af5694a

WIP adapter

a4b5587

WIP simplest singleton runtime adapter based on isinstance

577c506

WIP add initialisation of adapt_registry.py

2803fc3

WIP adapt registry: add registration mechanics for restore

31be976

WIP adapt registry: add docs

8e8ee75

WIP drop usages of adapter in GraphVerifier

39cbaa0

WIP drop usages of adapter in verification rules

e194b25

gkirgizov added 21 commits October 4, 2022 13:52

WIP tmp

8644de5

Fix accidental cleanup of AdaptRegistry singleton class

1d03795

Drop duplicating adapt/maybe_adapt methods

9a66a83

Drop extra metadata argument from adapter.restore

28665e0

Fix test for tuning using logger

c2369c1

Make strict type equality in Adapter for adaptation

93f7567

Fix imports in one example

11e17d6

Add proper tests for adapter

0b82a2b

fix pep8 issues

090ffa0

Fix without_tuning test for sever

23a0370

Drop unneeded TODO

f41512e

refactor logging in api composer

68519c5

fixup! refactor logging in api composer

f3afe2a

Unify restore/restore_ind methods in adapter

5643577

Unify restore/restore_population methods in adapter

167c68a

Add was_optimised status flag to ApiComposer

eaddfb7

Fix return types of adapter: return OptGraph instead of Individual wh…

3ad9db1

…en adapting population

Fix test

a508cc2

fix status flags in api omposer

66e4bdf

fixes after rebase

b4e9f1b

fix rst dos build

b63e0f1

gkirgizov force-pushed the 608-adapters branch from ece6a1c to b63e0f1 Compare October 4, 2022 10:53

remove dupl line in indes.rst

033e464

MorrisNein approved these changes Oct 5, 2022

View reviewed changes

Extend dostrings for adapter lasses

66d1232

nicl-nno reviewed Oct 5, 2022

View reviewed changes

nicl-nno approved these changes Oct 5, 2022

View reviewed changes

gkirgizov merged commit e7e21c0 into master Oct 5, 2022

gkirgizov deleted the 608-adapters branch October 5, 2022 13:55

gkirgizov mentioned this pull request Oct 19, 2022

Redesign GraphAdapter functionality given new Graph interface #742

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redesign Adapter subsystem #876

Redesign Adapter subsystem #876

gkirgizov commented Sep 13, 2022 •

edited

Loading

pep8speaks commented Sep 13, 2022 •

edited

Loading

gkirgizov commented Sep 15, 2022

codecov bot commented Sep 19, 2022 •

edited

Loading

nicl-nno left a comment

nicl-nno Oct 5, 2022

nicl-nno Oct 5, 2022

gkirgizov Oct 6, 2022

Redesign Adapter subsystem #876

Redesign Adapter subsystem #876

Conversation

gkirgizov commented Sep 13, 2022 • edited Loading

pep8speaks commented Sep 13, 2022 • edited Loading

Comment last updated at 2022-09-13 14:29:43 UTC

gkirgizov commented Sep 15, 2022

codecov bot commented Sep 19, 2022 • edited Loading

Codecov Report

nicl-nno left a comment

Choose a reason for hiding this comment

nicl-nno Oct 5, 2022

Choose a reason for hiding this comment

nicl-nno Oct 5, 2022

Choose a reason for hiding this comment

gkirgizov Oct 6, 2022

Choose a reason for hiding this comment

gkirgizov commented Sep 13, 2022 •

edited

Loading

pep8speaks commented Sep 13, 2022 •

edited

Loading

codecov bot commented Sep 19, 2022 •

edited

Loading