Rm all legacy #1790

lukpueh · 2022-01-21T13:24:44Z

Fixes #1745

Removes legacy code, tests and docs
Updates remaining docs and some project metadata accordingly

Preliminary work

PEP 458: update references to python-tuf repo python/peps#2240
Update references to python-tuf repo theupdateframework.io#31
doc: remove spec "forwarder" docs #1773
doc: remove duplicate outdated docs #1769
doc: remove deprecated tuf/server client docs #1768
Remove obsolete fossa license scan config #1767
Updated references in CII Badge App
Checked specification for stale references to python-tuf (found none)

Note to reviewers:

Excluding deleted files from diff should make review easier
I'd appreciate a quick look through the remaining files in the repo, if I missed anything.
Check commit messages for details

Note of appreciation
Many thanks to everyone who helped python-tuf to evolve! ❤️

Most notably (based on prior Acknowledgements and git shortlog and in alphabetical order):

Sebastien Awwad, Justin Cappos, Geremy Condra, Vladimir Diaz, Zane Fisher, Pankhuri Goyal, Jussi Kukkonen, Trishank Kuppusamy, Joshua Lock, Marina Moore, Justin Samuel, Teodora Sechkova, Tian Tian, Santiago Torres-Arias, Martin Vrachev, Yuyu Zheng, Kairo de Araujo

coveralls · 2022-01-21T13:41:26Z

Pull Request Test Coverage Report for Build 1788517524

2 of 2 (100.0%) changed or added relevant lines in 1 file are covered.
9 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.4%) to 98.071%

Files with Coverage Reduction	New Missed Lines	%
tuf/ngclient/updater.py	3	97.96%
tuf/api/metadata.py	6	98.11%

Totals
Change from base Build 1745632739:	0.4%
Covered Lines:	1107
Relevant Lines:	1125

💛 - Coveralls

Define TESTS_DIR constant in tests/util.py as full path to the parent directory of the util module. This may be used to reliably read other files in tests dir, such es "repository_data" or "simple_server", regardless of cwd. This commit also replaces a couple of `getcwd() + "filename"` with `TESTS_DIR + filename`, so that in the future (post theupdateframework#1790) we should be able to invoke the tests from anywhere, not only from within the tests directory as is now the case. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

Update new test modules to stop using unittest_toolbox, in preparation for its removal in theupdateframework#1790. The tools provided by unittest_toolbox can easily (in a more obvious way) be replaced by using the standard library modules `tempfile` and `random` (no more used) directly. In the case of tempdir and -file creation/removal, skipping the use of unittest_toolbox, which does this by default, also uncovers some test cleanup failures, which would occur when temporary test directories were removed while a test server hadn't released them. (see `except OSError: pass` in unittest_toolbox's `tearDown` method) **Change details** **test_fetcher_ng.py:** - Stop implicitly creating (setUp) and removing (tearDown) tmp test dirs. -Move now manual creation of an exemplary targets file to setUpClass, as the same file is used by all tests. And remove it explicitly in tearDownClass after killing the server (see note about failure above). - Trigger URL parsing error with a hardcoded invalid URL string instead of a random string. **test_updater_ng.py** - Stop implicitly creating (setUp) and removing (tearDown) tmp test dirs. - Explicitly create tmp test dirs in setUp, but don't remove them in tearDown to avoid above mentioned failures. They will be removed all at once when removing the tmp root test dir in tearDownClass Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

lukpueh · 2022-01-24T13:13:24Z

Just moved out test updates for unittest_toolbox removal in this PR to #1792 to make reviewing easier. That PR also fixes the failing windows run seen in this PR.

TODO: polish commit message ----- - Remove outdated ATTACKS.md requires basic_client.py which was renamed to client.py in 2018 and is removed in theupdateframework#1790 (host of problems with client.py described in theupdateframework#881) --> reference ticket TODO: create ticket - QUICKSTART.md also based on cli tools (see above) --> reference ticket TODO: create ticket Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

Legacy tools will be removed in subsequent commits. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

Remove documentation for legacy client, repository/developer tool and command line tools, which will be removed in subsequent commits. See theupdateframework#1797 and theupdateframework#1798 for replacing ATTACKS.md and QUICKSTART.md. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

Remove metadata generation scripts based on legacy repository/developer tools, which will be removed in subsequent commits. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

Old tests are not touched as they will be removed in subsequent commits, along with the custom log module. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

Remove tests for legacy client, repository/developer tool and command line tools, which will be removed in subsequent commits. This commits also removes obsolete test tooling: - Regarding simple_https_server + test certificates -- http/https is no longer handled by tuf client directly but transparently by the underlying requests module used by the default fetcher implementation. - For details about unittest_toolbox see theupdateframework#1792 Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

Remove legacy client, repository/developer tool, command line tools, and underlying libraries and utilities. See docs/1.0.0-ANNOUNCEMENT.md for details about their replacement, deprecation strategy and migration instructions. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

- Update linter config to no longer distinguish between legacy and new implementation. This requires addressing a linter warning in an until now not linted module (tuf/__init__.py). - Remove obsolete rules in MANIFEST.in (source distribution) and tests/.coveragerc (test coverage). Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

jku

Looks great 😁 I might have to take another look though, it's hard to cover everything in one go.

Some observations:

files in examples/repo_example/ reference repository_tool and repository_lib: they could be updated to indicate that those have now been deprecated
ATTACKS.md would indeed be nice to have but that's clearly a project of its own and probably won't look much like the old one so I'm fine with leaving that for now

lukpueh · 2022-01-27T14:02:01Z

files in examples/repo_example/ reference repository_tool and repository_lib: they could be updated to indicate that those have now been deprecated

Thanks for catching! This was on my mind at some point.. but then it wasn't. :D Will push another commit here.

jku

looks good to me. The only additional potential removal I found was the outdated AUTHORS.txt but that does not need to be in this PR.

EDIT: and the test data generation issue I just filed #1806

jku · 2022-01-28T12:01:12Z

$ tox -e py
...
Ran 128 tests in 2.058s

this is so good.

(if anyone is wondering why there are so many fewer tests: we're using a lot more subtests and those are not counted individually: my Stetson-Harrison assessment is that the new code on average has better coverage than the old one did, although specific areas may be worse).

MVrachev

I was waiting for this pr for a long time!
Great stuff here.

I totally forgot that the old updater is 3070 lines, repository_lib 2306 lines, and repository_tool 3291... wow

Also, thanks for the last commitment! I was planning on doing that.

Only one question: do we have a test for key revocation?

MVrachev · 2022-01-28T12:14:47Z

tox.ini

@@ -38,17 +38,12 @@ commands =

 [testenv:lint]
 changedir = {toxinidir}
-lint_dirs = tuf/api tuf/ngclient examples tests
+lint_dirs = tuf examples tests


Did you check if all of the tools recursively check the next folders in tuf/ - tuf/ngclient and tuf/api?

jku · 2022-01-28T12:38:49Z

Only one question: do we have a test for key revocation?

In the sense that the old tests used the word "revocation", yes we do.

test_updater_key_rotations.py contains loads of different cases of key rotations: some remove ("revoke") old keys, some don't. test_updater_top_level_update.py has tests for fast forward recovery which include complete key rotations (these are made easier to read in #1766).

jku · 2022-02-02T09:25:24Z

I should have stated this explicitly:

I think the PR is good and we should deal with any fallout with new issues after merging
I did not merge myself as I wanted other maintainers to have time to review as it's a major change

Rephrase deprecation info in repo_example modules doc headers to reflect that the deprecation has happened. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

lukpueh · 2022-02-03T09:17:33Z

Thanks for your reviews, @jku and @MVrachev!

files in examples/repo_example/ reference repository_tool and repository_lib: they could be updated to indicate that those have now been deprecated

Fixed in 9816c40.

Did you check if all of the tools recursively check the next folders in tuf/ - tuf/ngclient and tuf/api?

Yes, I did. And yes, they do.

Only one question: do we have a test for key revocation?

I admit that I didn't question test completeness. I just rm tests/test_*_old.py (relying on 22fe1e6) and cross-checked with effective test coverage, which didn't decrease.

lukpueh · 2022-02-03T09:21:03Z

I should have stated this explicitly:

I think the PR is good and we should deal with any fallout with new issues after merging

I did not merge myself as I wanted other maintainers to have time to review as it's a major change

@-'other (active) maintainers', i.e. @mnm678, @joshuagl, mind wrapping this up? 🚀 😃

mnm678

Thanks to all the people who worked on the new client! This is really exciting, and looks good to me.

lukpueh · 2022-02-04T13:01:50Z

Thanks for the reviews, everyone! Merging...

- metadata.staged: related to a removed tutorial and outdated deployment recommendation - project: related to the removed developer_tool (theupdateframework#1790) - map.json: related to TAP4, which is not supported by python-tuf Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

lukpueh force-pushed the rm-all branch from 1705448 to 7194f7c Compare January 21, 2022 13:38

lukpueh mentioned this pull request Jan 24, 2022

test: stop using unittest_toolbox in new tests #1792

Merged

3 tasks

lukpueh force-pushed the rm-all branch from 7194f7c to 3f54e70 Compare January 24, 2022 13:42

lukpueh force-pushed the rm-all branch from 3f54e70 to 9a968c1 Compare January 25, 2022 17:05

This was referenced Jan 26, 2022

doc: consider re-adding a "quickstart" document #1797

Open

doc: consider re-adding python-tuf attack protection demo #1798

Open

lukpueh added 7 commits January 26, 2022 14:43

doc: don't mention legacy tools in main README

8c72588

Legacy tools will be removed in subsequent commits. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

test: drop legacy test metadata generation scripts

1619290

Remove metadata generation scripts based on legacy repository/developer tools, which will be removed in subsequent commits. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

test: drop usage of custom log module in new test

2bb977c

Old tests are not touched as they will be removed in subsequent commits, along with the custom log module. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

lukpueh force-pushed the rm-all branch from 9a968c1 to ad6d2ca Compare January 26, 2022 14:25

lukpueh marked this pull request as ready for review January 27, 2022 09:04

lukpueh changed the title ~~Rm all legacy (WIP)~~ Rm all legacy Jan 27, 2022

jku reviewed Jan 27, 2022

View reviewed changes

jku approved these changes Jan 28, 2022

View reviewed changes

jku mentioned this pull request Jan 28, 2022

tests: static test data generation #1806

Closed

MVrachev approved these changes Jan 28, 2022

View reviewed changes

doc: update deprecation info in repo_example docs

9816c40

Rephrase deprecation info in repo_example modules doc headers to reflect that the deprecation has happened. Signed-off-by: Lukas Puehringer <lukas.puehringer@nyu.edu>

mnm678 approved these changes Feb 3, 2022

View reviewed changes

lukpueh merged commit 31fd8d4 into theupdateframework:develop Feb 4, 2022

lukpueh mentioned this pull request Feb 28, 2022

Add logging config and guidelines #1132

Closed

lukpueh mentioned this pull request Mar 16, 2022

tests: test infrastructure (setup/teardown) improvements #1348

Open

lukpueh mentioned this pull request Oct 10, 2023

tests: remove obsolete test metadata #2489

Closed

lukpueh mentioned this pull request Oct 11, 2023

tests: remove unused and obsolete test metadata #2491

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rm all legacy #1790

Rm all legacy #1790

lukpueh commented Jan 21, 2022 •

edited

Loading

coveralls commented Jan 21, 2022 •

edited

Loading

lukpueh commented Jan 24, 2022

jku left a comment

lukpueh commented Jan 27, 2022

jku left a comment •

edited

Loading

jku commented Jan 28, 2022

MVrachev left a comment

MVrachev Jan 28, 2022

jku commented Jan 28, 2022 •

edited

Loading

jku commented Feb 2, 2022

lukpueh commented Feb 3, 2022

lukpueh commented Feb 3, 2022

mnm678 left a comment

lukpueh commented Feb 4, 2022

Rm all legacy #1790

Rm all legacy #1790

Conversation

lukpueh commented Jan 21, 2022 • edited Loading

coveralls commented Jan 21, 2022 • edited Loading

Pull Request Test Coverage Report for Build 1788517524

💛 - Coveralls

lukpueh commented Jan 24, 2022

jku left a comment

Choose a reason for hiding this comment

lukpueh commented Jan 27, 2022

jku left a comment • edited Loading

Choose a reason for hiding this comment

jku commented Jan 28, 2022

MVrachev left a comment

Choose a reason for hiding this comment

MVrachev Jan 28, 2022

Choose a reason for hiding this comment

jku commented Jan 28, 2022 • edited Loading

jku commented Feb 2, 2022

lukpueh commented Feb 3, 2022

lukpueh commented Feb 3, 2022

mnm678 left a comment

Choose a reason for hiding this comment

lukpueh commented Feb 4, 2022

lukpueh commented Jan 21, 2022 •

edited

Loading

coveralls commented Jan 21, 2022 •

edited

Loading

jku left a comment •

edited

Loading

jku commented Jan 28, 2022 •

edited

Loading