diff --git a/docs/CLI.md b/docs/CLI.md deleted file mode 100644 index c07b73be7c..0000000000 --- a/docs/CLI.md +++ /dev/null @@ -1,447 +0,0 @@ -# Command-Line Interface # - -The TUF command-line interface (CLI) requires a full -[TUF installation](INSTALLATION.rst). Be sure to include the installation of -extra dependencies and C extensions ( -```python3 -m pip install securesystemslib[crypto,pynacl]```). - -The use of the CLI is documented with examples below. - ----- -# Basic Examples # - -## Create a repository ## - -Create a TUF repository in the current working directory. A cryptographic key -is created and set for each top-level role. The written Targets metadata does -not sign for any targets, nor does it delegate trust to any roles. The -`--init` call will also set up a client directory. By default, these -directories will be `./tufrepo` and `./tufclient`. - -```Bash -$ repo.py --init -``` - -Optionally, the repository can be written to a specified location. -```Bash -$ repo.py --init --path -``` - -The default top-level key files created with `--init` are saved to disk -encrypted, with a default password of 'pw'. Instead of using the default -password, the user can enter one on the command line for each top-level role. -These optional command-line options also work with other CLI actions (e.g., -repo.py --add). -```Bash -$ repo.py --init [--targets_pw, --root_pw, --snapshot_pw, --timestamp_pw] -``` - - - -Create a bare TUF repository in the current working directory. A cryptographic -key is *not* created nor set for each top-level role. -```Bash -$ repo.py --init --bare -``` - - - -Create a TUF repository with [consistent -snapshots](https://github.com/theupdateframework/specification/blob/master/tuf-spec.md#7-consistent-snapshots) -enabled, where target filenames have their hash prepended (e.g., -`.README.txt`), and metadata filenames have their version numbers -prepended (e.g., `.snapshot.json`). -```Bash -$ repo.py --init --consistent -``` - - - -## Add a target file ## - -Copy a target file to the repo and add it to the Targets metadata (or the -Targets role specified in --role). More than one target file, or directory, -may be specified in --add. The --recursive option may be toggled to also -include files in subdirectories of a specified directory. The Snapshot -and Timestamp metadata are also updated and signed automatically, but this -behavior can be toggled off with --no_release. -```Bash -$ repo.py --add -$ repo.py --add [--recursive] -``` - -Similar to the --init case, the repository location can be chosen. -```Bash -$ repo.py --add --path -``` - - - -## Remove a target file ## - -Remove a target file from the Targets metadata (or the Targets role specified -in --role). More than one target file or glob pattern may be specified in ---remove. The Snapshot and Timestamp metadata are also updated and signed -automatically, but this behavior can be toggled off with --no_release. - -```Bash -$ repo.py --remove ... -``` - -Examples: - -Remove all target files, that match `foo*.tgz,` from the Targets metadata. -```Bash -$ repo.py --remove "foo*.tgz" -``` - -Remove all target files from the `my_role` metadata. -```Bash -$ repo.py --remove "*" --role my_role --sign tufkeystore/my_role_key -``` - - -## Generate key ## -Generate a cryptographic key. The generated key can later be used to sign -specific metadata with `--sign`. The supported key types are: `ecdsa`, -`ed25519`, and `rsa`. If a keytype is not given, an Ed25519 key is generated. - -If adding a top-level key to a bare repo (i.e., repo.py --init --bare), -the filenames of the top-level keys must be "root_key," "targets_key," -"snapshot_key," "timestamp_key." The filename can vary for any additional -top-level key. -```Bash -$ repo.py --key -$ repo.py --key -$ repo.py --key [--path --pw [my_password], - --filename ] -``` - -Instead of using a default password, the user can enter one on the command -line or be prompted for it via password masking. -```Bash -$ repo.py --key ecdsa --pw my_password -``` - -```Bash -$ repo.py --key rsa --pw -Enter a password for the RSA key (...): -Confirm: -``` - - - -## Sign metadata ## -Sign, with the specified key(s), the metadata of the role indicated in --role. -The Snapshot and Timestamp role are also automatically signed, if possible, but -this behavior can be disabled with --no_release. -```Bash -$ repo.py --sign ... [--role , --path ] -``` - -For example, to sign the delegated `foo` metadata: -```Bash -$ repo.py --sign --role foo -``` - - - -## Trust keys ## - -The Root role specifies the trusted keys of the top-level roles, including -itself. The --trust command-line option, in conjunction with --pubkeys and ---role, can be used to indicate the trusted keys of a role. - -```Bash -$ repo.py --trust --pubkeys --role -``` - -For example: -```Bash -$ repo.py --init --bare -$ repo.py --trust --pubkeys tufkeystore/my_key.pub tufkeystore/my_key_too.pub - --role root -``` - - - -### Distrust keys ### - -Conversely, the Root role can discontinue trust of specified key(s). - -Example of how to discontinue trust of a key: -```Bash -$ repo.py --distrust --pubkeys tufkeystore/my_key_too.pub --role root -``` - - - -## Delegations ## - -Delegate trust of target files from the Targets role (or the one specified in ---role) to some other role (--delegatee). --delegatee is trusted to sign for -target files that match the delegated glob pattern(s). The --delegate option -does not create metadata for the delegated role, rather it updates the -delegator's metadata to list the delegation to --delegatee. The Snapshot and -Timestamp metadata are also updated and signed automatically, but this behavior -can be toggled off with --no_release. - -```Bash -$ repo.py --delegate ... --delegatee --pubkeys - ... [--role --terminating --threshold ---sign ] -``` - -For example, to delegate trust of `foo*.gz` packages to the `foo` role: - -``` -$ repo.py --delegate "foo*.tgz" --delegatee foo --pubkeys tufkeystore/foo.pub -``` - - - -## Revocations ## - -Revoke trust of target files from a delegated role (--delegatee). The -"targets" role performs the revocation if --role is not specified. The ---revoke option does not delete the metadata belonging to --delegatee, instead -it removes the delegation to it from the delegator's (or --role) metadata. The -Snapshot and Timestamp metadata are also updated and signed automatically, but -this behavior can be toggled off with --no_release. - - -```Bash -$ repo.py --revoke --delegatee [--role ---sign ] -``` - - - -## Verbosity ## - -Set the verbosity of the logger (2, by default). The lower the number, the -greater the verbosity. Logger messages are saved to `tuf.log` in the current -working directory. -```Bash -$ repo.py --verbose <0-5> -``` - - - -## Clean ## - -Delete the repo in the current working directory, or the one specified with -`--path`. Specifically, the `tufrepo`, `tufclient`, and `tufkeystore` -directories are deleted. - -```Bash -$ repo.py --clean -$ repo.py --clean --path -``` ----- - - - - - - - - -# Further Examples # - -## Basic Update Delivery ## - -Steps: - -(1) initialize a repo. - -(2) delegate trust of target files to another role. - -(3) add a trusted file to the delegated role. - -(4) fetch the trusted file from the delegated role. - -```Bash -Step (1) -$ repo.py --init - -Step (2) -$ repo.py --key ed25519 --filename mykey -$ repo.py --delegate "README.*" --delegatee myrole --pubkeys tufkeystore/mykey.pub -$ repo.py --sign tufkeystore/mykey --role myrole -Enter a password for the encrypted key (tufkeystore/mykey): -$ echo "my readme text" > README.txt - -Step (3) -$ repo.py --add README.txt --role myrole --sign tufkeystore/mykey -Enter a password for the encrypted key (tufkeystore/mykey): -``` - -Serve the repo -```Bash -$ python3 -m http.server 8001 -``` - -```Bash -Step (4) -$ client.py --repo http://localhost:8001 README.txt -$ tree . -. -├── tuf.log -├── tufrepo -│   └── metadata -│   ├── current -│   │   ├── 1.root.json -│   │   ├── myrole.json -│   │   ├── root.json -│   │   ├── snapshot.json -│   │   ├── targets.json -│   │   └── timestamp.json -│   └── previous -│   ├── 1.root.json -│   ├── root.json -│   ├── snapshot.json -│   ├── targets.json -│   └── timestamp.json -└── tuftargets - └── README.txt - - 5 directories, 13 files -``` - - -## Correcting a Key ## -The filename of the top-level keys must be "root_key," "targets_key," -"snapshot_key," and "root_key." The filename can vary for any additional -top-level key. - -Steps: - -(1) initialize a repo containing default keys for the top-level roles. -(2) distrust the default key for the root role. -(3) create a new key and trust its use with the root role. -(4) sign the root metadata file. - -```Bash -Step (1) -$ repo.py --init - -Step (2) -$ repo.py --distrust --pubkeys tufkeystore/root_key.pub --role root - -Step (3) -$ repo.py --key ed25519 --filename root_key -$ repo.py --trust --pubkeys tufkeystore/root_key.pub --role root - -Step (4) -$ repo.py --sign tufkeystore/root_key --role root -Enter a password for the encrypted key (tufkeystore/root_key): -``` - - -## More Update Delivery ## - -Steps: - -(1) create a bare repo. - -(2) add keys to the top-level roles. - -(3) delegate trust of particular target files to another role X, where role X -has a signature threshold 2 and is marked as a terminating delegation. The -keys for role X and Y should be created prior to performing the delegation. - -(4) Delegate from role X to role Y. - -(5) have role X sign for a file also signed by the Targets role, to demonstrate -the expected file that should be downloaded by the client. - -(6) perform an update. - -(7) halt the server, add README.txt to the Targets role, restart the server, -and fetch the Target's role README.txt. - -(8) Add LICENSE to 'role_y' and demonstrate that the client must not fetch it -because 'role_x' is a terminating delegation (and hasn't signed for it). - -```Bash -Steps (1) and (2) -$ repo.py --init --consistent --bare -$ repo.py --key ed25519 --filename root_key -$ repo.py --trust --pubkeys tufkeystore/root_key.pub --role root -$ repo.py --key ecdsa --filename targets_key -$ repo.py --trust --pubkeys tufkeystore/targets_key.pub --role targets -$ repo.py --key rsa --filename snapshot_key -$ repo.py --trust --pubkeys tufkeystore/snapshot_key.pub --role snapshot -$ repo.py --key ecdsa --filename timestamp_key -$ repo.py --trust --pubkeys tufkeystore/timestamp_key.pub --role timestamp -$ repo.py --sign tufkeystore/root_key --role root -Enter a password for the encrypted key (tufkeystore/root_key): -$ repo.py --sign tufkeystore/targets_key --role targets -Enter a password for the encrypted key (tufkeystore/targets_key): -``` - -```Bash -Steps (3) and (4) -$ repo.py --key ed25519 --filename key_x -$ repo.py --key ed25519 --filename key_x2 - -$ repo.py --delegate "README.*" "LICENSE" --delegatee role_x --pubkeys - tufkeystore/key_x.pub tufkeystore/key_x2.pub --threshold 2 --terminating -$ repo.py --sign tufkeystore/key_x tufkeystore/key_x2 --role role_x - -$ repo.py --key ed25519 --filename key_y - -$ repo.py --delegate "README.*" "LICENSE" --delegatee role_y --role role_x - --pubkeys tufkeystore/key_y.pub --sign tufkeystore/key_x tufkeystore/key_x2 - -$ repo.py --sign tufkeystore/key_y --role role_y -``` - -```Bash -Steps (5) and (6) -$ echo "role_x's readme" > README.txt -$ repo.py --add README.txt --role role_x --sign tufkeystore/key_x tufkeystore/key_x2 -``` - -Serve the repo -```Bash -$ python3 -m http.server 8001 -``` - -Fetch the role x's README.txt -```Bash -$ client.py --repo http://localhost:8001 README.txt -$ cat tuftargets/README.txt -role_x's readme -``` - - -```Bash -Step (7) -halt the server... - -$ echo "Target role's readme" > README.txt -$ repo.py --add README.txt - -restart the server... -``` - -```Bash -$ rm -rf tuftargets/ tuf.log -$ client.py --repo http://localhost:8001 README.txt -$ cat tuftargets/README.txt -Target role's readme -``` - -```Bash -Step (8) -$ echo "role_y's license" > LICENSE -$ repo.py --add LICENSE --role role_y --sign tufkeystore/key_y -``` - -```Bash -$ rm -rf tuftargets/ tuf.log -$ client.py --repo http://localhost:8001 LICENSE -Error: 'LICENSE' not found. -``` diff --git a/docs/GETTING_STARTED.rst b/docs/GETTING_STARTED.rst deleted file mode 100644 index f958373975..0000000000 --- a/docs/GETTING_STARTED.rst +++ /dev/null @@ -1,10 +0,0 @@ -Getting Started ---------------- - -- `Overview of TUF `_ -- `Installation `_ -- Beginner Tutorials (using the basic command-line interface): - - `Quickstart `_ - - `CLI Documentation and Examples `_ -- `Advanced Tutorial `_ -- `Guidelines for Contributors `_ diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md deleted file mode 100644 index 6d35fb1d7d..0000000000 --- a/docs/QUICKSTART.md +++ /dev/null @@ -1,149 +0,0 @@ -# Quickstart # - -In this quickstart tutorial, we'll use the basic TUF command-line interface -(CLI), which includes the `repo.py` script and the `client.py` script, to set -up a repository with an update and metadata about that update, then download -and verify that update as a client. - -Unlike the underlying TUF modules that the CLI uses, the CLI itself is a bit -bare-bones. Using the CLI is the easiest way to familiarize yourself with -how TUF works, however. It will serve as a very basic update system. - ----- - -**Step (0)** - Make sure TUF is installed. - -Make sure that TUF is installed, along with some of the optional cryptographic -libraries and C extensions. Try this command to do that: -`python3 -m pip install securesystemslib[colors,crypto,pynacl] tuf` - -If you run into errors during that pip command, please consult the more -detailed [TUF Installation Instructions](INSTALLATION.rst). (There are some -system libraries that you may need to install first.) - - -**Step (1)** - Create a basic repository and client. - -The following command will set up a basic update repository and basic client -that knows about the repository. `tufrepo`, `tufkeystore`, and -`tufclient` directories will be created in the current directory. - -```Bash -$ repo.py --init -``` - -Four sets of keys are created in the `tufkeystore` directory. Initial metadata -about the repository is created in the `tufrepo` directory, and also provided -to the client in the `tufclient` directory. - - -**Step (2)** - Add an update to the repository. - -We'll create a target file that will later be delivered as an update to clients. -Metadata about that file will be created and signed, and added to the -repository's metadata. - -```Bash -$ echo 'Test file' > testfile -$ repo.py --add testfile -$ tree tufrepo/ -tufrepo/ -├── metadata -│   ├── 1.root.json -│   ├── root.json -│   ├── snapshot.json -│   ├── targets.json -│   └── timestamp.json -├── metadata.staged -│   ├── 1.root.json -│   ├── root.json -│   ├── snapshot.json -│   ├── targets.json -│   └── timestamp.json -└── targets - └── testfile - - 3 directories, 11 files -``` - -The new file `testfile` is added to the repository, and metadata is updated in -the `tufrepo` directory. The Targets metadata (`targets.json`) now includes -the file size and hashes of the `testfile` target file, and this metadata is -signed by the Targets role's key, so that clients can verify that metadata -about `testfile` and then verify `testfile` itself. - - -**Step (3)** - Serve the repo. - -We'll host a toy http server containing the `testfile` update and the -repository's metadata. - -```Bash -$ cd "tufrepo/" -$ python3 -m http.server 8001 -``` - -**Step (4)** - Obtain and verify the `testfile` update on a client. - -The client can request the package `testfile` from the repository. TUF will -download and verify metadata from the repository as necessary to determine -what the trustworthy hashes and length of `testfile` are, then download -the target `testfile` from the repository and keep it only if it matches that -trustworthy metadata. - -```Bash -$ cd "../tufclient/" -$ client.py --repo http://localhost:8001 testfile -$ tree -. -├── tufrepo -│   └── metadata -│   ├── current -│   │   ├── 1.root.json -│   │   ├── root.json -│   │   ├── snapshot.json -│   │   ├── targets.json -│   │   └── timestamp.json -│   └── previous -│   ├── 1.root.json -│   ├── root.json -│   ├── snapshot.json -│   ├── targets.json -│   └── timestamp.json -└── tuftargets - └── testfile - - 5 directories, 11 files -``` - -Now that a trustworthy update target has been obtained, an updater can proceed -however it normally would to install or use the update. - ----- - -### Next Steps - -TUF provides functionality for both ends of a software update system, the -**update provider** and the **update client**. - -`repo.py` made use of `tuf.repository_tool`'s functionality for an update -provider, helping you produce and sign metadata about your updates. - -`client.py` made use of `tuf.client.updater`'s client-side functionality, -performing download and the critical verification steps for metadata and the -update itself. - -You can look at [CLI.md](CLI.md) to toy with the TUF CLI a bit more. -After that, try out using the underlying modules for a great deal more control. -The more detailed [Advanced Tutorial](TUTORIAL.md) shows you how to use the -underlying modules, `repository_tool` and `updater`. - -Ultimately, a sophisticated update client will use or re-implement those -underlying modules. The TUF design is intended to play well with any update -workflow. - -Please provide feedback or questions for this or other tutorials, or -TUF in general, by checking out -[our contact info](https://github.com/theupdateframework/python-tuf#contact), or -creating [issues](https://github.com/theupdateframework/python-tuf/issues) in this -repository! diff --git a/docs/TUTORIAL.md b/docs/TUTORIAL.md deleted file mode 100644 index d8659e7213..0000000000 --- a/docs/TUTORIAL.md +++ /dev/null @@ -1,696 +0,0 @@ -# Advanced Tutorial # - -## Table of Contents ## -- [How to Create and Modify a TUF Repository](#how-to-create-and-modify-a-tuf-repository) - - [Overview](#overview) - - [Keys](#keys) - - [Create RSA Keys](#create-rsa-keys) - - [Import RSA Keys](#import-rsa-keys) - - [Create and Import Ed25519 Keys](#create-and-import-ed25519-keys) - - [Create Top-level Metadata](#create-top-level-metadata) - - [Create Root](#create-root) - - [Create Timestamp, Snapshot, Targets](#create-timestamp-snapshot-targets) - - [Targets](#targets) - - [Add Target Files](#add-target-files) - - [Remove Target Files](#remove-target-files) - - [Delegations](#delegations) - - [Revoke Delegated Role](#revoke-delegated-role) - - [Wrap-up](#wrap-up) -- [Delegate to Hashed Bins](#delegate-to-hashed-bins) -- [Consistent Snapshots](#consistent-snapshots) -- [How to Perform an Update](#how-to-perform-an-update) - -## How to Create and Modify a TUF Repository ## - -### Overview ### -A software update system must follow two steps to integrate The Update -Framework (TUF). First, it must add the framework to the client side of the -update system. The [tuf.client.updater](../tuf/client/README.md) module assists in -integrating TUF on the client side. Second, the software repository on the -server side must be modified to include a minimum of four top-level metadata -(root.json, targets.json, snapshot.json, and timestamp.json). No additional -software is required to convert a software repository to a TUF one. The -low-level repository tool that generates the required TUF metadata for a -software repository is the focus of this tutorial. There is also separate -document that [demonstrates how TUF protects against malicious -updates](../tuf/ATTACKS.md). - -The [repository tool](../tuf/repository_tool.py) contains functions to generate -all of the files needed to populate and manage a TUF repository. The tool may -either be imported into a Python module, or used with the Python interpreter in -interactive mode. - -A repository object that encapsulates the metadata files of the repository can -be created or loaded by the repository tool. Repository maintainers can modify -the repository object to manipulate the metadata files stored on the -repository. TUF clients use the metadata files to validate files requested and -downloaded. In addition to the repository object, where the majority of -changes are made, the repository tool provides functions to generate and -persist cryptographic keys. The framework utilizes cryptographic keys to sign -and verify metadata files. - -To begin, cryptographic keys are generated with the repository tool. However, -before metadata files can be validated by clients and target files fetched in a -secure manner, public keys must be pinned to particular metadata roles and -metadata signed by role's private keys. After covering keys, the four required -top-level metadata are created next. Examples are given demonstrating the -expected work flow, where the metadata roles are created in a specific order, -keys imported and loaded, and metadata signed and written to disk. Lastly, -target files are added to the repository, and a custom delegation performed to -extend the default roles of the repository. By the end, a fully populated TUF -repository is generated that can be used by clients to securely download -updates. - -### Keys ### -The repository tool supports multiple public-key algorithms, such as -[RSA](https://en.wikipedia.org/wiki/RSA_%28cryptosystem%29) and -[Ed25519](https://ed25519.cr.yp.to/), and multiple cryptography libraries. - -Using [RSA-PSS](https://tools.ietf.org/html/rfc8017#section-8.1) or -[ECDSA](https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm) -signatures requires the [cryptography](https://cryptography.io/) library. If -generation of Ed25519 signatures is needed -[PyNaCl](https://github.com/pyca/pynacl) library should be installed. This -tutorial assumes both dependencies are installed: refer to -[Installation Instructions](INSTALLATION.rst#install-with-more-cryptographic-flexibility) -for details. - -The Ed25519 and ECDSA keys are stored in JSON format and RSA keys are stored in PEM -format. Private keys are encrypted and passphrase-protected (strengthened with -PBKDF2-HMAC-SHA256.) Generating, importing, and loading cryptographic key -files can be done with functions available in the repository tool. - -To start, a public and private RSA key pair is generated with the -`generate_and_write_rsa_keypair()` function. The keys generated next are -needed to sign the repository metadata files created in upcoming sub-sections. - -Note: In the instructions below, lines that start with `>>>` denote commands -that should be entered by the reader, `#` begins the start of a comment, and -text without prepended symbols is the output of a command. - -#### Create RSA Keys #### -```python ->>> from tuf.repository_tool import * - -# Generate and write the first of two root keys for the TUF repository. The -# following function creates an RSA key pair, where the private key is saved to -# "root_key" and the public key to "root_key.pub" (both saved to the current -# working directory). ->>> generate_and_write_rsa_keypair(password="password", filepath="root_key", bits=2048) - -# If the key length is unspecified, it defaults to 3072 bits. A length of less -# than 2048 bits raises an exception. A similar function is available to supply -# a password on the prompt. If an empty password is entered, the private key -# is saved unencrypted. ->>> generate_and_write_rsa_keypair_with_prompt(filepath="root_key2") -enter password to encrypt private key file '/path/to/root_key2' -(leave empty if key should not be encrypted): -Confirm: -``` -The following four key files should now exist: - -1. **root_key** -2. **root_key.pub** -3. **root_key2** -4. **root_key2.pub** - -If a filepath is not given, the KEYID of the generated key is used as the -filename. The key files are written to the current working directory. -```python -# Continuing from the previous section . . . ->>> generate_and_write_rsa_keypair_with_prompt() -enter password to encrypt private key file '/path/to/KEYID' -(leave empty if key should not be encrypted): -Confirm: -``` - -### Import RSA Keys ### -```python -# Continuing from the previous section . . . - -# Import an existing public key. ->>> public_root_key = import_rsa_publickey_from_file("root_key.pub") - -# Import an existing private key. Importing a private key requires a password, -# whereas importing a public key does not. ->>> private_root_key = import_rsa_privatekey_from_file("root_key") -enter password to decrypt private key file '/path/to/root_key' -(leave empty if key not encrypted): -``` - -### Create and Import Ed25519 Keys ### -```Python -# Continuing from the previous section . . . - -# The same generation and import functions as for rsa keys exist for ed25519 ->>> generate_and_write_ed25519_keypair_with_prompt(filepath='ed25519_key') -enter password to encrypt private key file '/path/to/ed25519_key' -(leave empty if key should not be encrypted): -Confirm: - -# Import the ed25519 public key just created . . . ->>> public_ed25519_key = import_ed25519_publickey_from_file('ed25519_key.pub') - -# and its corresponding private key. ->>> private_ed25519_key = import_ed25519_privatekey_from_file('ed25519_key') -enter password to decrypt private key file '/path/to/ed25519_key' -(leave empty if key should not be encrypted): -``` - -Note: Methods are also available to generate and write keys from memory. -* generate_ed25519_key() -* generate_ecdsa_key() -* generate_rsa_key() - -* import_ecdsakey_from_pem(pem) -* import_rsakey_from_pem(pem) - -### Create Top-level Metadata ### -The [metadata document](METADATA.md) outlines the JSON files that must exist -on a TUF repository. The following sub-sections demonstrate the -`repository_tool.py` calls repository maintainers may issue to generate the -required roles. The top-level roles to be created are `root`, `timestamp`, -`snapshot`, and `target`. - -We begin with `root`, the locus of trust that specifies the public keys of the -top-level roles, including itself. - - -#### Create Root #### -```python -# Continuing from the previous section . . . - -# Create a new Repository object that holds the file path to the TUF repository -# and the four top-level role objects (Root, Targets, Snapshot, Timestamp). -# Metadata files are created when repository.writeall() or repository.write() -# are called. The repository directory is created if it does not exist. You -# may see log messages indicating any directories created. ->>> repository = create_new_repository("repository") - -# The Repository instance, 'repository', initially contains top-level Metadata -# objects. Add one of the public keys, created in the previous section, to the -# root role. Metadata is considered valid if it is signed by the public key's -# corresponding private key. ->>> repository.root.add_verification_key(public_root_key) - -# A role's verification key(s) (to be more precise, the verification key's -# keyid) may be queried. Other attributes include: signing_keys, version, -# signatures, expiration, threshold, and delegations (attribute available only -# to a Targets role). ->>> repository.root.keys -['b23514431a53676595922e955c2d547293da4a7917e3ca243a175e72bbf718df'] - -# Add a second public key to the root role. Although previously generated and -# saved to a file, the second public key must be imported before it can added -# to a role. ->>> public_root_key2 = import_rsa_publickey_from_file("root_key2.pub") ->>> repository.root.add_verification_key(public_root_key2) - -# The threshold of each role defaults to 1. Maintainers may change the -# threshold value, but repository_tool.py validates thresholds and warns users. -# Set the threshold of the root role to 2, which means the root metadata file -# is considered valid if it's signed by at least two valid keys. We also load -# the second private key, which hasn't been imported yet. ->>> repository.root.threshold = 2 ->>> private_root_key2 = import_rsa_privatekey_from_file("root_key2", password="password") - -# Load the root signing keys to the repository, which writeall() or write() -# (write multiple roles, or a single role, to disk) use to sign the root -# metadata. ->>> repository.root.load_signing_key(private_root_key) ->>> repository.root.load_signing_key(private_root_key2) - -# repository.status() shows missing verification and signing keys for the -# top-level roles, and whether signatures can be created (also see #955). -# This output shows that so far only the "root" role meets the key threshold and -# can successfully sign its metadata. ->>> repository.status() -'targets' role contains 0 / 1 public keys. -'snapshot' role contains 0 / 1 public keys. -'timestamp' role contains 0 / 1 public keys. -'root' role contains 2 / 2 signatures. -'targets' role contains 0 / 1 signatures. - -# In the next section we update the other top-level roles and create a repository -# with valid metadata. -``` - -#### Create Timestamp, Snapshot, Targets -Now that `root.json` has been set, the other top-level roles may be created. -The signing keys added to these roles must correspond to the public keys -specified by the Root role. - -On the client side, `root.json` must always exist. The other top-level roles, -created next, are requested by repository clients in (Root -> Timestamp -> -Snapshot -> Targets) order to ensure required metadata is downloaded in a -secure manner. - -```python -# Continuing from the previous section . . . - -# 'datetime' module needed to optionally set a role's expiration. ->>> import datetime - -# Generate keys for the remaining top-level roles. The root keys have been set above. ->>> generate_and_write_rsa_keypair(password='password', filepath='targets_key') ->>> generate_and_write_rsa_keypair(password='password', filepath='snapshot_key') ->>> generate_and_write_rsa_keypair(password='password', filepath='timestamp_key') - -# Add the verification keys of the remaining top-level roles. - ->>> repository.targets.add_verification_key(import_rsa_publickey_from_file('targets_key.pub')) ->>> repository.snapshot.add_verification_key(import_rsa_publickey_from_file('snapshot_key.pub')) ->>> repository.timestamp.add_verification_key(import_rsa_publickey_from_file('timestamp_key.pub')) - -# Import the signing keys of the remaining top-level roles. ->>> private_targets_key = import_rsa_privatekey_from_file('targets_key', password='password') ->>> private_snapshot_key = import_rsa_privatekey_from_file('snapshot_key', password='password') ->>> private_timestamp_key = import_rsa_privatekey_from_file('timestamp_key', password='password') - -# Load the signing keys of the remaining roles so that valid signatures are -# generated when repository.writeall() is called. ->>> repository.targets.load_signing_key(private_targets_key) ->>> repository.snapshot.load_signing_key(private_snapshot_key) ->>> repository.timestamp.load_signing_key(private_timestamp_key) - -# Optionally set the expiration date of the timestamp role. By default, roles -# are set to expire as follows: root(1 year), targets(3 months), snapshot(1 -# week), timestamp(1 day). ->>> repository.timestamp.expiration = datetime.datetime(2080, 10, 28, 12, 8) - -# Mark roles for metadata update (see #964, #958) ->>> repository.mark_dirty(['root', 'snapshot', 'targets', 'timestamp']) - -# Write all metadata to "repository/metadata.staged/" ->>> repository.writeall() -``` - -### Targets ### -TUF makes it possible for clients to validate downloaded target files by -including a target file's length, hash(es), and filepath in metadata. The -filepaths are relative to a `targets/` directory on the software repository. A -TUF client can download a target file by first updating the latest copy of -metadata (and thus available targets), verifying that their length and hashes -are valid, and saving the target file(s) locally to complete the update -process. - -In this section, the target files intended for clients are added to a -repository and listed in `targets.json` metadata. - -#### Add Target Files #### - -The repository maintainer adds target files to roles (e.g., `targets` and -`unclaimed`) by specifying their filepaths. The target files must exist at the -specified filepaths before the repository tool can generate and add their -(hash(es), length, and filepath) to metadata. - -First, the actual target files are manually created and saved to the `targets/` -directory of the repository: - -```Bash -# Create and save target files to the targets directory of the software -# repository. -$ cd repository/targets/ -$ echo 'file1' > file1.txt -$ echo 'file2' > file2.txt -$ echo 'file3' > file3.txt -$ mkdir myproject; echo 'file4' > myproject/file4.txt -$ cd ../../ -``` - -With the target files available on the `targets/` directory of the software -repository, the `add_targets()` method of a Targets role can be called to add -the target filepaths to metadata. - -```python -# Continuing from the previous section . . . - -# NOTE: If you exited the Python interactive interpreter above you need to -# re-import the repository_tool-functions and re-load the repository and -# signing keys. ->>> from tuf.repository_tool import * - -# The 'os' module is needed to gather file attributes, which will be included -# in a custom field for some of the target files added to metadata. ->>> import os - -# Load the repository created in the previous section. This repository so far -# contains metadata for the top-level roles, but no target paths are yet listed -# in targets metadata. ->>> repository = load_repository('repository') - -# Create a list of all targets in the directory. ->>> list_of_targets = ['file1.txt', 'file2.txt', 'file3.txt'] - -# Add the list of target paths to the metadata of the top-level Targets role. -# Any target file paths that might already exist are NOT replaced, and -# add_targets() does not create or move target files on the file system. Any -# target paths added to a role must fall under the expected targets directory, -# otherwise an exception is raised. The targets added to a role should actually -# exist once writeall() or write() is called, so that the hash and size of -# these targets can be included in Targets metadata. ->>> repository.targets.add_targets(list_of_targets) - -# Individual target files may also be added to roles, including custom data -# about the target. In the example below, file permissions of the target -# (octal number specifying file access for owner, group, others e.g., 0755) is -# added alongside the default fileinfo. All target objects in metadata include -# the target's filepath, hash, and length. -# Note: target path passed to add_target() method has to be relative -# to the targets directory or an exception is raised. ->>> target4_filepath = 'myproject/file4.txt' ->>> target4_abspath = os.path.abspath(os.path.join('repository', 'targets', target4_filepath)) ->>> octal_file_permissions = oct(os.stat(target4_abspath).st_mode)[4:] ->>> custom_file_permissions = {'file_permissions': octal_file_permissions} ->>> repository.targets.add_target(target4_filepath, custom_file_permissions) -``` - -The private keys of roles affected by the changes above must now be imported and -loaded. `targets.json` must be signed because a target file was added to its -metadata. `snapshot.json` keys must be loaded and its metadata signed because -`targets.json` has changed. Similarly, since `snapshot.json` has changed, the -`timestamp.json` role must also be signed. - -```Python -# Continuing from the previous section . . . - -# The private key of the updated targets metadata must be re-loaded before it -# can be signed and written (Note the load_repository() call above). ->>> private_targets_key = import_rsa_privatekey_from_file('targets_key') -enter password to decrypt private key file '/path/to/targets_key' -(leave empty if key not encrypted): - ->>> repository.targets.load_signing_key(private_targets_key) - -# Due to the load_repository() and new versions of metadata, we must also load -# the private keys of Snapshot and Timestamp to generate a valid set of metadata. ->>> private_snapshot_key = import_rsa_privatekey_from_file('snapshot_key') -enter password to decrypt private key file '/path/to/snapshot_key' -(leave empty if key not encrypted): ->>> repository.snapshot.load_signing_key(private_snapshot_key) - ->>> private_timestamp_key = import_rsa_privatekey_from_file('timestamp_key') -enter password to decrypt private key file '/path/to/timestamp_key' -(leave empty if key not encrypted): ->>> repository.timestamp.load_signing_key(private_timestamp_key) - -# Mark roles for metadata update (see #964, #958) ->>> repository.mark_dirty(['snapshot', 'targets', 'timestamp']) - -# Generate new versions of the modified top-level metadata (targets, snapshot, -# and timestamp). ->>> repository.writeall() -``` - -#### Remove Target Files #### - -Target files previously added to roles may also be removed. Removing a target -file requires first removing the target from a role and then writing the -new metadata to disk. -```python -# Continuing from the previous section . . . - -# Remove a target file listed in the "targets" metadata. The target file is -# not actually deleted from the file system. ->>> repository.targets.remove_target('myproject/file4.txt') - -# Mark roles for metadata update (see #964, #958) ->>> repository.mark_dirty(['snapshot', 'targets', 'timestamp']) - ->>> repository.writeall() -``` - -#### Excursion: Dump Metadata and Append Signature #### - -The following two functions are intended for those that wish to independently -sign metadata. Repository maintainers can dump the portion of metadata that is -normally signed, sign it with an external signing tool, and append the -signature to already existing metadata. - -First, the signable portion of metadata can be generated as follows: - -```Python ->>> signable_content = dump_signable_metadata('repository/metadata.staged/timestamp.json') -``` - -Then, use a tool like securesystemslib to create a signature over the signable -portion. *Note, to make the signing key count towards the role's signature -threshold, it needs to be added to `root.json`, e.g. via -`repository.timestamp.add_verification_key(key)` (not shown in below snippet).* -```python ->>> from securesystemslib.formats import encode_canonical ->>> from securesystemslib.keys import create_signature ->>> private_ed25519_key = import_ed25519_privatekey_from_file('ed25519_key') -enter password to decrypt private key file '/path/to/ed25519_key' ->>> signature = create_signature( -... private_ed25519_key, encode_canonical(signable_content).encode()) -``` - -Finally, append the signature to the metadata -```Python ->>> append_signature(signature, 'repository/metadata.staged/timestamp.json') -``` - -Note that the format of the signature is the format expected in metadata, which -is a dictionary that contains a KEYID, the signature itself, etc. See the -specification and [METADATA.md](METADATA.md) for a detailed example. - -### Delegations ### -All of the target files available on the software repository created so far -have been added to one role (the top-level Targets role). However, what if -multiple developers are responsible for the files of a project? What if -responsibility separation is desired? Performing a delegation, where one role -delegates trust of some paths to another role, is an option for integrators -that require additional roles on top of the top-level roles available by -default. - -In the next sub-section, the `unclaimed` role is delegated from the top-level -`targets` role. The `targets` role specifies the delegated role's public keys, -the paths it is trusted to provide, and its role name. - -```python -# Continuing from the previous section . . . - -# Generate a key for a new delegated role named "unclaimed". ->>> generate_and_write_rsa_keypair(password='password', filepath='unclaimed_key', bits=2048) ->>> public_unclaimed_key = import_rsa_publickey_from_file('unclaimed_key.pub') - -# Make a delegation (delegate trust of 'myproject/*.txt' files) from "targets" -# to "unclaimed", where "unclaimed" initially contains zero targets. ->>> repository.targets.delegate('unclaimed', [public_unclaimed_key], ['myproject/*.txt']) - -# Thereafter, we can access the delegated role by its name to e.g. add target -# files, just like we did with the top-level targets role. ->>> repository.targets("unclaimed").add_target("myproject/file4.txt") - -# Load the private key of "unclaimed" so that unclaimed's metadata can be -# signed, and valid metadata created. ->>> private_unclaimed_key = import_rsa_privatekey_from_file('unclaimed_key', password='password') - ->>> repository.targets("unclaimed").load_signing_key(private_unclaimed_key) - -# Mark roles for metadata update (see #964, #958) ->>> repository.mark_dirty(['snapshot', 'targets','timestamp', 'unclaimed']) - ->>> repository.writeall() -``` - - - - -#### Wrap-up #### - -In summary, the five steps a repository maintainer follows to create a TUF -repository are: - -1. Create a directory for the software repository that holds the TUF metadata and the target files. -2. Create top-level roles (`root.json`, `snapshot.json`, `targets.json`, and `timestamp.json`.) -3. Add target files to the `targets` role. -4. Optionally, create delegated roles to distribute target files. -5. Write the changes. - -The repository tool saves repository changes to a `metadata.staged` directory. -Repository maintainers may push finalized changes to the "live" repository by -copying the staged directory to its destination. -```Bash -# Copy the staged metadata directory changes to the live repository. -$ cp -r "repository/metadata.staged/" "repository/metadata/" -``` - -## Consistent Snapshots ## -The basic TUF repository we have generated above is adequate for repositories -that have some way of guaranteeing consistency of repository data. A community -software repository is one example where consistency of files and metadata can -become an issue. Repositories of this kind are continually updated by multiple -maintainers and software authors uploading their packages, increasing the -likelihood that a client downloading version X of a release unexpectedly -requests the target files of a version Y just released. - -To guarantee consistency of metadata and target files, a repository may -optionally support multiple versions of `snapshot.json` simultaneously, where a -client with version 1 of `snapshot.json` can download `target_file.zip` and -another client with version 2 of `snapshot.json` can also download a different -`target_file.zip` (same file name, but different file digest.) If the -`consistent_snapshot` parameter of writeall() or write() are `True`, metadata -and target file names on the file system have their digests prepended (note: -target file names specified in metadata do not contain digests in their names.) - -The repository maintainer is responsible for the duration of multiple versions -of metadata and target files available on a repository. Generating consistent -metadata and target files on the repository is enabled by setting the -`consistent_snapshot` argument of `writeall()` or `write()` . Note that -changing the consistent_snapshot setting involves writing a new version of -root. - - - -## Delegate to Hashed Bins ## -Why use hashed bin delegations? - -For software update systems with a large number of target files, delegating to -hashed bins (a special type of delegated role) might be an easier alternative -to manually performing the delegations. How many target files should each -delegated role contain? How will these delegations affect the number of -metadata that clients must additionally download in a typical update? Hashed -bin delegations are available to integrators that rather not deal with the -management of delegated roles and a great number of target files. - -A large number of target files may be distributed to multiple hashed bins with -`delegate_hashed_bins()`. The metadata files of delegated roles will be nearly -equal in size (i.e., target file paths are uniformly distributed by calculating -the target filepath's digest and determining which bin it should reside in.) -The updater client will use "lazy bin walk" (visit and download the minimum -metadata required to find a target) to find a target file's hashed bin -destination. This method is intended for repositories with a large number of -target files, a way of easily distributing and managing the metadata that lists -the targets, and minimizing the number of metadata files (and size) downloaded -by the client. - -The `delegate_hashed_bins()` method has the following form: -```Python -delegate_hashed_bins(list_of_targets, keys_of_hashed_bins, number_of_bins) -``` - -We next provide a complete example of retrieving target paths to add to hashed -bins, performing the hashed bin delegations, signing them, and delegating paths -to some role. - -```Python -# Continuing from the previous section . . . - -# Remove 'myproject/file4.txt' from unclaimed role and instead further delegate -# all targets in myproject/ to hashed bins. ->>> repository.targets('unclaimed').remove_target("myproject/file4.txt") - -# Get a list of target paths for the hashed bins. ->>> targets = ['myproject/file4.txt'] - -# Delegate trust to 32 hashed bin roles. Each role is responsible for the set -# of target files, determined by the path hash prefix. TUF evenly distributes -# hexadecimal ranges over the chosen number of bins (see output). -# To initialize the bins we use one key, which TUF warns us about (see output). -# However, we can assign separate keys to each bin, with the method used in -# previous sections, accessing a bin by its hash prefix range name, e.g.: -# "repository.targets('00-07').add_verification_key('public_00-07_key')". ->>> repository.targets('unclaimed').delegate_hashed_bins( -... targets, [public_unclaimed_key], 32) -Creating hashed bin delegations. -1 total targets. -32 hashed bins. -256 total hash prefixes. -Each bin ranges over 8 hash prefixes. -Adding a verification key that has already been used. [repeated 32x] - -# The hashed bin roles can also be accessed by iterating the "delegations" -# property of the delegating role, which we do here to load the signing key. ->>> for delegation in repository.targets('unclaimed').delegations: -... delegation.load_signing_key(private_unclaimed_key) - -# Mark roles for metadata update (see #964, #958) ->>> repository.mark_dirty(['00-07', '08-0f', '10-17', '18-1f', '20-27', '28-2f', -... '30-37', '38-3f', '40-47', '48-4f', '50-57', '58-5f', '60-67', '68-6f', -... '70-77', '78-7f', '80-87', '88-8f', '90-97', '98-9f', 'a0-a7', 'a8-af', -... 'b0-b7', 'b8-bf', 'c0-c7', 'c8-cf', 'd0-d7', 'd8-df', 'e0-e7', 'e8-ef', -... 'f0-f7', 'f8-ff', 'snapshot', 'timestamp', 'unclaimed']) - ->>> repository.writeall() - -``` - -## How to Perform an Update ## - -The following [repository tool](../tuf/repository_tool.py) function creates a directory -structure that a client downloading new software using TUF (via -[tuf/client/updater.py](../tuf/client/updater.py)) expects. The `root.json` metadata file must exist, and -also the directories that hold the metadata files downloaded from a repository. -Software updaters integrating TUF may use this directory to store TUF updates -saved on the client side. - -```python ->>> from tuf.repository_tool import * ->>> create_tuf_client_directory("repository/", "client/tufrepo/") -``` - -`create_tuf_client_directory()` moves metadata from `repository/metadata` to -`client/` in this example. The repository in `repository/` may be the -repository example created earlier in this document. - -## Test TUF Locally ## -Run the local TUF repository server. -```Bash -$ cd "repository/"; python3 -m http.server 8001 -``` - -We next retrieve targets from the TUF repository and save them to `client/`. -The `client.py` script is available to download metadata and files from a -specified repository. In a different command-line prompt, where `tuf` is -installed . . . -```Bash -$ cd "client/" -$ ls -tufrepo/ - -$ client.py --repo http://localhost:8001 file1.txt -$ ls . tuftargets/ -.: -tufrepo tuftargets - -tuftargets/: -file1.txt -``` diff --git a/docs/conf.py b/docs/conf.py index 9caa8feae3..1f7ad7aa1c 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -38,11 +38,6 @@ # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -# This pattern also affects html_static_path and html_extra_path. -exclude_patterns = ['GETTING_STARTED.rst'] - # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for diff --git a/docs/images/repository_tool-diagram.png b/docs/images/repository_tool-diagram.png deleted file mode 100644 index 6bfbdeb0b7..0000000000 Binary files a/docs/images/repository_tool-diagram.png and /dev/null differ diff --git a/tuf/ATTACKS.md b/tuf/ATTACKS.md deleted file mode 100644 index 416e164ddb..0000000000 --- a/tuf/ATTACKS.md +++ /dev/null @@ -1,323 +0,0 @@ -# Demonstrate protection against malicious updates - -## Table of Contents ## -- [Blocking Malicious Updates](#blocking-malicious-updates) - - [Arbitrary Package Attack](#arbitrary-package-attack) - - [Rollback Attack](#rollback-attack) - - [Indefinite Freeze Attack](#indefinite-freeze-attack) - - [Endless Data Attack](#endless-data-attack) - - [Compromised Key Attack](#compromised-key-attack) - - [Slow Retrieval Attack](#slow-retrieval-attack) -- [Conclusion](#conclusion) - -## Blocking Malicious Updates ## -TUF protects against a number of attacks, some of which include rollback, -arbitrary package, and mix and match attacks. We begin this document on -blocking malicious updates by demonstrating how the client rejects a target -file downloaded from the software repository that doesn't match what is listed -in TUF metadata. - -The following demonstration requires and operates on the repository created in -the [repository management -tutorial](https://github.com/theupdateframework/python-tuf/blob/develop/tuf/README.md). - -### Arbitrary Package Attack ### -In an arbitrary package attack, an attacker installs anything they want on the -client system. That is, an attacker can provide arbitrary files in response to -download requests and the files will not be detected as illegitimate. We -simulate an arbitrary package attack by creating a "malicious" target file -that our client attempts to fetch. - -```Bash -$ mv 'repository/targets/file2.txt' 'repository/targets/file2.txt.backup' -$ echo 'bad_target' > 'repository/targets/file2.txt' -``` - -We next reset our local timestamp (so that a new update is prompted), and -the target files previously downloaded by the client. -```Bash -$ rm -rf "client/targets/" "client/metadata/current/timestamp.json" -``` - -The client now performs an update and should detect the invalid target file... -Note: The following command should be executed in the "client/" directory. -```Bash -$ python3 basic_client.py --repo http://localhost:8001 -Error: No working mirror was found: - localhost:8001: BadHashError() -``` - -The log file (tuf.log) saved to the current working directory contains more -information on the update procedure and the cause of the BadHashError. - -```Bash -... - -BadHashError: Observed -hash ('f569179171c86aa9ed5e8b1d6c94dfd516123189568d239ed57d818946aaabe7') != -expected hash (u'67ee5478eaadb034ba59944eb977797b49ca6aa8d3574587f36ebcbeeb65f70e') -[2016-10-20 19:45:16,079 UTC] [tuf.client.updater] [ERROR] [_get_file:1415@updater.py] -Failed to update /file2.txt from all mirrors: {u'http://localhost:8001/targets/file2.txt': BadHashError()} -``` - -Note: The "malicious" target file should be removed and the original file2.txt -restored, otherwise the following examples will fail with BadHashError -exceptions: - -```Bash -$ mv 'repository/targets/file2.txt.backup' 'repository/targets/file2.txt' -``` - -### Indefinite Freeze Attack ### -In an indefinite freeze attack, an attacker continues to present a software -update system with the same files the client has already seen. The result is -that the client does not know that new files are available. Although the -client would be unable to prevent an attacker or compromised repository from -feeding it stale metadata, it can at least detect when an attacker is doing so -indefinitely. The signed metadata used by TUF contains an "expires" field that -indicates when metadata should no longer be trusted. - -In the following simulation, the client first tries to perform an update. - -```Bash -$ python3 basic_client.py --repo http://localhost:8001 -``` - -According to the logger (`tuf.log` file in the current working directory), -everything appears to be up-to-date. The remote server should also show that -the client retrieved only the timestamp.json file. Let's suppose now that an -attacker continues to feed our client the same stale metadata. If we were to -move the time to a future date that would cause metadata to expire, the TUF -framework should raise an exception or error to indicate that the metadata -should no longer be trusted. - -```Bash -$ sudo date -s '2080-12-25 12:34:56' -Wed Dec 25 12:34:56 EST 2080 - -$ python3 basic_client.py --repo http://localhost:8001 -Error: No working mirror was found: - u'localhost:8001': ExpiredMetadataError(u"Metadata u'root' expired on Tue Jan 1 00:00:00 2030 (UTC).",) -``` - -Note: Reset the date to continue with the rest of the attacks. - - -### Rollback Attack ### -In a rollback attack, an attacker presents a software update system with older -files than those the client has already seen, causing the client to use files -older than those the client knows about. We begin this example by saving the -current version of the Timestamp file available on the repository. This saved -file will later be served to the client to see if it is rejected. The client -should not accept versions of metadata that is older than previously trusted. - -Navigate to the directory containing the server's files and save the current -timestamp.json to a temporary location: -```Bash -$ cp repository/metadata/timestamp.json /tmp -``` - -We should next generate a new Timestamp file on the repository side. -```Bash -$ python3 ->>> from tuf.repository_tool import * ->>> repository = load_repository('repository') ->>> repository.timestamp.version -1 ->>> repository.timestamp.version = 2 ->>> repository.dirty_roles() -Dirty roles: [u'timestamp'] ->>> private_timestamp_key = import_rsa_privatekey_from_file("keystore/timestamp_key") -Enter a password for the encrypted RSA file (/path/to/keystore/timestamp_key): ->>> repository.timestamp.load_signing_key(private_timestamp_key) ->>> repository.write('timestamp') - -$ cp repository/metadata.staged/* repository/metadata -``` - -Now start the HTTP server from the directory containing the 'repository' -subdirectory. -```Bash -$ python3 -m SimpleHTTPServer 8001 -``` - -And perform an update so that the client retrieves the updated timestamp.json. -```Bash -$ python3 basic_client.py --repo http://localhost:8001 -``` - -Finally, move the previous timestamp.json file to the current live repository -and have the client try to download the outdated version. The client should -reject it! -```Bash -$ cp /tmp/timestamp.json repository/metadata/ -$ cd repository; python3 -m SimpleHTTPServer 8001 -``` - -On the client side, perform an update... -```Bash -$ python3 basic_client.py --repo http://localhost:8001 -Error: No working mirror was found: - u'localhost:8001': ReplayedMetadataError() -``` - -The tuf.log file contains more information about the ReplayedMetadataError -exception and update process. Please reset timestamp.json to the latest -version, which can be found in the 'repository/metadata.staged' subdirectory. - -```Bash -$ cp repository/metadata.staged/timestamp.json repository/metadata -``` - - -### Endless Data Attack ### -In an endless data attack, an attacker responds to a file download request with -an endless stream of data, causing harm to clients (e.g., a disk partition -filling up or memory exhaustion). In this simulated attack, we append extra -data to one of the target files available on the software repository. The -client should only download the exact number of bytes it expects for a -requested target file (according to what is listed in trusted TUF metadata). - -```Bash -$ cp repository/targets/file1.txt /tmp -$ python3 -c "print 'a' * 1000" >> repository/targets/file1.txt -``` - -Now delete the local metadata and target files on the client side so -that remote metadata and target files are downloaded again. -```Bash -$ rm -rf client/targets/ -$ rm client/metadata/current/snapshot.json* client/metadata/current/timestamp.json* -``` - -Lastly, perform an update to verify that the file1.txt is downloaded up to the -expected size, and no more. The target file available on the software -repository does contain more data than expected, though. - -```Bash -$ python3 basic_client.py --repo http://localhost:8001 -``` - -At this point, part of the "file1.txt" file should have been fetched. That is, -up to 31 bytes of it should have been downloaded, and the rest of the maliciously -appended data ignored. If we inspect the logger, we'd discover the following: - -```Bash -[2016-10-06 21:37:39,092 UTC] [tuf.download] [INFO] [_download_file:235@download.py] -Downloading: u'http://localhost:8001/targets/file1.txt' - -[2016-10-06 21:37:39,145 UTC] [tuf.download] [INFO] [_check_downloaded_length:610@download.py] -Downloaded 31 bytes out of the expected 31 bytes. - -[2016-10-06 21:37:39,145 UTC] [tuf.client.updater] [INFO] [_get_file:1372@updater.py] -Not decompressing http://localhost:8001/targets/file1.txt - -[2016-10-06 21:37:39,145 UTC] [tuf.client.updater] [INFO] [_check_hashes:778@updater.py] -The file's sha256 hash is correct: 65b8c67f51c993d898250f40aa57a317d854900b3a04895464313e48785440da -``` - -Indeed, the sha256 sum of the first 31 bytes of the "file1.txt" available -on the repository should match to what is trusted. The client did not -downloaded the appended data. - -Note: Restore file1.txt - -```Bash -$ cp /tmp/file1.txt repository/targets/ -``` - - -### Compromised Key Attack ### -An attacker who compromise less than a given threshold of keys is limited in -scope. This includes relying on a single online key (such as only being -protected by SSL) or a single offline key (such as most software update systems -use to sign files). In this example, we attempt to sign a role file with -less-than-a-threshold number of keys. A single key (suppose this is a -compromised key) is used to demonstrate that roles must be signed with the -total number of keys required for the role. In order to compromise a role, an -attacker would have to compromise a threshold of keys. This approach of -requiring a threshold number of signatures provides compromise resilience. - -Let's attempt to sign a new snapshot file with a less-than-threshold number of -keys. The client should reject the partially signed snapshot file served by -the repository (or imagine that it is a compromised software repository). - -```Bash -$ python3 ->>> from tuf.repository_tool import * ->>> repository = load_repository('repository') ->>> version = repository.root.version ->>> repository.root.version = version + 1 ->>> private_root_key = import_rsa_privatekey_from_file("keystore/root_key", password="password") ->>> repository.root.load_signing_key(private_root_key) ->>> private_root_key2 = import_rsa_privatekey_from_file("keystore/root_key2", password="password") ->>> repository.root.load_signing_key(private_root_key2) - ->>> repository.snapshot.version = 8 ->>> repository.snapshot.threshold = 2 ->>> private_snapshot_key = import_rsa_privatekey_from_file("keystore/snapshot_key", password="password") ->>> repository.snapshot.load_signing_key(private_snapshot_key) - ->>> repository.timestamp.version = 8 ->>> private_timestamp_key = import_rsa_privatekey_from_file("keystore/timestamp_key", password="password") ->>> repository.timestamp.load_signing_key(private_timestamp_key) - ->>> repository.write('root') ->>> repository.write('snapshot') ->>> repository.write('timestamp') - -$ cp repository/metadata.staged/* repository/metadata -``` - -The client now attempts to refresh the top-level metadata and the -partially written snapshot.json, which should be rejected. - -```Bash -$ python3 basic_client.py --repo http://localhost:8001 -Error: No working mirror was found: - u'localhost:8001': BadSignatureError() -``` - - -### Slow Retrieval Attack ### -In a slow retrieval attack, an attacker responds to clients with a very slow -stream of data that essentially results in the client never continuing the -update process. In this example, we simulate a slow retrieval attack by -spawning a server that serves data at a slow rate to our update client data. -TUF should not be vulnerable to this attack, and the framework should raise an -exception or error when it detects that a malicious server is serving it data -at a slow enough rate. - -We first spawn the server that slowly streams data to the client. The -'slow_retrieval_server_old.py' module (can be found in the tests/ directory of the -source code) should be copied over to the server's 'repository/' directory from -which to launch it. - -```Bash -# Before launching the slow retrieval server, copy 'slow_retrieval_server_old.py' -# to the 'repository/' directory and run it from that directory as follows: -$ python3 slow_retrieval_server_old.py 8002 mode_2 -``` - -The client may now make a request to the slow retrieval server on port 8002. -However, before doing so, we'll reduce (for the purposes of this demo) the -minimum average download rate allowed and download chunk size. Open the -'settings.py' module and set MIN_AVERAGE_DOWNLOAD_SPEED = 5 and CHUNK_SIZE = 1. -This should make it so that the client detects the slow retrieval server's -delayed streaming. - -```Bash -$ python3 basic_client.py --verbose 1 --repo http://localhost:8002 -Error: No working mirror was found: - u'localhost:8002': SlowRetrievalError() -``` - -The framework should detect the slow retrieval attack and raise a -SlowRetrievalError exception to the client application. - - -## Conclusion ## -These are just some of the attacks that TUF provides protection against. For -more attacks and updater weaknesses, please see the -[Security](https://theupdateframework.io/security/) -page. diff --git a/tuf/README-developer-tools.md b/tuf/README-developer-tools.md deleted file mode 100644 index 1b593400a5..0000000000 --- a/tuf/README-developer-tools.md +++ /dev/null @@ -1,342 +0,0 @@ -# The Update Framework Developer Tool: How to Update your Project Securely on a TUF Repository - -## Table of Contents -- [Overview](#overview) -- [Creating a Simple Project](#creating_a_simple_project) - - [Generating a Key](#generating_a_key) - - [The Project Class](#the_project_class) - - [Signing and Writing the Metadata](#signing_and_writing_the_metadata) -- [Loading an Existing Project](#loading_an_existing_project) -- [Delegations](#delegations) -- [Managing Keys](#managing_keys) -- [Managing Targets](#managing_targets) - - -## Overview -The Update Framework (TUF) is a Python-based security system for software -updates. In order to prevent your users from downloading vulnerable or malicious -code disguised as updates to your software, TUF requires that each update you -release include certain metadata verifying your authorship of the files. - -The TUF developer tools are a Python Library that enables you to create and -maintain the required metadata for files hosted on a TUF Repository. (We call -these files “targets,” to distinguish them from the metadata associated with -them. Both of these together comprise a complete “project”.) You will use these -tools to generate the keys and metadata you need to claim and secure your files -on the repository, and to update the metadata and sign it with those keys -whenever you upload a new version of those files. - -This document will teach you how to use these tools in two parts. The first -part walks through the creation of a minimal-complexity TUF project, which is -all you need to get started, and can be expanded later. The second part details -the full functionality of the tools, which offer a finer degree of control in -securing your project. - - -## Creating a Simple Project -This section walks through the creation of a small example project with just -one target. Once created, this project will be fully functional, and can be -modified as needed. - - -### Generating a Key -First, we will need to generate a key to sign the metadata. Keys are generated -in pairs: one public and the other private. The private key is -password-protected and is used to sign metadata. The public key can be shared -freely, and is used to verify signatures made by the private key. You will need -to share your public key with the repository hosting your project so they can -verify your metadata is signed by the right person. - -The generate\_and\_write\_rsa\_keypair function will create two key files named -"path/to/key.pub", which is the public key and "path/to/key", which -is the private key. - -``` ->>> from tuf.developer_tool import * ->>> generate_and_write_rsa_keypair_with_prompt(filepath="path/to/key") -enter password to encrypt private key file 'path/to/key' -(leave empty if key should not be encrypted): -Confirm: ->>> -``` - -We can also use the bits parameter to set a different key length (the default -is 3072). We can also `generate_and_write_rsa_keypair` with a `password` -parameter if a prompt is not desired. - -In this example we will be using rsa keys, but ed25519 keys are also supported. - -Now we have a key for our project, we can proceed to create our project. - - -### The Project Class -The TUF developer tool is built around the Project class, which is used to -organize groups of targets associated with a single set of metadata. A single -Project instance is used to keep track of all the target files and metadata -files in one project. The Project also keeps track of the keys and signatures, -so that it can update all the metadata with the correct changes and signatures -on a single command. - -Before creating a project, you must know where it will be located in the TUF -Repository. In the following example, we will create a project to be hosted as -"repo/unclaimed/example_project" within the repository, and store a local copy -of the metadata at "path/to/metadata". The project will comprise a single -target file, "local/path/to/example\_project/target\_1" locally, and we will -secure it with the key generated above. - -First, we must import the generated keys. We can do that by issuing the -following command: - -``` ->>> public_key = import_rsa_publickey_from_file("path/to/keys.pub") -``` - -After importing the key, we can generate a new project with the following -command: - -``` ->>> project = create_new_project(project_name="example_project", -... metadata_directory="local/path/to/metadata/", -... targets_directory="local/path/to/example_project", -... location_in_repository="repo/unclaimed", key=public_key) -``` - -Let's list the arguments and make sense out of this rather long function call: - -- create a project named example_project: the name of the metadata file will match this name -- the metadata will be located in "local/path/to/metadata", this means all of the generated files -for this project will be located here -- the targets are located in local/path/to/example project. If your targets are located in some other -place, you can point the targets directory there. Files must reside under the path local/path/to/example_project or else it won't be possible to add them. -- location\_in\_repository points to repo/unclaimed, this will be prepended to the paths in the generated metadata so the signatures all match. - -Now the project is in memory and we can do different operations on it such as -adding and removing targets, delegating files, changing signatures and keys, -etc. For the moment we are interested in adding our one and only target inside -the project. - -To add a target, we issue the following method: - -``` ->>> project.add_target("local/path/to/example_project/target_1") -``` - -Note that the file "target\_1" should be located in -"local/path/to/example\_project", or this method will throw an -error. - -At this point, the metadata is not valid. We have assigned a key to the -project, but we have not *signed* it with that key. Signing is the process of -generating a signature with our private key so it can be verified with the -public key by the server (upon uploading) and by the clients (when updating). - - -### Signing and Writing the Metadata ### -In order to sign the metadata, we need to import the private key corresponding -to the public key we added to the project. One the key is loaded to the project, -it will automatically be used to sign the metadata whenever it is written. - -``` ->>> private_key = import_rsa_privatekey_from_file("path/to/key") -Enter password for the RSA key: ->>> project.load_signing_key(private_key) ->>> project.write() -``` - -When all changes to the project have been written, the metadata is ready to be -uploaded to the repository, and it is safe to exit the Python interpreter, or -to delete the Project instance. - -The project can be loaded later to update changes to the project. The metadata -contains checksums that have to match the actual files or else it won't be -accepted by the upstream repository. - -At this point, if you have followed all the steps in this document so far -(substituting appropriate names and filepaths) you will have created a basic -TUF project, which can be expanded as needed. The simplest way to get your -project secured is to add all your files using add\_target() (or see [Managing -Keys](#managing_keys) on how to add whole directories). If your project has -several contributors, you may want to consider adding -[delegations](#delegations) to your project. - - -## Loading an Existing Project -To make changes to existing metadata, we will need the Project again. We can -restore it with the load_project() function. - -``` ->>> from tuf.developer_tool import * ->>> project = load_project("local/path/to/metadata") -``` -Each time the project is loaded anew, the necessary private keys must also be -loaded in order to sign metadata. - -``` ->>> private_key = import_rsa_privatekey_from_file("path/to/key") -Enter a password for the RSA key: ->>> project.load_signing_key(private_key) ->>> project.write() -``` - -If your project does not use any delegations, the five commands above are all -you need to update your project's metadata. - - -## Delegations - -The project we created above is secured entirely by one key. If you want to -allow someone else to update part of your project independently, you will need -to delegate a new role for them. For example, we can do the following: - -``` ->>> other_key = import_rsa_publickey_from_file(“another_public_key.pub”) ->>> targets = ['local/path/to/newtarget'] ->>> project.delegate(“newrole”, [other_key], targets) -``` - -The new role is now an attribute of the Project instance, and contains the same -methods as Project. For example, we can add targets in the same way as before: - -``` ->>> project(“newrole”).add_target(“delegated_1”) -``` - -Recall that we input the other person’s key as part of a list. That list can -contain any number of public keys. We can also add keys to the role after -creating it using the [add\_verification\_key()](#adding_a_key_to_a_delegation) -method. - -### Delegated Paths - -By default, a delegated role is permitted to add and modify targets anywhere in -the Project's targets directory. We can delegate trust of paths to a role to -limit this permission. - -``` ->>> project.add_paths(["delegated/filepath"], "newrole") -``` - -This will prevent the delegated role from signing targets whose local filepaths -do not begin with "delegated/filepath". We can delegate several filepaths to a -role by adding them to the list in the first parameter, or by invoking the -method again. A role with multiple delegated paths can add targets to any of -them. - -Note that this method is invoked from the parent role (in this case, the Project) -and takes the delegated role name as an argument. - -### Nested Delegations - -It is possible for a delegated role to have delegations of its own. We can do -this by calling delegate() on a delegated role: - -``` ->>> project("newrole").delegate(“nestedrole”, [key], targets) -``` - -Nested delegations function no differently than first-order delegations. to -demonstrate, adding a target to nested delegation looks like this: - -``` ->>> project("newrole")("nestedrole").add_target("foo") -``` - -### Revoking Delegations -Delegations can be revoked, removing the delegated role from the project. - -``` ->>> project.revoke("newrole") -``` - - -## Managing Keys -This section describes the key-related functions and parameters not covered in -the [Creating a Simple Project](#creating_a_simple_project) section. - -### Additional Parameters for Key Generation -When generating keys, it is possible to specify the length of the key in bits -and its password as parameters: - -``` ->>> generate_and_write_rsa_keypair(password="pw", filepath="path/to/key", bits=2048) -``` -The bits parameter defaults to 3072, and values below 2048 will raise an error. -The password parameter is only intended to be used in scripts. - - -### Adding a Key to a Delegation -New verifications keys can be added to an existing delegation using -add\_verification\_key(): - -``` ->>> project("rolename").add_verification_key(pubkey) -``` - -A delegation can have several verification keys at once. By default, a -delegated role with multiple keys can be written using any one of their -corresponding signing keys. To modify this behavior, you can change the -delegated role's [threshold](#delegation_thrsholds). - -### Removing a Key from a Delegation -Verification keys can also be removed, like this: - -``` ->>> project("rolename").remove_verification_key(pubkey) -``` - -Remember that a project can only have one key, so this method will return an -error if there is already a key assigned to it. In order to replace a key we -must first delete the existing one and then add the new one. It is possible to -omit the key parameter in the create\_new\_project() function, and add the key -later. - -### Changing the Project Key -Each Project instance can only have one verification key. This key can be -replaced by removing it and adding a new key, in that order. - -``` ->>> project.remove_verification_key(oldkey) ->>> project.add_verification_key(new) -``` - - -### Delegation Thresholds - -Every delegated role has a threshold, which determines how many of its signing -keys need to be loaded to write the role. The threshold defaults to 1, and -should not exceed the number of verification keys assigned to the role. The -threshold can be accessed as a property of a delegated role. - -``` ->>> project("rolename").threshold = 2 -``` - -The above line will set the "rolename" role's threshold to 2. - - -## Managing Targets -There are supporting functions of the targets library to make the project -maintenance easier. These functions are described in this section. - -### Adding Targets by Directory -This function is especially useful when creating a new project to add all the -files contained in the targets directory. The following code block illustrates -the usage of this function: - -``` ->>> list_of_targets = \ -... project.get_filepaths_in_directory(“path/within/targets/folder”, -... recursive_walk=False, follow_links=False) ->>> project.add_targets(list_of_targets) -``` - -### Deleting Targets from a Project -It is possible that we want to delete existing targets inside our project. To -stop the developer tool from tracking this file we can issue the following -command: - -``` ->>> project.remove_target(“target_1”) -``` - -Now the target file won't be part of the metadata. diff --git a/tuf/README.md b/tuf/README.md deleted file mode 100644 index dbc53b61b5..0000000000 --- a/tuf/README.md +++ /dev/null @@ -1,5 +0,0 @@ -[Quickstart](../docs/QUICKSTART.md) - -[CLI](../docs/CLI.md) - -[Tutorial](../docs/TUTORIAL.md) diff --git a/tuf/client/README.md b/tuf/client/README.md deleted file mode 100644 index 29b838bc4d..0000000000 --- a/tuf/client/README.md +++ /dev/null @@ -1,151 +0,0 @@ -# updater.py -**updater.py** is intended as the only TUF module that software update -systems need to utilize for a low-level integration. It provides a single -class representing an updater that includes methods to download, install, and -verify metadata or target files in a secure manner. Importing -**tuf.client.updater** and instantiating its main class is all that is -required by the client prior to a TUF update request. The importation and -instantiation steps allow TUF to load all of the required metadata files -and set the repository mirror information. - -The **tuf.repository_tool** module can be used to create a TUF repository. See -[tuf/README](../README.md) for more information on creating TUF repositories. - - -## Overview of the Update Process - -1. The software update system instructs TUF to check for updates. - -2. TUF downloads and verifies timestamp.json. - -3. If timestamp.json indicates that snapshot.json has changed, TUF downloads and -verifies snapshot.json. - -4. TUF determines which metadata files listed in snapshot.json differ from those -described in the last snapshot.json that TUF has seen. If root.json has changed, -the update process starts over using the new root.json. - -5. TUF provides the software update system with a list of available files -according to targets.json. - -6. The software update system instructs TUF to download a specific target -file. - -7. TUF downloads and verifies the file and then makes the file available to -the software update system. - - -If at any point in the above procedure there is a problem (i.e., if unexpired, -signed, valid metadata cannot be retrieved from the repository), the Root file -is downloaded and the process is retried once more (and only once to avoid an -infinite loop). Optionally, the software update system using the framework -can decide how to proceed rather than automatically downloading a new Root file. - - -## Example Client -### Refresh TUF Metadata -```Python -# The client first imports the 'updater.py' module, the only module the -# client is required to import. The client will utilize a single class -# from this module. -import tuf.client.updater -import tuf.settings - -# The only other module the client interacts with is 'settings'. The -# client accesses this module solely to set the repository directory. -# This directory will hold the files downloaded from a remote repository. -tuf.settings.repositories_directory = 'path/to/local_repository' - -# Next, the client creates a dictionary object containing the repository -# mirrors. The client may download content from any one of these mirrors. -# In the example below, a single mirror named 'mirror1' is defined. The -# mirror is located at 'http://localhost:8001', and all of the metadata -# and targets files can be found in the 'metadata' and 'targets' directory, -# respectively. If the client wishes to only download target files from -# specific directories on the mirror, the 'confined_target_dirs' field -# should be set. In this example, the client hasn't set confined_target_dirs, -# which is interpreted as no confinement. In other words, the client can download -# targets from any directory or subdirectories. If the client had chosen -# 'targets1/', they would have been confined to the '/targets/targets1/' -# directory on the 'http://localhost:8001' mirror. -repository_mirrors = {'mirror1': {'url_prefix': 'http://localhost:8001', - 'metadata_path': 'metadata', - 'targets_path': 'targets'}} - -# The updater may now be instantiated. The Updater class of 'updater.py' -# is called with two arguments. The first argument assigns a name to this -# particular updater and the second argument the repository mirrors defined -# above. -updater = tuf.client.updater.Updater('updater', repository_mirrors) - -# The client calls the refresh() method to ensure it has the latest -# copies of the top-level metadata files (i.e., Root, Targets, Snapshot, -# Timestamp). -updater.refresh() -``` - - -### Download Specific Target File -```Python -# Example demonstrating an update that downloads a specific target. - -# Refresh the metadata of the top-level roles (i.e., Root, Targets, Snapshot, Timestamp). -updater.refresh() - -# get_one_valid_targetinfo() updates role metadata when required. In other -# words, if the client doesn't possess the metadata that lists 'LICENSE.txt', -# get_one_valid_targetinfo() will try to fetch / update it. -target = updater.get_one_valid_targetinfo('LICENSE.txt') -updated_target = updater.updated_targets([target], destination_directory) - -for target in updated_target: - updater.download_target(target, destination_directory) - # Client code here may also reference target information (including 'custom') - # by directly accessing the dictionary entries of the target. The 'custom' - # entry is additional file information explicitly set by the remote repository. - target_path = target['filepath'] - target_length = target['fileinfo']['length'] - target_hashes = target['fileinfo']['hashes'] - target_custom_data = target['fileinfo']['custom'] - - # Remove any files from the destination directory that are no longer being - # tracked. For example, a target file from a previous snapshot that has since - # been removed on the remote repository. - updater.remove_obsolete_targets(destination_directory) -``` - -### A Simple Integration Example with client.py -``` Bash -# Assume a simple TUF repository has been setup with 'repo.py'. -$ client.py --repo http://localhost:8001 - -# Metadata and target files are silently updated. An exception is only raised if an error, -# or attack, is detected. Inspect 'tuf.log' for the outcome of the update process. - -$ cat tuf.log -[2013-12-16 16:17:05,267 UTC] [tuf.download] [INFO][_download_file:726@download.py] -Downloading: http://localhost:8001/metadata/timestamp.json - -[2013-12-16 16:17:05,269 UTC] [tuf.download] [WARNING][_check_content_length:589@download.py] -reported_length (545) < required_length (2048) - -[2013-12-16 16:17:05,269 UTC] [tuf.download] [WARNING][_check_downloaded_length:656@download.py] -Downloaded 545 bytes, but expected 2048 bytes. There is a difference of 1503 bytes! - -[2013-12-16 16:17:05,611 UTC] [tuf.download] [INFO][_download_file:726@download.py] -Downloading: http://localhost:8001/metadata/snapshot.json - -[2013-12-16 16:17:05,612 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] -The file's sha256 hash is correct: 782675fadd650eeb2926d33c401b5896caacf4fd6766498baf2bce2f3b739db4 - -[2013-12-16 16:17:05,951 UTC] [tuf.download] [INFO][_download_file:726@download.py] -Downloading: http://localhost:8001/metadata/targets.json - -[2013-12-16 16:17:05,952 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] -The file's sha256 hash is correct: a5019c28a1595c43a14cad2b6252c4d1db472dd6412a9204181ad6d61b1dd69a - -[2013-12-16 16:17:06,299 UTC] [tuf.download] [INFO][_download_file:726@download.py] -Downloading: http://localhost:8001/targets/file1.txt - -[2013-12-16 16:17:06,303 UTC] [tuf.client.updater] [INFO][_check_hashes:636@updater.py] -The file's sha256 hash is correct: ecdc5536f73bdae8816f0ea40726ef5e9b810d914493075903bb90623d97b1d8