Skip to content

Commit

Permalink
Add guide for setting up cross-project testing (#710)
Browse files Browse the repository at this point in the history
Add guide for setting up cross-project testing

This is based on https://cockpit-project.org/blog/tmt-cross-project-testing.html, focusing on the configuration.

Reviewed-by: Tomas Tomecek <tomas@tomecek.net>
  • Loading branch information
softwarefactory-project-zuul[bot] committed Aug 8, 2023
2 parents 8c70e01 + 65577c7 commit 97fde53
Show file tree
Hide file tree
Showing 2 changed files with 100 additions and 1 deletion.
99 changes: 99 additions & 0 deletions docs/cross-project-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
---
title: Setting up cross-project testing
sidebar_position: 9
---
# Setting up cross-project testing
Cross-project testing enables you to ensure the compatibility and reliability of interconnected repositories.
And with a little bit of configuration on Packit and tmt parts, it can be easily automated within Packit!

Big kudos to [Martin Pitt](https://github.com/martinpitt/) who experimented with these settings in
[`cockpit` repositories](https://github.com/cockpit-project/) and wrote an amazing blog post about it!
If you are interested in all the details, make sure to read the
[full blog post](https://cockpit-project.org/blog/tmt-cross-project-testing.html). Alternatively, the following sections
will describe the needed configuration for setting it up.

After determining the dependency (or dependencies) you want to test, the configuration is relatively easy:
1. For the dependent repository, Packit should be configured to build every commit of the `main` branch, or any other
similar branch containing recent changes, into a dedicated Copr repository.
2. For the dependency repository, you should add a separate Packit test job with a tmt test plan to run on all pull requests.

Let's break down these steps using examples from the `cockpit` setup (where `cockpit-podman` depends on the `cockpit` repo)!

### Configuring Packit builds on commits of the dependent repository
The goal here is to build the most recent code (usually in the `main` branch) of the repo in a dedicated Copr repository,
which can be easily done using Packit.
Firstly a new dedicated [Copr](https://copr.fedorainfracloud.org) project needs to be created and configured as needed.
For Packit to build in this project, the `Packit allowed forge projects` needs to be filled as well.

Then you can configure a Packit `copr_build` job to do a build in the new Copr repo for each commit that lands in the desired
branch of the dependent repo (in this example `main` branch of `cockpit-podman`):
```yaml
- job: copr_build
trigger: commit
branch: "^main$"
owner: "@cockpit"
project: "main-builds"
preserve_project: True
```
You will need to approve the `packit` Copr user to be able to trigger builds in the Copr repo.

The additional details of the setup are described [here](/docs/configuration/upstream/copr_build#using-a-custom-copr-project).

### Configuring the test plan
After that, a new test plan `plans/<your-project>.fmf` needs to be added to the dependency repository, which selects the
tests you want to run from this repo. Unfortunately, it is impossible to auto-import all plans
at the moment (but this may be useful since you can select a subset of tests that apply to the tested dependency).

So this is mostly a copy of the dependent repo test plan, with one modification: they should not run by default,
but only in a `revdeps` tmt context. The test plan can then look like this (an example from `cockpit`):
```yaml
# reverse dependency test
enabled: false
adjust+:
when: revdeps == yes
enabled: true
discover:
how: fmf
url: https://github.com/cockpit-project/cockpit-podman
ref: "main"
execute:
how: tmt
/podman-system:
summary: Run cockpit-podman system tests
discover+:
test: /test/browser/system
# ... possibly more test plans here
```
For the other plans in this repository, you should mark it to not run in the `revdeps` scenario:
```yaml
adjust+:
when: revdeps == yes
enabled: false
```

### Configuring the Packit test job
Finally, to plug everything together, a Packit test job needs to be configured that will run the `revdeps` context test plan
against the builds from `main` branch of the dependent repo by specifying [`tf_extra_params`](https://packit.dev/docs/configuration/upstream/tests#optional-parameters).
The key here is to install the latest `main` build from the dependency: we are making sure we are not breaking `cockpit`.
Make sure to also give it a meaningful `identifier` so that you can differentiate the statuses:

```yaml
- job: tests
identifier: revdeps
trigger: pull_request
targets:
- fedora-latest-stable
tf_extra_params:
environments:
- artifacts:
- type: repository-file
id: https://copr.fedorainfracloud.org/coprs/g/cockpit/main-builds/repo/fedora-$releasever/group_cockpit-main-builds-fedora-$releasever.repo
tmt:
context:
revdeps: "yes"
```
2 changes: 1 addition & 1 deletion docs/fedora-releases-guide.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: How to do Fedora releases with Packit
sidebar_position: 9
sidebar_position: 8
---
# How to do Fedora releases with Packit

Expand Down

0 comments on commit 97fde53

Please sign in to comment.