Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation and links review (close #12) #13

Merged
merged 3 commits into from
Jul 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 51 additions & 54 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,26 +7,24 @@

# QACover - SQL Query Aware Test Data Coverage Evaluation for Java and .Net applications

*QACover* is a component to evaluate the test data coverage in relation with the
*QACover* is a component to evaluate the test data coverage in relation to the
SQL queries that are executed in a Java or .NET application.
The coverage meassured according to the *SQL Full Predicate Coverage* (SQLFpc) criterion,
The coverage is measured according to the *SQL Full Predicate Coverage* (SQLFpc) criterion,
a variant of MCDC tailored to SQL queries.
The criterion determines the situations of interest (test coverage items)
to test a SQL query against the test database.
These situations are represented as a set of *Coverage Rules*.

Each time that the application executes a query, QACover intercepts the query execution,
generates and evaluates the coverage rules, storing the results in the local development environment.
generates and evaluates the coverage rules, and stores the results in the local development environment.

At the end of the tests you can get the summary and detail reports.
At the end of the tests you can get the summary and detail reports of test data coverage.
This is an example of the summary report of a test session:

![Example of a summary report](docs/image-index.png)

## Contents

- [QACover - SQL Query Aware Test Data Coverage Evaluation for Java and .Net applications](#qacover---sql-query-aware-test-data-coverage-evaluation-for-java-and-net-applications)
- [Contents](#contents)

- [Quick Start](#quick-start)
- [Example scenario](#example-scenario)
- [QACover Components](#qacover-components)
Expand All @@ -47,10 +45,10 @@ This is an example of the summary report of a test session:
Example for Java:

- Add the dependency
[`io.github.giis-uniovi:qacover-core`](https://img.shields.io/maven-central/v/io.github.giis-uniovi/qacover-core) to your pom.xml
- Copy to the root of the project the files
[`qacover-core`](https://central.sonatype.com/artifact/io.github.giis-uniovi/qacover-core) to your pom.xml
- Copy the files
[`qacover.properties`](qacover-core/qacover.properties) and
[`spy.properties`](qacover-core/spy.properties) from the `qacover-core` folder.
[`spy.properties`](qacover-core/spy.properties) from the `qacover-core` folder to the root of your project.
- Edit the connection string of your application and insert `:p6spy` afer `jdbc`
(eg. if your connection string is `jdbc:sqlite:./target/TestDB.db` it must become `jdbc:p6spy:sqlite:./target/TestDB.db`).
- Run your tests and wait to finish.
Expand All @@ -61,10 +59,10 @@ To generate an html report:
- Download the *the standalone reporter jar file*
[`qacover-model-<VERSION>-report.jar`](https://central.sonatype.com/artifact/io.github.giis-uniovi/qacover-model)
from Maven Central (go to Versions and then Browse the selected version to download).
- From the root of your workspace, run
- Run this command from the root of your workspace:
```bash
java -jar qacover-model-<VERSION>-report.jar target/qacover/rules target/qacover/reports
```
```
- Open the `index.html` that you will found in the `target/qacover/reports` folder.

If you find that the class names are not the ones at the interaction point that executes the query,
Expand All @@ -74,14 +72,13 @@ you will need to tweak the configuration to include some exclusions for their pa
## Example scenario

Folder with the test package [qacoversample](qacover-core/src/test/java/test4giis/qacoversample)
contains an example on how to use the coverage information to improve the test data and test cases
contains an example of how to use the coverage information to improve the test data and test cases
to reveal hidden bugs. It contains three sequential scenarios:

1. Execute test cases and let QACover to evaluate the coverage.
Test data was designed to cover a number of test situations that where manually determined
and test pass.
2. Use the coverge information to automatically determine uncovered situations
to complete the test data and test cases. This allows revealing two hidden faults.
1. Execute test cases and let QACover to evaluate the test coverage.
Test data was designed to cover a number of test situations that were manually determined. All tests pass.
2. Use the test coverage information to automatically determine uncovered situations
to complete the previous test data and the test cases. This allows revealing two hidden faults.
3. Final debug and fix.

## QACover Components
Expand All @@ -94,30 +91,30 @@ There are two different artifacts:
- [`qacover-core`](https://central.sonatype.com/artifact/io.github.giis-uniovi/qacover-core):
The main artifact to use as a a dependency in your application (as shown in the Quick Start).
- [`qacover-model`](https://central.sonatype.com/artifact/io.github.giis-uniovi/qacover-model):
It only includes the model and classes to do reporting and inspecting the ruels.
It only includes the model and classes to do reporting and to inspect the coverage rules.
Use it if you only need access to previously generated coverage rules (e.g. to generate reports from a program).

Each of them has another downloadable jar that includes additional qualifier:

- [`qacover-core` uber jar](https://central.sonatype.com/artifact/io.github.giis-uniovi/qacover-core).
It includes all needed dependencies (excluding `slf4j``) and they are *shaded*
It includes all needed dependencies (excluding `slf4j`) and they are *shaded*
(i.e. renamed in a different namespace to avoid dependency conflicts):
- Download the artifact with the `-uber` qualifier if for any reason you can't use it as a dependency in your application
- Download the artifact with the `-uber` qualifier if for any reason you cannot use it as a dependency in your application
(e.g. to deploy in an application server).
You simply need to put the jar in your server library and set the configuration to use QACover.
- Use the uber jar as a dependency declared in your pom.xml if you experiment conflicts with versions:
Add `<qualifier>uber</qualifier>` to the dependency declaration.
- [`qacover-model` standalone reporter](https://central.sonatype.com/artifact/io.github.giis-uniovi/qacover-model):
Download the artifact with the suffix `-report` to generate the reports from the commandline as shown in the quick start
Download the artifact with the `-report` qualifier to generate the reports from the command line as shown in the [quick start](#quick-start).

### .NET Packages in NuGet

Releases for .NET platform are published in NuGet. Same as for Java, there are two different packages:
Releases for .NET platform are published in NuGet. The same as for Java, there are two different packages:

- [`QACover`](https://www.nuget.org/packages/QACover/):
The main package (netstandard2.9) to nclude as a package reference in your project configuration (e.g. the .csproj file if you are using C#).
The main package (netstandard2.9) to include as a package reference in your project configuration (e.g. the .csproj file if you are using C#).
- [`QACoverReport`](https://www.nuget.org/packages/QACoverReport/):
A dotnet tool (netcore2.0) to generate the reports from the commandline:
A dotnet tool (netcore2.0) to generate the reports from the command line:
Install the tool with `dotnet tool install QACoverReport`
and execute it as a command `QACoverReport <rules folder> <reports folder>`.

Expand All @@ -131,26 +128,26 @@ On .NET you only need the first one along with some additional code to intercept
### QACover configuration file

QACover looks for the `qacover.properties` in this order:
- System properties
- The application classpath
- The default path where the applcation or the tests are executed.
- System properties.
- The application classpath.
- The default path where the application or the tests are executed.

The [`qacover.properties`](qacover-core/qacover.properties) available in the `qacover-core` module of this
repo contains a general configuration suitable for common scenarios, but sometimes it must be customized.
See the file for details on each configuration parameter. Next, we highligth the most important ones that are the
inclusion and exlusion criteria.
See the file for details on each configuration parameter. Next, we highlight the most important ones that are the
inclusion and exclusion criteria.

When a line of a method in your application executes a SQL query (*interaction point*),
a chain of calls to methods of your framework
is executed until the reaching driver methods that actually executes the query.
is executed until reaching the driver method that actually executes the query.
Here is the point in which the actual execution of the query is detected,
but what we want is to determine the interaction point at the application.
but what we want is to determine the interaction point in the application.
To achieve this, QACover checks the call stack at the point of the actual execution
and succesively excludes every call made in any framework package until it locates the point of the
and successively excludes every call made in any framework package until it locates the point of the
database interaction in your method.

QACover excludes the system packages like the java, System, P6Spy or the QACover packages, but depending on the framework
you must configure additional exclusions by setting the `qacover.stack.exclusions` property in `qacover.properties`.
you must configure additional exclusions by setting the `qacover.stack.exclusions` property in the file [`qacover.properties`](qacover-core/qacover.properties).

**Example**: Folder `it/spring-petclinic-main` contains a typical sample from Spring Boot.
The exclusion is declared as:
Expand Down Expand Up @@ -179,7 +176,7 @@ repo contains the minimal configuration required by P6Spy:
- Also, you may need to configure the formats for boolean, dates an times.

See the [`spy.properties`](qacover-core/spy.properties) file
or the [`P6Spy documentation`](https://p6spy.readthedocs.io/en/latest/configandusage.html) for more details
or the [`P6Spy documentation`](https://p6spy.readthedocs.io/en/latest/configandusage.html) for more details.

### Configuration for .NET

Expand All @@ -200,7 +197,7 @@ evaluation results.

In addition to standard logs, other folders `log-*` are created in the `rules` folder
to display additional debug information about the queries that are evaluated, the database schema,
and the SQLFpc rules.
and the coverage rules.

## Reporting

Expand All @@ -211,7 +208,7 @@ to easily inspect summary and details of the coverage data.

To generate reports you have three options:

- From the commandline: Download the
- From the command line: Download the
[`qacover-model` standalone reporter](https://central.sonatype.com/artifact/io.github.giis-uniovi/qacover-model)
as shown in the quick start and execute:
```bash
Expand Down Expand Up @@ -251,32 +248,32 @@ To generate reports you have three options:

### Content of the reports

The `index.html` file contains the summary of coverage for each class:
The `index.html` file contains the summary of test data coverage for each class:

![Example of a summary report](docs/image-index.png)
where:
- %: Total percent coverage (number of rules covered divided by total number of rules)
- qrun: total number of query evaluations
- qcount: number of different queries that have been evaluated
- qerror: number of queries that have not been evaluated because some error
- dead: number of rules covered
- count: number of rules generated
- error: number of rules that have not been evaluated because some error
- %: Total percent of coverage (number of coverage rules covered divided by total number of coverage rules generated).
- qrun: total number of query evaluations.
- qcount: number of different queries that have been evaluated.
- qerror: number of queries that have not been evaluated because some error.
- dead: number of coverage rules covered.
- count: number of coverage rules generated.
- error: number of coverage rules that have not been evaluated because some error.

Class name is clickable to display a report that contains the details for queries that have been evaluated,
identified by the method name and line number:

![Example of a summary report](docs/image-class.png)

Method name is clickable to show the detail of each coverage rule (covered in green, uncovered in yellow):
- A textual message that explains the situation that the rule represents.
If a rule is not covered, a test and/or the appropriate test data may be added in order to cover it.
- The SQL representation of the rule
- A textual message that explains the test situation that the coverage rule represents.
If a coverage rule is not covered, a test and/or the appropriate test data may be added in order to cover it.
- The SQL representation of the coverage rule.
- Additional indicators
- Sequential ID
- dead: number of times that the rule has been covered
- count: number of times that the rule has been executed
- category, type, subtype, location: A classificacion about where the rule comes from.
- Sequential ID.
- dead: number of times that the coverage rule has been covered.
- count: number of times that the coverage rule has been executed.
- category, type, subtype, location: A classification about where the coverage rule comes from.

## Contributing and Architecture

Expand All @@ -288,10 +285,10 @@ Now we include some additional background technical information:
QACover makes use of
[p6spy](https://github.com/p6spy/p6spy) to intercept the jdbc calls,
[TdRules](https://github.com/giis-uniovi/tdrules) to get the database schema and invoke the
[SQLRules Service](https://in2test.lsi.uniovi.es/sqlrules/) to generate the Coverage Rules.
[SQLRules Service](https://in2test.lsi.uniovi.es/sqlrules/) to generate the coverage rules.
The execution of everything is made in local against the database configured in the connection string.

The internal structure of the main QACover packages (prefix `giis.qacover.`) is shown below (the prefixes are ommitted for simplicity):
The internal structure of the main QACover packages (prefix `giis.qacover.`) is shown below (the prefixes are omitted for simplicity):
- **`core` module**: Contains the `driver`, `core` and `core.sevices` packages.
- **`model` module**: Contains the `model`, `storage`, `reader` and `report` packages.

Expand Down
2 changes: 1 addition & 1 deletion qacover-core/qacover.properties
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
# qacover.store.root=.

# Folder to store the generated rules (relative to the root folder)
# Default: target/qacover/reports (java), reports/qacover/reports (.NET)
# Default: target/qacover/rules (java), reports/qacover/rules (.NET)
# qacover.store.rules=target/qacover/rules

# Folder to store the generated reports (relative to the root folder)
Expand Down
Loading