Enhancement- add skip count, total and passing % to karate summary HTML report

[rc2201]

We are using karate.abort() extensively in our project to control the execution of scenarios depending upon the test data pre-requirement, environment limitations etc. Along with this,our runner executes multiple feature files.

Below enhancement to karate summary report would greatly improve our experience with the reporting-

1. Adding the count of skipped scenario.
2. In case of multiple feature files being executed, adding total# to provide total of passed / failed / skipped scenarios. This is already provided on the console.
3. % of passing scenarios.

Attaching the screenshot for refrence.

Hopefully, other karate users will also find these features useful as well.

[Jasmine-maryj]

I'd like to work on this!

[ptrthomas]

@Jasmine-maryj sure go ahead, here is the Hacktoberfest process we follow: https://github.com/karatelabs/karate/wiki/Hacktoberfest

[ptrthomas]

some more requests to consider: https://stackoverflow.com/q/77664133/143475

[Mike-c-Jackson]

Thanks Peter, I think these would be helpful additions

[codehackerr]

@ptrthomas this is open for a while. I would like to work on this.

[ptrthomas]

@codehackerr sure, please proceed with a PR

[codehackerr]

@ptrthomas

Approach to fixing this issue:

1. Adding Skipped Count: Whenever a scenario is skipped in ScenarioRuntime, skippedCount variable is incremented on FeatureResult. This is later used in the feature-summary report.

2. Change to Totals calculation on a Feature:

• As of now Scenario Total = Scenario Success + Scenario Failed.
• With the introduction of Skipped Scenarios, new way to calculate total is Scenario Total = Scenario Success + Scenario Failed + Scenario Skipped.
• Existing Scenario Total will be refactored to Scenario Run Total, such that Scenario Run Total = Scenario Success + Scenario Failed

3. Totals Row: Will use Results class and add skippedScenarios to this. Will add a totals row in feature-summary.html
4. Percentage Calculations:
   Row: ( Passed Scenarios Feature / Total Scenarios In Feature) * 100 - Rounded to whole number
   Total: ( Passed Scenarios across All Features / Total Scenarios across all Features) * 100 - Rounded to whole number

Proposed Summary Report Screenshot:

Kindly review and suggest any modifications. If approach is okay, I can push a PR soon. Thanks!

[codehackerr]

I am getting a 403 on branch push.
Permission to karatelabs/karate.git denied to codehackerr.
I am able to push to other repos, so authentication is okay.
Is there any access grants required ?

[ptrthomas]

@codehackerr please refer to the typical PR process: https://github.com/karatelabs/karate/blob/master/.github/CONTRIBUTING.md

[shassankani]

I am also waiting for the Karate summary report enhancement, for a while. Good to see progress on this issue. When will this be released?

[Mike-c-Jackson]

Will the sticky headers or frames be included?

…

On Tue, Jan 16, 2024, 2:56 PM shassankani ***@***.***> wrote: I am also waiting for the Karate summary report enhancement, for a while. Good to see progress on this issue. When will this be released? — Reply to this email directly, view it on GitHub <#2340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6CKQUIDDMAQOPFHXOKTUSLYO3ZQPAVCNFSM6AAAAAAZEAILT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJUGU3TQNZXGE> . You are receiving this because you commented.Message ID: ***@***.***>

[codehackerr]

@Mike-c-Jackson From what I understand, not in the scope of this issue. Could you link me to existing feature requests/issues or create one with the details?

[Mike-c-Jackson]

This was mentioned in the linked post that @ptrthomas added on Dev 14th. That is the 4th post in this chain

[ptrthomas]

this has been merged to develop

[rc2201]

I tested the changes. changes for pass % and total are working fine. But i was not able to verify skipped scenario count. Here is the use case which was tried and mentioned-

1. Exited the scenarios / features using the karate.abort() function.
2. But those scenario are still being counted as passed in the report.

Even though it should not matter, i have tried with abortedStepsShouldPass value as true and false.

@codehackerr what was the criteria to count a scenario as skipped?

[ptrthomas]

@rc2201 - please if this is is not resolved soon, we will be undo-ing all related changes, no other team has requested this. I don't think karate.abort() should be used extensively. anyone is welcome to open a new discussion to discuss if any fundamental changes need to be made to karate.

trying to put conditional logic in tests is an anti-pattern, but I'm open to exploring if testing of complex business-rules can be made easier - but I'd like to understand the context first

[ptrthomas]

all: to be clear why we are considering to undo all related changes is because we have a report of breaking changes to existing users of karate, refer: #2477 (comment)

especially as this is a cosmetic request and not something we see many teams ask for - we'll play it safe and undo these changes. anyone is welcome to submit a PR to fix this and get confirmation that there are no breaking changes to existing users - but I don't consider this a priority

[rc2201]

@ptrthomas i send an email to you with context around this enhancement request.

[ptrthomas]

changes have been reverted, closing as wontfix