[XML] Failed test cases now take precedence during submission

[jazerix]

In JUnit XML files where multiple test_case ids overlap, TestRail CLI currently reports both of them. This behavior results in only the last being "persisted" on the platform, which is especially problematic, when the first test_id fails and subsequent ones succeeds, as the platform now reports the test as being successful.

In instances where testcase ids collide, TestRail should opt for a pessimistic approach where the negative outcome takes the highest priority.

Issue being resolved: #161

Solution description

During upload, data_provider/api_data_provider.py now checks if another tests case is to be submitted with the same case_id , if not the test case is added to the list of `bodies.

If the code encounters the same case id, one of two things happen:

1. The current case has a status 5 (failed), subsequent cases are ignored as the outcome is a failure.
2. The current case is not a status 5, subsequent cases are ignored unless a failure is found in which case it overwrites the request for the case id.

This is a very simplistic approach that only takes the failure status code into account.

Changes

A takes_priority(self, statusId1, statusId2) method has been added that returns true or false if the first statusId takes priority (is a failure):

def takes_priority(self, statusId1, statusId2):
        """Determines if the first argument takes precendece in cases where an earlier result has failed (5).
        """
        return (statusId1 is 5 and statusId2 is not 5)

This method definitely leaves a lot to be desired, and it could be worth looking into creating a more sophisticated way of ordering the different status codes.

Appending a result body has been refactored and extracted to its own method: append_result_body(self, bodies, case)

def append_result_body(self, bodies, case):
        """Appends result to bodies. Overlapping case ids are ignored, unless a case has status of 5, in which case
        it overwrites the body."""
        case.result.add_global_result_fields(self.result_fields)

        overlapping_case_index = next(iter([i for i, body in enumerate(bodies) if body['case_id'] == case.case_id]), None)
        if overlapping_case_index is None:
            bodies.append(case.result.to_dict())
            return;

        if self.takes_priority(case.result.status_id, bodies[overlapping_case_index]['status_id']):
            bodies[overlapping_case_index] = case.result.to_dict();

It is probably also worth noting that this implementation also does not take into account overlapping test_ids from different files.

Potential impacts

This change specifically impacts instances where JUnit xml files have repetitions of the same test_id. However, it should not have any impact on other instances.

All tests pass.

Steps to test

• Create a JUnit xml file
• Create two test cases, one where the first one fails and the second succeeds

PR Tasks

• PR reference added to issue
• README updated
• Unit tests added/updated

[jazerix]

I have added a test case for this scenario :)

[Testinator-X]

@bitcoder will this be merged?

[bitcoder]

@Testinator-X I don't know at this point. Needs to be analyzed as other pending issues. I don't have any ETA for it, sorry

[bitcoder]

@jazerix ,
would the originating JUnit XML leading to this issue be something like this?

<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="3049" name="tests.LoginTests">
    <properties>
      <property name="setting1" value="True"/>
      <property name="setting2" value="value2"/>
    </properties>
    <testcase classname="tests.LoginTests" name="dummy" time="159">
      <skipped type="pytest.skip" message="Please skip">skipped by user</skipped>
    </testcase>
    <testcase classname="tests.LoginTests" name="dummy" time="121">
      <failure type="pytest.failure" message="Fail due to...">failed due to...</failure>
    </testcase>
    <testcase classname="tests.LoginTests" name="dummy" time="650">
    </testcase>
  </testsuite>
</testsuites>

[jazerix]

@bitcoder, yes exactly, whenever the classname would overlap :)

[kiebak3r]

is there an ETA on this merge?

[bitcoder]

is there an ETA on this merge?

Hi everyone, and happy new year. No, currently there's still no ETA.