Native Testing: Google test failure output does not capture custom diagnostics written to result_listener

[ryan-holt-1]

We have a custom GTest Matcher called containsTokens. It does the following:

#include <string>
#include <sstream>

std::string stripWhitespace(const std::string& aString) {
    std::ostringstream tmp;
    auto i = aString.begin();
    auto end = aString.end();
    while (i != end) {
        if (!isspace(*i)) {
            tmp << *i;
        }
        ++i;
    }
    return tmp.str();
}

MATCHER_P(containsTokens, aSubstring, "") {
    auto findMe = stripWhitespace(aSubstring);
    auto withinMe = stripWhitespace(arg);
    if (withinMe.find(findMe) != std::string::npos) {
        return true;
    }
    *result_listener << "\n\n" << aSubstring << "\n\nnot found within\n\n" << arg;
    return false;
}

Note that we append additional failure diagnostics to the result_listener.

which is used as follows:

TEST(MySuite, MyTest) {
    auto input =
        R"(
            func foo() {
                x = 4;
            }
         )";

    auto expected =
        R"(
    func foo() {
        x = 5;
    }
    )";

    ASSERT_THAT(input, containsTokens(expected));
}

This test will fail with the following message (just running the test binary):

[ RUN ] MySuite.MyTest
test/unittest/tTest.cpp:119: Failure
Value of: input
Expected: contains tokens "\n func foo() {\n x = 5;\n }\n "
Actual: 0x5d04e0 pointing to "\n func foo() {\n x = 4;\n }\n " (of type char const*),

func foo() {
    x = 5;
}

not found within

        func foo() {
            x = 4;
        }

[ FAILED ] MySuite.MyTest (0 ms)

However, if I run it in C++ TestMate, It prints the following:

# Failure:
Value of: input
Expected: contains tokens "\n func foo() {\n x = 5;\n }\n "
Actual: 0x5d04e0 pointing to "\n func foo() {\n x = 4;\n }\n " (of type char const*),

Note that C++ TestMate does not print out the extra diagnostics that we wrote to the result_listener. This used to work before the switch to Native Testing. I'm not sure if this is a bug with TestMate or VSCode.

[matepek]

please add the definition of stripWhitepace

[ryan-holt-1]

Added to my original post.

Though, the easiest way to reproduce would be to just do:

MATCHER_P(containsTokens, aSubstring, "") {
    *result_listener << "\nPRINT ME\n";
    // Make the test always fail
    return false;
}

TEST(MySuite, MyTest) {
    auto input =
        R"(
            func foo() {
                x = 4;
            }
         )";

    auto expected =
        R"(
    func foo() {
        x = 5;
    }
    )";

    ASSERT_THAT(input, containsTokens(expected));
}

The test will always fail and you should see "PRINT ME" in the output. However, you won't see it when running with TestMate.

[matepek]

Fixed in **v4.0.25**.

This issue was mentioned in CHANGELOG.md under a released entry so it is assumed to be fixed.
User verifications are always welcome.

[ryan-holt-1]

Thanks for working on this. However, I updated to the latest version and still have the same problem. I also still have the same problem as #329 . I do not see cout/cerr messages in the output.

I'm on Linux (Debian 10) if that matters.

[mkuehn94]

I'm having a very similar problem as in #329 stderr output are shown in terminal when running manually but not in the extension. however stdout output's work as expected.

I'm on windows sadly (windows 10)