android_interop CI is crashing

[ejona86]

It has been flaky for a while because of firebase infrastructure issues. That's "okay." But last week runs started failing because of crashes. But I don't see the crash in the device logs. It is random; it is impacting API 26 the most, but doesn't always fail.

Random example:

More details are available at [https://console.firebase.google.com/project/grpc-testing/testlab/histories/bh.6e2cfdde264f9eb3/matrices/6757898078985350132].
WARNING: Need help for test execution [Nexus6P-23-en-portrait]? Please join the #test-lab Slack channel at https://firebase.community/ and include execution ID [6757898078985350132] with your question.
+--------------+------------------------+-------------------------------+
|   OUTCOME    |    TEST_AXIS_VALUE     |          TEST_DETAILS         |
+--------------+------------------------+-------------------------------+
| Failed       | Nexus6-22-en-portrait  | 1 test cases failed           |
| Failed       | Nexus6P-26-en-portrait | 1 test cases failed, 1 passed |
| Passed       | Nexus6-21-en-portrait  | 2 test cases passed           |
| Passed       | Nexus6P-24-en-portrait | 2 test cases passed           |
| Passed       | Nexus6P-25-en-portrait | 2 test cases passed           |
| Passed       | Nexus6P-27-en-portrait | 2 test cases passed           |
| Inconclusive | Nexus6P-23-en-portrait | Infrastructure failure        |
+--------------+------------------------+-------------------------------+

[temawi]

One persistent issue is the use of stream() by Mockito:

java.lang.NoSuchMethodError: No interface method stream()Ljava/util/stream/Stream; in class Ljava/util/List; or its super classes (declaration of 'java.util.List' appears in /system/framework/core-libart.jar)
at org.mockito.internal.verification.DefaultRegisteredInvocations.getAll(DefaultRegisteredInvocations.java:46)
at org.mockito.internal.stubbing.InvocationContainerImpl.getInvocations(InvocationContainerImpl.java:139)
at org.mockito.internal.verification.VerificationDataImpl.getAllInvocations(VerificationDataImpl.java:31)
at org.mockito.internal.verification.Times.verify(Times.java:33)
at org.mockito.internal.verification.VerificationOverTimeImpl.verify(VerificationOverTimeImpl.java:87)
at org.mockito.internal.verification.VerificationWrapper.verify(VerificationWrapper.java:21)
at org.mockito.internal.verification.MockAwareVerificationMode.verify(MockAwareVerificationMode.java:30)
at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:75)
at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:56)
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:161)
at io.grpc.internal.ManagedClientTransport$Listener$MockitoMock$r8q6ua1c.transportReady(Unknown Source)
at io.grpc.internal.AbstractTransportTest.startTransport(AbstractTransportTest.java:2139)
at io.grpc.internal.AbstractTransportTest.authorityPropagation(AbstractTransportTest.java:964)

Streams are not available at API levels < 24 which explains that we see our tests for 21, 22 and 23 fail. The error first appeared on 7/21/23 when we upgraded the mockito-android library to version 4.11.0 in PR #10359.

The change to use stream was introduced to Mockito in mockito/mockito#2593 that was first released with Mockito 4.5.0

I will bring us down to Mockito 4.4.0 and see how that affects the situation.

[temawi]

I downgraded Mockito and so far I have not seen this problem in the last five test runs. So hopefully that aspect has been resolved.

I will monitor the situation with the timeout problem.