Fix google-cloud-java on GAE Std Java8 Issues

[vam-google]

Fix the issues, found during the testing performed in #1752.

Current List of Issues:

1. CGLib: https://gist.github.com/vam-google/862b1dc2c1badf0e675ef04a552ac872#file-gae_std_java8_test_failure___cglib-txt
2. EasyMock verify not allowed: https://gist.github.com/vam-google/862b1dc2c1badf0e675ef04a552ac872#file-gae_std_java8_test_failure___easymock_verify_not_allowed-txt
3. System Prooperty: https://gist.github.com/vam-google/862b1dc2c1badf0e675ef04a552ac872#file-gae_std_java8_test_failure___env_variable-txt
4. Wierd NoClassDefFound: https://gist.github.com/vam-google/862b1dc2c1badf0e675ef04a552ac872#file-gae_std_java8_test_failure___noclassdeffounderror_serialization_test-txt
5. ThreadManager: https://gist.github.com/vam-google/862b1dc2c1badf0e675ef04a552ac872#file-gae_std_java8_test_failure___threadmanager-txt

[chadbr]

Is there any way to track what's going on with these?

We're trying to use both the PubSub and Storage clients from Flex in our environment and both suffer from the ThreadManager issue.

Thanks

[vam-google]

This is weird. The reported ThreadManager issue is for Java8 Standard, not for Java Flex. Can you please post the stacktrace of the exception your receive? Is it similar to https://gist.github.com/vam-google/862b1dc2c1badf0e675ef04a552ac872#file-gae_std_java8_test_failure___threadmanager-txt?

[chadbr]

yes - very similar:

From: Chad Brockman chadbr@slb.com
Date: Wednesday, February 15, 2017 at 7:01 PM
To: "java8-on-gae@googlegroups.com" java8-on-gae@googlegroups.com
Subject: PubSub authentication failure?

One of our cron based services is failing with PubSub authentication failures… It looks like there is a missing native call for memcache?

Can't make API call memcache.Get in a thread that is neither the original request thread nor a thread created by ThreadManager at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:836)

Any ideas welcome – Thanks, Chad

Callstack from logging…

18:45:03.323com.slb.rtfrac.ingestion.servlets.InteractDownloadServlet lambda$invoke$0: Stage[InteractDownload]: WellId[5649391675244544] Update failed with an exception io.grpc.StatusRuntimeException: UNAUTHENTICATED com.google.cloud.pubsub.PubSubException: io.grpc.StatusRuntimeException: UNAUTHENTICATED at com.google.cloud.pubsub.spi.DefaultPubSubRpc$1.apply(DefaultPubSubRpc.java:174) at com.google.cloud.pubsub.spi.DefaultPubSubRpc$1.apply(DefaultPubSubRpc.java:168) at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:205) at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:193) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:132) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53) at com.google.api.gax.grpc.BundlingFuture.setException(BundlingFuture.java:87) at com.google.api.gax.grpc.BundlingContext.sendResult(BundlingContext.java:97) at com.google.api.gax.grpc.BundleExecutor.processBundle(BundleExecutor.java:92) at com.google.api.gax.bundling.ThresholdBundlingForwarder$BundleForwardingRunnable.processBundle(ThresholdBundlingForwarder.java:110) at com.google.api.gax.bundling.ThresholdBundlingForwarder$BundleForwardingRunnable.run(ThresholdBundlingForwarder.java:93) at java.lang.Thread.run(Thread.java:745) Caused by: com.google.api.gax.grpc.ApiException: io.grpc.StatusRuntimeException: UNAUTHENTICATED at com.google.api.gax.grpc.ExceptionTransformingCallable$1.onFailure(ExceptionTransformingCallable.java:91) at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:634) at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1135) at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1073) at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66) at com.google.api.gax.grpc.RetryingCallable$Retryer.run(RetryingCallable.java:132) at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:87) at com.google.api.gax.grpc.ChannelBindingCallable.futureCall(ChannelBindingCallable.java:64) at com.google.api.gax.grpc.ApiCallable.futureCall(ApiCallable.java:239) at com.google.api.gax.grpc.ApiCallable.futureCall(ApiCallable.java:250) at com.google.api.gax.grpc.ApiCallable.call(ApiCallable.java:287) at com.google.api.gax.grpc.BundleExecutor.processBundle(BundleExecutor.java:85) ... 3 more Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED at io.grpc.Status.asRuntimeException(Status.java:545) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:417) at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:203) at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47) at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:35) at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:248) at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:227) at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:186) at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58) at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:65) ... 10 more Caused by: java.io.IOException: Could not get the access token. at com.google.cloud.AuthCredentials$AppEngineAuthCredentials$AppEngineCredentials.refreshAccessToken(AuthCredentials.java:117) at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149) at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:135) at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150) at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64) at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96) at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195) ... 17 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.google.cloud.AuthCredentials$AppEngineAuthCredentials$AppEngineCredentials.refreshAccessToken(AuthCredentials.java:112) ... 23 more Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call memcache.Get in a thread that is neither the original request thread nor a thread created by ThreadManager at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:836) at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:183) at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:180) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:93) at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28) at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java:51) at com.google.appengine.api.appidentity.AppIdentityServiceImpl.getAccessToken(AppIdentityServiceImpl.java:300) ... 28 more

[chadbr]

@vam-google sorry for the delay - I missed the notification...

[ludoch]

The google Cloud APIs, when running in Flex (or Java8 Standard) should try to not rely on com.google.appengine.api.appidentity* APIs.
But they do this here:
https://github.com/GoogleCloudPlatform/google-cloud-java/blob/1c456ead3455680ef198900712ff481d634ec530/google-cloud-core/src/main/java/com/google/cloud/ServiceOptions.java#L384

How can we avoid this code path when running in Java8 Standard?

[lesv]

We are fixing this stuff - I wrote issues on GH issues last week about. :(

…

On Wed, May 10, 2017 at 10:26 AM, Ludovic Champenois < ***@***.***> wrote: The google Cloud APIs, when running in Flex (or Java8 Standard) should try to not rely on com.google.appengine.api.appidentity* APIs. But they do this here: https://github.com/GoogleCloudPlatform/google-cloud-java/blob/ 1c456ea/google-cloud-core/ src/main/java/com/google/cloud/ServiceOptions.java#L384 How can we avoid this code path when running in Java8 Standard? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1918 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABFVjkHidykrLuNjXOBb0v_fOtSu2gMXks5r4fNYgaJpZM4M9OI9> .

-- * • **Les Vogel* * • *Cloud Developer Relations * • *lesv@google.com * • *+1-4 <%2B1-650-338-7103>08-676-7023

[lesv]

Oops - wrong one. I found these in java-docs-samples last week.

…

On Thu, May 11, 2017 at 2:53 PM, Les Vogel ***@***.***> wrote: We are fixing this stuff - I wrote issues on GH issues last week about. :( On Wed, May 10, 2017 at 10:26 AM, Ludovic Champenois < ***@***.***> wrote: > The google Cloud APIs, when running in Flex (or Java8 Standard) should > try to not rely on com.google.appengine.api.appidentity* APIs. > But they do this here: > https://github.com/GoogleCloudPlatform/google-cloud-java/ > blob/1c456ead3455680ef198900712ff481d634ec530/google-cloud- > core/src/main/java/com/google/cloud/ServiceOptions.java#L384 > > How can we avoid this code path when running in Java8 Standard? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#1918 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABFVjkHidykrLuNjXOBb0v_fOtSu2gMXks5r4fNYgaJpZM4M9OI9> > . > -- * • **Les Vogel* * • *Cloud Developer Relations * • ***@***.*** * • *+1-4 <%2B1-650-338-7103>08-676-7023

-- * • **Les Vogel* * • *Cloud Developer Relations * • *lesv@google.com * • *+1-4 <%2B1-650-338-7103>08-676-7023

[ludoch]

BTW, Java8 Standard runtime will have the metadata server APIs to get the project ID soon...
This may help this issue for Java8 Standard and avoid access to the appidentity API (only for Java8 Standard?)

[neozwu]

@ludoch before metadata server APIs are ready, is there any other way / workaround to expose project ID on GAE ?

[ludoch]

The only things we really expose via sys properties are:

com.google.appengine.application.id=abcdef
com.google.appengine.application.version=abcdef.397109454210762736
appengine.mail.filenamePreventsInlining=true
appengine.mail.supportExtendedAttachmentEncodings=true
com.google.appengine.runtime.environment=Production

where abcdef is the App Engine Application Name.

The Metadata server will be there really soon now (couple of weeks), but only for Java8 Standard.

[ludoch]

You can track internally b/37168510

[lesv]

Can you retest - there have been some changes that should make things work better.

[neozwu]

For 1) and 2), they are due to issues in objenesis. A new version of objenesis should fix this issue.
3), 4) and 5) are gone on GAE std java 8.

[lesv]

Objenesis 2.6 has been released.

[garrettjonesgoogle]

Filed a separate issue to upgrade to objenesis 2.6: #2172

[lesv]

Ping - I received an email from another customer about this. Please consider this a P0.

[lesv]

Strike my last comment and I'll move it to the pubsub issue.

[neozwu]

The issues listed here are resolved, so close this issue.

[elmer-garduno]

QQ, how does the error on #1918 (comment) gets fixed?

There's an other report of the same issue here:
https://stackoverflow.com/questions/45019692/not-able-to-use-cache-in-google-app-engine-getting-com-google-apphosting-api-a

[neozwu]

It seems to me those errors are due to the fact appengine APIs were used. For flex and j8 standard, one shouldn't rely on those APIs anymore. J8 standard supports native I/O and threading.

[elmer-garduno]

Thanks for the prompt response, the problem I'm having right now is that in addition to my google-cloud-java dependencies (com.google.cloud.bigtable/bigtable-hbase-1.2) I also depend on the Google client APIs (com.google.apis/google-api-services-ml) which requires AppIdentityCredential from the appengine APIs.

Any insights on how to handle this?

[elmer-garduno]

I see the old API is in maintenance mode, are there guidelines on how to use the google-cloud-java APIs to invoke non built-in APIs like CloudML? (Basically to get the authentication and basic REST support in place)?