[Java][CI] Enable JDK 21

[danepitkin]

Describe the enhancement requested

Java 21 is the next long-term support (LTS) version after Java 17 and is scheduled to be released in September 2023.

Component(s)

Java

[jarohen]

I was getting Unhandled java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

Thankfully fixed in #35053 before I got here (but not yet released).

[raulcd]

@davisusanibar @danepitkin @lidavidm what is the status of this one? Is this a release blocker?

[danepitkin]

We can push it to Arrow v15. It would have been really nice to verify Java 21 in CI, but the docker image from Eclipse Temurin is still not ready.

[danepitkin]

Actually, Eclipse Temurin JDK is starting to be published now. Docker images will take a little bit longer, but maybe it will be ready in time! 🤞

[Midhunpottammal]

while enabling arrow with spark in java 21 unhandled java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available . is this issue fixed or how to proceed

[raulcd]

@Midhunpottammal can you share more details? which version of Arrow and spark are you using and the stack trace.

[Midhunpottammal]

@Midhunpottammal can you share more details? which version of Arrow and spark are you using and the stack trace.
@raulcd
java -version

java version "21.0.2" 2024-01-16 LTS
Java(TM) SE Runtime Environment (build 21.0.2+13-LTS-58)
Java HotSpot(TM) 64-Bit Server VM (build 21.0.2+13-LTS-58, mixed mode, sharing)

Spark - Version

pyspark==3.5.0

pyarrow -Version

pyarrow 15.0.0

PySpark code

import time
import pandas as pd
from pyspark.sql import SparkSession
extra_java_options = os.getenv("SPARK_EXECUTOR_EXTRA_JAVA_OPTIONS", "")
spark = SparkSession.builder \
    .appName("ArrowPySparkExample") \
    .getOrCreate()
spark.conf.set("Dio.netty.tryReflectionSetAccessible", "true")
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
pdf = pd.DataFrame(["midhun"])
df = spark.createDataFrame(pdf)
result_pdf = df.select("*").toPandas()

Error Log
TaskContextImpl: Error in TaskCompletionListener
java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (49152)
Allocator(toArrowBatchIterator) 0/49152/49152/9223372036854775807 (res/actual/peak/limit)

at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:476)
at org.apache.spark.sql.execution.arrow.ArrowConverters$ArrowBatchIterator.$anonfun$new$2(ArrowConverters.scala:97)
at org.apache.spark.sql.execution.arrow.ArrowConverters$ArrowBatchIterator.$anonfun$new$2$adapted(ArrowConverters.scala:95)
at org.apache.spark.TaskContext$$anon$1.onTaskCompletion(TaskContext.scala:132)
at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1(TaskContextImpl.scala:144)
at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1$adapted(TaskContextImpl.scala:144)
at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:199)
at org.apache.spark.TaskContextImpl.invokeTaskCompletionListeners(TaskContextImpl.scala:144)
at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:137)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:172)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

24/03/05 14:02:59 ERROR Executor: Exception in task 11.0 in stage 0.0 (TID 11)
java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available
at org.apache.arrow.memory.util.MemoryUtil.directBuffer(MemoryUtil.java:174)
at org.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:229)

Full Stack Trace link

link

[iskandari]

Facing the same issue:

pyarrow 15.0.0
pyspark 3.5.0
java = openjdk/21.0.1

java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

[danepitkin]

I don't think you can set spark.conf.set("Dio.netty.tryReflectionSetAccessible", "true") at runtime here. Try setting this configuration option before you run pyspark.

[Midhunpottammal]

@iskandari @danepitkin @raulcd I managed to get Arrow working with a lower version of Java in Spark 3.5.0. Here's my stack:

pyarrow==15.0.0
pyspark==3.5.0
java == Java(TM) SE Runtime Environment (build 17.0.10+11-LTS-240)

When I try to move to Java version 21, I encounter the same error

[danepitkin]

It turns out Spark 3.X does not support Java 21, but Spark 4.0 does. Resolved issue here #40287