Calling _io().close() on ByteBufferKaitaiStream leaves open file descriptors

[brunchboy]

In the Java runtime, once a Kaitai Struct has been opened using fromFile() with the default stream implementation generated by the compiler, it is impossible to properly close all open file descriptors using the API. lsof on the Java process shows two file descriptors being opened by fromFile(), and only one of them gets closed when you call _io().close(). This is a problem because it prevents removable media from which structures have been read from being ejected until the entire process is terminated, and that is a common desirable use case for me.

[brunchboy]

Explicitly constructing objects using RandomAccessFileKaitaiStream does not exhibit this problem.

[GreyCat]

I did some examination and this is the result: kaitai-io/kaitai_struct_java_runtime@83ee786

I don't think we can do more at this point. To summarize, it's a defiency of Java APIs, which does not allow us to explicitly "close" mmap, thus removing mmap. Current solution will allow you to "close" mmap-backed KaitaiStream by invoking close() and then running System.gc() manually.

I've tested in a simple application. Before this change, I was getting these in lsof:

# Before close()
java    4082 greycat  mem    REG                8,2      138 29142950 /path/to/file
java    4082 greycat   23r   REG                8,2      138 29142950 /path/to/file
# After close()
java    4082 greycat  mem    REG                8,2      138 29142950 /path/to/file

After this change + System.gc(), I'm getting more or less consistent closure:

# Before close()
java    4685 greycat  mem    REG                8,2      138 29142950 /path/to/file
java    4685 greycat   23r   REG                8,2      138 29142950 /path/to/file
# After close()

For historical purposes, this is the lsof invocation line I've been using:

MYPID=$(ps awwx | grep testfileclose.Main | grep -v grep | cut -c1-5); MYPID=$(($MYPID)); lsof -n -p $MYPID | grep /path/to/file

[brunchboy]

Thanks for the explanation and link to the JDK bug discussion. What an ugly mess! This does perfectly explain some of the exceptions one of my Windows users was encountering as well.

I agree that this is the best that the Kaitai project can do until the underlying Java issue is fixed. The hack-arounds that some people were suggesting are a terrible idea.

Since System.gc() is not guaranteed to do anything, from my perspective this means that ByteBufferKaitaiStream simply cannot be used in Java until Java is fixed. Thankfully, you do offer RandomAccessFileKaitaiStream which I am now successfully using.