Error message when writing SCollection of size 0 to hdfs sink.

[andrewsmartin]

SDK throws a FileNotFoundException when HadoopWriteOperation tries to finalize, expecting temporary files to be there in all cases.

May 27, 2016, 3:05:37 PM
(430f73e8aa5bd9ce): java.io.FileNotFoundException: File hdfs://[hdfs-path]/_temporary/0 does not exist. at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795) at 
org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106) at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853) at 
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849) at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860) at 
org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515) at 
org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555) at 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:283) at 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:325) at 
com.google.cloud.dataflow.contrib.hadoop.HadoopFileSink$HadoopWriteOperation.finalize(HadoopFileSink.java:132) at 
com.google.cloud.dataflow.contrib.hadoop.simpleauth.SimpleAuthHadoopFileSink$SimpleAuthHadoopWriteOperation._finalize(SimpleAuthHadoopFileSink.java:74) at 
com.google.cloud.dataflow.contrib.hadoop.simpleauth.SimpleAuthHadoopFileSink$SimpleAuthHadoopWriteOperation.access$000(SimpleAuthHadoopFileSink.java:51) at 
com.google.cloud.dataflow.contrib.hadoop.simpleauth.SimpleAuthHadoopFileSink$SimpleAuthHadoopWriteOperation$1.run(SimpleAuthHadoopFileSink.java:67) at 
com.google.cloud.dataflow.contrib.hadoop.simpleauth.SimpleAuthHadoopFileSink$SimpleAuthHadoopWriteOperation$1.run(SimpleAuthHadoopFileSink.java:64)

[andrewsmartin]

Should handle this gracefully without crashing - this should be considered a normal circumstance