Metadata-prefetch (aka Recursive listing or ls -R during mount)

[gargnitingoogle]

Description

Metadata-prefetch (aka Recursive listing or ls -R during mount)

Metadata-prefetch is equivalent to running ls -R
on the mount-directory to prime/prefill/prefetch the
metadata caches (both type and stat).

This change adds support for mountconfig-flag metadata-prefetch-on-mount
in config-file. The type of value passed is a string.
The supported values are: (a) disabled, (b) sync, (c)
async.
If not set, the value disabled is taken by default.

If value disabled is set, then no difference from the existing behaviour.

If value sync is set, then mount is blocked
until the metadata prefetch is complete. In this mode, if
the metadata-prefetch fails, the mount itself fails.

If value is set to async, then mount is not
blocked, but metadata prefetch is
initiated in a go-routine and will complete sometime
later. In this case, if the metadata-prefetch fails,
then an error is logged in the GCSFuse log-file.

Link to the issue in case of a bug fix.

NA

Testing details

1. Manual - Done
   • Logs with metadata-prefetch-on-mount: disabled or with metadata-prefetch-on-mount not set:

      message="Mounting file system \"gargnitin-fuse-test-bucket1\"..."
      message="File system has been successfully mounted."

   • Logs with metadata-prefetch-on-mount: sync:

      message="Mounting file system \"gargnitin-fuse-test-bucket1\"..."
      message="Started recursive metadata-prefetch of directory:  \"blah/blah/blah\" ..."
      message="... Completed recursive metadata-prefetch of directory: \"blah/blah/blah\". Number of items discovered: 61"
      message="File system has been successfully mounted."

   • Logs with metadata-prefetch-on-mount: async:

      message="Mounting file system \"gargnitin-fuse-test-bucket1\"..."
      message="File system has been successfully mounted."
      message="Started recursive metadata-prefetch of directory:  \"blah/blah/blah\" ..."

     Here, control comes back to the user. Later log, after some time,

       message="... Completed recursive metadata-prefetch of directory: \"blah/blah/blah\". Number of items discovered: 61"

2. Unit tests - Ran locally and all passed.
3. Integration tests - Ran through presubmit.

[codecov]

Codecov Report

Attention: Patch coverage is 55.55556% with 28 lines in your changes are missing coverage. Please review.

Project coverage is 60.76%. Comparing base (1d0f0c0) to head (e554469).

Files	Patch %	Lines
main.go	37.50%	25 Missing ⚠️
flags.go	95.23%	1 Missing ⚠️
mount.go	0.00%	1 Missing ⚠️
tools/mount_gcsfuse/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1930      +/-   ##
==========================================
- Coverage   61.25%   60.76%   -0.50%     
==========================================
  Files         130      130              
  Lines       12350    12397      +47     
==========================================
- Hits         7565     7533      -32     
- Misses       4445     4526      +81     
+ Partials      340      338       -2     

Flag	Coverage Δ
unittests	60.76% <55.55%> (-0.50%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gargnitingoogle]

Integration tests are currently running. I'll add result for that once they finish.

[gargnitingoogle]

Integration tests are currently running. I'll add result for that once they finish.

Tests have passed. logs

[gargnitingoogle]

@kislaykishore As discussed offline, I have replaced config-file metadata-prefetch-mode with cli flag of same name and type. PTAL.
I have removed the code for the config-file flag in commit, which can be reverted in future, if we ever want to have both the ways of setting the configuration.

[raj-prince]

Added a couple of comments.

[gargnitingoogle]

Added a couple of comments.

@raj-prince I've addressed your comments. PTAL.