Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata-prefetch (aka Recursive listing or ls -R during mount) #1930

Merged
merged 36 commits into from
May 24, 2024
Merged
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
b5b6378
Call ls -R from within mount
gargnitingoogle May 16, 2024
c6cc8dc
Temp - [WIP] concurrent listings of sibling directories
gargnitingoogle May 20, 2024
a24e655
Revert "Temp - [WIP] concurrent listings of sibling directories"
gargnitingoogle May 20, 2024
7c356e2
Add mount-config metadata-prefetch-mode
gargnitingoogle May 20, 2024
4e606c6
minor improvements
gargnitingoogle May 20, 2024
5ba736b
Migrate main_test.go to stretchr/testify
gargnitingoogle May 20, 2024
824c3aa
metadata-prefetch-mode configs in listing e2e tests
gargnitingoogle May 20, 2024
419d7a0
fix lint errors
gargnitingoogle May 20, 2024
f434cd4
temp-fix for integration-test failure
gargnitingoogle May 21, 2024
7e57d36
fix some self-comments
gargnitingoogle May 21, 2024
ed19a5d
add unit tests for populateList
gargnitingoogle May 21, 2024
db55591
handle dynamic-mount scenario
gargnitingoogle May 21, 2024
903c7e9
beautify test code
gargnitingoogle May 21, 2024
b7351e3
increase code-coverage
gargnitingoogle May 21, 2024
7f3887b
add code for coverage
gargnitingoogle May 21, 2024
b7da79a
fix typo in prev commit
gargnitingoogle May 21, 2024
b8413c7
remove type metadataPrefetchModeValue
gargnitingoogle May 22, 2024
18dc77e
address a review comment
gargnitingoogle May 22, 2024
efd1234
Add commandline flag metadata-prefetch-mode
gargnitingoogle May 22, 2024
82fa823
Put back unintentionally ignored error
gargnitingoogle May 22, 2024
7f5d346
Migrate flags_test to stretchr/testify
gargnitingoogle May 22, 2024
ab9c962
remove config-flag metadata-prefetch-mode
gargnitingoogle May 22, 2024
7eb076f
fix for persistent mounting
gargnitingoogle May 22, 2024
427abfe
address couple of review comments
gargnitingoogle May 22, 2024
2b64ed1
addressed some more comments
gargnitingoogle May 22, 2024
d380e47
add linter-error abt missing error-check
gargnitingoogle May 22, 2024
9a81f6f
Rename metadata-prefetch-mode
gargnitingoogle May 23, 2024
282e962
Change [a]synchronous to [a]sync for md-prefetch
gargnitingoogle May 23, 2024
b30cee0
address review comments
gargnitingoogle May 23, 2024
d4eb6d2
MetadataPrefetchMode -> MetadataPrefetchOnMount
gargnitingoogle May 23, 2024
bd85941
Optimize code
gargnitingoogle May 23, 2024
e3c27fe
handle _ as bucket for dynamic-mount
gargnitingoogle May 23, 2024
a54ef3e
optimize code
gargnitingoogle May 23, 2024
74b2d37
improve error logs
gargnitingoogle May 23, 2024
041f881
Remove callList from daemon-parent
gargnitingoogle May 23, 2024
e554469
Code optimization
gargnitingoogle May 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 46 additions & 4 deletions flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ import (
const (
// maxSequentialReadSizeMb is the max value supported by sequential-read-size-mb flag.
maxSequentialReadSizeMb = 1024

// MetadataPrefetchOnMountFlag is the name of the commandline flag for enabling
// metadata-prefetch mode aka 'ls -R' during mount.
MetadataPrefetchOnMountFlag = "metadata-prefetch-on-mount"
)

// Set up custom help text for gcsfuse; in particular the usage section.
Expand Down Expand Up @@ -360,6 +364,16 @@ func newApp() (app *cli.App) {
Name: "debug_mutex",
Usage: "Print debug messages when a mutex is held too long.",
},

/////////////////////////
// Post-mount actions
/////////////////////////

cli.StringFlag{
Name: MetadataPrefetchOnMountFlag,
Value: config.DefaultMetadataPrefetchOnMount,
Usage: "This indicates whether or not to prefetch the metadata (prefilling of metadata caches and creation of inodes) of the mounted bucket at the time of mounting the bucket. Supported values: \"disabled\", \"sync\" and \"async\". Any other values will return error on mounting. This is applicable only to single-bucket mount-points, and not to dynamic-mount points.",
},
},
}

Expand Down Expand Up @@ -423,6 +437,15 @@ type flagStorage struct {
DebugHTTP bool
DebugInvariants bool
DebugMutex bool

// Post-mount actions

// MetadataPrefetchOnMount indicates whether or not to prefetch the metadata of the mounted bucket at the time of mounting the bucket.
// Supported values: MetadataPrefetchOnMountDisabled, MetadataPrefetchOnMountSynchronous, and MetadataPrefetchOnMountAsynchronous.
// Any other values will return error on mounting.
// This is applicable only to single-bucket mount-points, and not to dynamic-mount points. This is because dynamic-mounts don't mount the bucket(s) at the time of
// gcsfuse command itself, which flag is targeted at.
MetadataPrefetchOnMount string
}

func resolveFilePath(filePath string, configKey string) (resolvedPath string, err error) {
Expand Down Expand Up @@ -563,6 +586,9 @@ func populateFlags(c *cli.Context) (flags *flagStorage, err error) {
DebugFS: c.Bool("debug_fs"),
DebugInvariants: c.Bool("debug_invariants"),
DebugMutex: c.Bool("debug_mutex"),

// Post-mount actions
MetadataPrefetchOnMount: c.String(MetadataPrefetchOnMountFlag),
}

// Handle the repeated "-o" flag.
Expand All @@ -575,17 +601,33 @@ func populateFlags(c *cli.Context) (flags *flagStorage, err error) {
return
}

func validateMetadataPrefetchOnMount(mode string) error {
gargnitingoogle marked this conversation as resolved.
Show resolved Hide resolved
switch mode {
case config.MetadataPrefetchOnMountDisabled:
fallthrough
case config.MetadataPrefetchOnMountSynchronous:
fallthrough
case config.MetadataPrefetchOnMountAsynchronous:
return nil
default:
return fmt.Errorf(config.UnsupportedMetadataPrefixModeError, mode)
}
}

func validateFlags(flags *flagStorage) (err error) {
if flags.SequentialReadSizeMb < 1 || flags.SequentialReadSizeMb > maxSequentialReadSizeMb {
err = fmt.Errorf("SequentialReadSizeMb should be less than %d", maxSequentialReadSizeMb)
return
return fmt.Errorf("SequentialReadSizeMb should be less than %d", maxSequentialReadSizeMb)
}

if !flags.ClientProtocol.IsValid() {
err = fmt.Errorf("client protocol: %s is not valid", flags.ClientProtocol)
return fmt.Errorf("client protocol: %s is not valid", flags.ClientProtocol)
}

return
if err = validateMetadataPrefetchOnMount(flags.MetadataPrefetchOnMount); err != nil {
return fmt.Errorf("%s: is not valid; error = %w", MetadataPrefetchOnMountFlag, err)
}

return nil
}

// A cli.Generic that can be used with cli.GenericFlag to obtain an int flag
Expand Down
Loading
Loading