High CPU usage from make dev due to Gulp (?) using inefficient file watching

[robertknight]

The gulp watch tasks across our various frontend projects have high CPU usage while idle. Using wtfnode I tracked this down to instances of StatWatcher which is Node's polling-based file watcher, created by fs.watchFile API calls.

By monkey-patching fs.watchFile at the start of node_modules/.bin/gulp to log its arguments with console.trace, we can see that a lot of polling-based watchers with super-short intervals are being registered:

Trace: Watching file /Users/robert/hypothesis/client/src/boot { persistent: true, interval: 100 } [Function (anonymous)]
    at fs.watchFile (/Users/robert/hypothesis/client/node_modules/gulp-cli/bin/gulp.js:9:11)
    at setFsWatchFileListener (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:174:19)
    at NodeFsHandler._watchWithNodeFs (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:228:14)
    at NodeFsHandler._handleDir (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:429:19)
    at FSWatcher.<anonymous> (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:477:19)
    at FSWatcher.<anonymous> (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:482:16)
    at FSReqCallback.oncomplete (node:fs:197:5)
    at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)
Trace: Watching file /Users/robert/hypothesis/client/build { persistent: true, interval: 100 } [Function (anonymous)]
    at fs.watchFile (/Users/robert/hypothesis/client/node_modules/gulp-cli/bin/gulp.js:9:11)
    at setFsWatchFileListener (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:174:19)
    at NodeFsHandler._watchWithNodeFs (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:228:14)
    at NodeFsHandler._handleDir (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:429:19)
    at FSWatcher.<anonymous> (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:477:19)
    at FSWatcher.<anonymous> (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:482:16)
    at FSReqCallback.oncomplete (node:fs:197:5)
    at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)
Trace: Watching file /Users/robert/hypothesis/client/src/annotator/components { persistent: true, interval: 100 } [Function (anonymous)]
    at fs.watchFile (/Users/robert/hypothesis/client/node_modules/gulp-cli/bin/gulp.js:9:11)
    at setFsWatchFileListener (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:174:19)
    at NodeFsHandler._watchWithNodeFs (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:228:14)
    at NodeFsHandler._handleDir (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:429:19)
    at FSWatcher.<anonymous> (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:477:19)
    at FSWatcher.<anonymous> (/Users/robert/hypothesis/client/node_modules/glob-watcher/node_modules/chokidar/lib/nodefs-handler.js:482:16)
    at FSReqCallback.oncomplete (node:fs:197:5)
    at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)
...

[robertknight]

This issue was fixed by Gulp v5, which upgraded the underlying file watching libraries. I did encounter an issue with Gulp crashing if the file tree contained a very large number of files in a single folder (~4000). That is been fixed upstream (see gulpjs/glob-stream#125).