[v2] improve how we load GraphQL query results in development/production (aka Ludicrous Mode) #4555
Conversation
ghost
assigned 
…ng it with webpack import/require
|
Here's dirty WIP for build based on this PR json-loader...pieh:json-loader (I reverted change to write json directly to public): My current TO-DO:
Results:
|
|
this is awesome |
|
@pieh woah! That's huge!!! Bigger impact than I expected actually. It'll be even faster with webpack 4 & multi-core support for HTML building (we should do that post-v2 launch). |
|
Tweet for this is up to 319 likes haha https://twitter.com/gatsbyjs/status/974507205121617920 Great job @m-allanson! |
|
@pieh yeah, just use Node's built-in crypto functionality to create a hash of the query result and use that to write out the file. Then store that as part of the pages object so pages-writer has access to it. |
KyleAMathews
changed the title
| ws.pushResult(result) | ||
| const sockets = ws.instance() | ||
| if (sockets) { | ||
| sockets.emit(`queryResult`, result) |
There was a problem hiding this comment.
This should only emit the resultJSON — for large sites, this could turn into megabytes of JSON.
|
Freecodecamp is running into scaling troubles so let's test these new changes on their wip branch and get them upgraded as soon as we can things stable https://github.com/freeCodeCamp/guides/tree/feat/new-challenge-stubs |
…e fetch to not create additional request, with any `cache` or `mode` configuration
| @@ -111,14 +111,6 @@ function shouldUpdateScroll(prevRouterProps, { location: { pathname } }) { | |||
| return true | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
Was this moved elsewhere?
There was a problem hiding this comment.
This removal probably shouldn't be in this PR - this is dead code even in master - https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/cache-dir/root.js#L120 addNotFoundRoute is defined but not used anywhere and so is noMatch after removal of previous function
…f path already has query in flight
…cessing (#4785) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log
* Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * Only log file events if we're past bootstrap
* prevent adding duplicate redirects * don't write new `redirects.json` if redirects didn't change prevents webpack recompilation on data change
* dont emit results for layouts * [develop] store query results in memory, read json data from file only if we don't have it stored yet (we didn't run this query, but results are cached)
Query running is sadly not very ludicrous right now on gatsbyjs.org — not sure why — each markdown file change causes ~20 queries to run but even with prioritizing the active page's query, it's still ~2 seconds before the page updates. This sort of thing will be much easier to debug with tracing support.
KyleAMathews
changed the title
|
🎉 thanks @m-allanson and @pieh!!!!!!! This is a huge improvement for development and large site builds. So happy to get this in :-D |
|
I'll push out a new alpha so people can try this out. |
| s.on(`disconnect`, s => { | ||
| leaveRoom(activePath) | ||
| }) | ||
|
|
There was a problem hiding this comment.
Just checking here - I think after removing this handler, when browser tab will be closed (so unregisterPath event won't be emitted), active path won't get unregistered and therefore query for that path will continue to be prioritized
There was a problem hiding this comment.
Yeah... that's right — lemme add this back real quick
|
So cool!!! Great job guys! BTW how can you do a preview as you type? Mine only updates after I save the file. I guess now, we should be able to do a live preview on production:
|
|
@HriBB @m-allanson used VSCode with AutoSave enabled and autosavedelay set to 10 (milliseconds) |
…ion (aka Ludicrous Mode) (gatsbyjs#4555) * Create placeholder JSON store * Rename * Websocket placeholder * Push query results JSON over websockets * More descriptive variable name * Fix queries being overwritten * Remove eslint-disable flag * Remove junk * test require error fix for windows * dont require json data in sync-require * dont add layout data to json array multiple times * initial async loading * revert saving json directly to public for now * updated production-app to sync with prop name change in ComponentRenderer * we load json data via json-loader component in develop and not handling it with webpack import/require * hashes for json files * fix preloading, use xhr instead of fetch - for some reason can't force fetch to not create additional request, with any `cache` or `mode` configuration * dont use full paths in dataPath - remove static/d/ path and .json ext - results in smaller app bundle especially with large ammount of pages * Enable cached query results to be loaded * Don't dump all query results out to the client Instead only push results out if the data is for a path that's currently being viewed in a client. * fix preload link to json data * remove not used function * remove more not used code * Update to latest webpack/mini-css-extract-plugin * don't write new (a)sync-requires.js if components didn't change (gatsbyjs#4759) * create just one websocket client (gatsbyjs#4763) * Filter out duplicate query jobs and create secondary queue for jobs if path already has query in flight * [json-loader] Don't emit new file node until previous is finished processing (gatsbyjs#4785) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * [json-loader] Only log file events if we're past bootstrap (gatsbyjs#4826) * Don't emit new file node until previous is finished processing This is an experiment to use [xstate](http://davidkpiano.github.io/xstate/docs/#/) to setup state machines to better handle complex state changes as we sometimes have. Ideally this happens in core and then gatsby-source-filesystem just has a simple queue and emits a new file node every time the system returns to idle. In a future refactor we'll do that plus refactor other parts of core that should be handled in a state machine e.g. pages-query-runner.js This PR also reinforced the need for us to implement [tracing](https://github.com/jaegertracing/jaeger) in core / plugins as that'd make it far far easier to understand what's happening and when. * Document state machine and remove extraneous Chokidar states * Remove console.log * Only log file events if we're past bootstrap * [json-loader] dont recompile on data change - part 2 (gatsbyjs#4837) * prevent adding duplicate redirects * don't write new `redirects.json` if redirects didn't change prevents webpack recompilation on data change * [json-loader] develop - reading results from file improvments (gatsbyjs#4850) * dont emit results for layouts * [develop] store query results in memory, read json data from file only if we don't have it stored yet (we didn't run this query, but results are cached) * Add query prioritization based on what page(s) user(s) are on Query running is sadly not very ludicrous right now on gatsbyjs.org — not sure why — each markdown file change causes ~20 queries to run but even with prioritizing the active page's query, it's still ~2 seconds before the page updates. This sort of thing will be much easier to debug with tracing support. * Add initial forward slash * Actually this is how we add back the initial forward slash
The beginnings of a custom JSON loader for query results. In
developmode this pushes query results straight to the browser via websockets: