[bug]: ☂️ umbrella issue for cache and over-eager persistence issues

[DSchau]

Description

In this umbrella issue, I'd like to provide a means for others to share any issues encountered regarding caching, specifically in instances when the solution to a problem you were facing was remediated by removing the cache (e.g. rm -rf .cache).

Issues that we have seen before (but generally have not been able to reproduce) can usually be classified as one of the following:

1. Remote content sourced by a plugin is available (e.g. in the database, CMS, etc.) but is not available after a local re-build
2. A local plugin source code changed, but the change(s) do not seem to have taken effect
3. Local content (e.g. images, Markdown, etc.) were changed, but the changes do not seem to show up after a build

If any of these scenarios, or more broadly speaking, if you've ran into a problem where content wasn't appearing appropriately and deleting the cache solved this problem--then please chime in!

Intent

If we can reliably reproduce these types of issues and scenarios, we can harden our cache and improve the experience for all Gatsby users. Specifically, if we can reliably reproduce we can fix the underlying issue and then author end-to-end tests to ensure that the caching issue remains fixed.

Template

Please use the following template to report an issue so that we can most effectively debug the underlying issue.

Note: it is extremely important that you provide as much information as possible for the reproduction step. The more info and clearer the reproduction, the better we will be able to debug and ship fixes to these issues!

## Description

<!-- What led you to delete the cache? What was the exact problem that deleting the cache seemed to have fixed? -->

## Reproduction

<!-- Please, please provide detailed steps to reproduce. Ideally a Github repository where we can reproduce the issue(s) ourselves, as well. -->

## Environment Info

<!-- Run `npx gatsby info --clipboard` in the root of your Gatsby app and paste the output here -->

[mrispoli24]

I think I ran into this issue and after much frustration I started this issue but ended up closing it after deleting the public folder and getting everything to work. I do believe this sounds a lot like what others are experiencing though.

[nickytonline]

I built my site off of the Netlify CMS starter, https://github.com/nickytonline/www.iamdeveloper.com. ‪It has deleting the .cache as part of their build steps. See https://github.com/netlify-templates/gatsby-starter-netlify-cms/blob/master/package.json#L40‬

Not sure why, but they must have a good reason to delete the .cache.

I remember when I started off with another starter site, I had to delete the .cache frequently, but this was pre-Gatsby 2.

[keithmifsud]

Please bear in mind that I am not 100% sure this was the cause of the issue forcing me to delete the .cache directory but it did fix the issue as soon as I did. I have a json file with an array of objects. I decided to give each object an identifier key. Gatsby does not like this key name, maybe it conflicts with an internal key name. I renamed the identifier to name but the related changes were not being built until I deleted the .cache directory.

[endymion1818]

In working with several developers using GatsbyJS over the past 9 months I’ve only had this isssue come up once. A colleague was experiencing graphql errors because an API wasn’t being reached. We had recently changed how we fetch data (using gatsby-source-Wordpress) in Gatsby-config.js. When he updated his master branch, running the cli command “gatsby develop” still used the old settings. Deleting .cache solved the issue.

[oobidin]

I've been working with GatsbyJS last 6 months and I faced some issues when you update some local content (MDs and JSONs in our case) and it doesn't update. Actually, we've created the yarn clean command with the content: rm -rf public && rm -rf .cache to solve such kind of issues.

[trotzig]

We updated our content types in contentful (removed some types, added some). Removing the cache folder made the dev server start up again. Cc @axhamre.

[kylefraser]

We've run into an issue with contentful where, when we've added images via a rich-text, we've had to delete the .cache folder in order to query additional properties with GraphQL on the embedded asset.

For instance, we've added images, re-started the dev server, and were still unable to query image properties like

target {
   fields {
      file {
         url
      }
   }
}

deleting the .cache folder, solved this for us.

[vacas5]

I abstracted a lot of functions utilized inside my sourceNodes function into a separate file. Changes made in that file were not detected by cache busting, and so any time I changed the code in my abstracted file, I would have to clear the cache manually. Any way that Gatsby could detect changes in required files within gatsby-node.js would be advantageous to my use case.

[amankkg]

Not sure if it counts...

Recently, I had to press Clear cache and retry deploy button in Netlify's CI during this PR - that checks turned green only after cache clean.

It started to fail with adding of prismjs and its theme CSS import in gatsby-browser.js.

[u12206050]

Though it is undocumented it seems one can use Gatsby's cache object to set and get json objects. Data is stored in the .cache/cache/{plugin-name}/ directory.

In your plugin gatsby-node.js in one of the many apis include cache as an argument; eg.

exports.onPostBuild = async function(
  { cache, store, graphql },
  { some, plugin, options} ) {
    const cacheKey = 'some-key-name';
    ...
    let obj = await cache.get(cacheKey);

    if (!obj) {
      obj = { created: Date.now() };
    }
    ...
    obj.modified = Date.now();

    await cache.set(cacheKey, obj);
  }

[DSchau]

@u12206050 correct! A bit out-of-scope for this discussion though, right?

We'd love if you were able to rectify the "not documented" piece and document that API :)

[danoc]

Hi @DSchau! Thanks for making this.

Here's one that me and my team have been struggling with lately: #11109

I spent some time debugging but couldn't figure it out.

[kbrgl]

Oops, unpinned this accidentally. Sorry!

(Perhaps GitHub should make repository actions more obvious 😛)

[aarondack]

This is a pretty systemic problem whenever adding a new image. Pretty much exclusively I have to rm -rf .cache/ and re-install. Sometimes I also just nuke the public/ folder. I know this is sort of already known, just figured that I'd share that I have had this experience.

[mrispoli24]

Running into this issue again myself now, deleting the public folder and .cache folders don't seem to do the trick anymore. The images are there, I am willing to invite someone to the repository to check it out if you like but it seems that when working with static image files this happens a lot. They are present in the gh-pages branch but showing up broken on the deployed site.

[mrispoli24]

I figured out what was going on in the above case. I have a series of image files in a /static directory that are divided by folders. In my template I had the wrong case for a character so /static/Folder instead of /static/folder. In development and on local building this ran just fine, upon deployment to github this rightly broke. Not sure exactly how gatsby builds things under the hood but it would have been a faster catch had it broken in development mode locally. This might not have had anything to do with the cache but I think it's worth letting ya'll know in case this is a symptom or indicator of something here.

[smerth]

I wrote plugin gatsby-source-gh-readme to pull file content data from a user's github account, it does the job it should do but I need to clear the cache before running gatsby develop or most of the time the build fails. I added a comment about this here because I think it might be related

[mkaradeniz]

I'm running into trouble with gatsby-source-filesystem and remote assets.

I'm using gatsby-source-graphql to use GraphCMS in my project. I added a custom resolver to download the GraphCMS assets via createRemoteNode to use gatsby-image later.

gatsby-node.js:

exports.createResolvers = ({ actions, cache, createNodeId, createResolvers, store, reporter }) => {
  const { createNode } = actions;

  createResolvers({
    GCMS_Asset: {
      file: {
        type: `File`,
        resolve(source, args, context, info) {
          return createRemoteFileNode({
            url: source.url,
            store,
            cache,
            createNode,
            createNodeId,
            reporter,
          });
        },
      },
    },
  });
};

When first starting gatsby develop or gatsby build without having a .cache, it will download the asset(s) and this query works fine:

    query {
      gcms {
        assets(first: 1) {
          url
          file {
            childImageSharp {
              fluid {
                src
                originalName
              }
            }
          }
        }
      }
    }

the second time I start develop or build with the .cache present, I get an ENOENT error:

Unexpected error value: "failed to process https://media.graphcms.com/ASSET_HANDLE\nError: ENOENT: no such file or directo
ry, stat 'C:\\...\\.cache\\gatsby-source-filesystem\\a7762a6a4f2571864804231ff8b216e6\\ASSET_HANDLE
gR'"

The file does exist:

C:\..\.cache\gatsby-source-filesystem\a7762a6a4f2571864804231ff8b216e6\ASSET_HANDLE.jpg

The only thing that I do find odd, is that it's looking for the file ASSET_HANDLE not ASSET_HANDLE.jpg.

Deleting the .cache would work, but in the future I'll have about 3k images in my project, so downloading on every build would be a waste of bandwith and time.

Edit:
After some digging, I think my suspicion was right. The file can't be found because we don't get a file extensions from looking at the url, so ext is empty. When the file is not in cache, we get the ext from the buffer, but that will not be called when trying to get the file from .cache.

As a workaround I could add the ext: '.jpg' when using createRemoteFileNode, but that would limit me to only use one extension for all assets, which might not be what I want or, since I'm using GraphCMS in this example, I could get the original filename and get the extension from that, but that also doesn't seem optimal.

[DmacTorstar]

Having this issue, and detailed my experience in this ticket:
#18030

[slaterbbx]

I think I have found that when using gatsby-background-image I have random .cache issues that are only on the client side when viewing the website, If I clear my local cach or turn off the local cache and then reload the pages, it will work fine after. Anyone have a similar issue?

Has me scared to do webhook based rebuilds on my project, If I do, can I just run gatsby clean then gatsby build to guarantee a clean build?

[sidharthachatterjee]

Let’s go ahead and close this for now since it’s been relatively quiet and people seem to have been hitting this less often than before.

Please open independent issues with reproductions for cases where caching breaks your gatsby build so that we can action those!

Thanks everyone!

[jonahallibone]

Unsure if this is an issue related solely to Gatsby, but I've had to use this command twice in the last two days. I am using the gatsby-source-contentful plugin, and have found that it will often cache images when content changes, leading to GraphQL throwing errors on those models. This appears to only happen with SVG images, but it's possible I am changing those most often.

[NigeAtAnd]

I'm fairly new to Gatsby and also had to implement a clean step to get it to pick up changes in page property values from gatsby-source-contentful. The issue seems to be that despite the Contentful plugin seeing that the item in Contentful has been updated, the page's HTML isn't being regenerated - after the build, the index.html file has an old timestamp.

Here's a log in case this helps:

> gatsby build

success open and validate gatsby-configs - 0.185s
success load plugins - 2.945s
success onPreInit - 0.049s
success initialize cache - 0.006s
success copy gatsby files - 0.581s
success onPreBootstrap - 0.036s
success createSchemaCustomization - 0.014s
success Contentful: Sync changed items - 0.289s - 1/1000 3461.54/s
success Contentful: Fetch data (xxx) - 1.235s
info Updated entries 1
info Deleted entries 0
info Updated assets 0
info Deleted assets 0
success Contentful: Process data (xxx) - 0.131s
info Creating 2 Contentful -- App download nodes
info Creating 2 Contentful -- Pull quote nodes
info Creating 1 Contentful -- HTML Template nodes
info Creating 19 Contentful -- Image and text card nodes
info Creating 23 Contentful -- Internal link nodes
info Creating 18 Contentful -- External link nodes
info Creating 63 Contentful -- Text nodes
info Creating 28 Contentful -- Hero nodes
info Creating 2 Contentful -- Video nodes
info Creating 14 Contentful -- Signpost link nodes
info Creating 7 Contentful -- Image nodes
info Creating 73 Contentful -- Section nodes
info Creating 1 Contentful -- Statement block nodes
info Creating 31 Contentful - Page nodes        
info Creating 27 Contentful -- Editorial card nodes
info Creating 38 Contentful asset nodes
success Contentful: Create nodes (xxx) - 0.106s
success Checking for changed pages - 0.003s
success source and transform nodes - 1.743s
success building schema - 2.078s
info Total nodes: 755, SitePage nodes: 32 (use --verbose for breakdown)
success createPages - 1.125s
success Checking for changed pages - 0.001s
success createPagesStatefully - 0.064s
success Cleaning up stale page-data - 0.007s
success update schema - 0.404s
success onPreExtractQueries - 0.005s
success extract queries from components - 2.220s
success write out redirect data - 0.006s
success Build manifest and related icons - 0.100s
success onPostBootstrap - 0.106s
info bootstrap finished - 19.075s
success write out requires - 0.019s
success Building production JavaScript and CSS bundles - 21.221s
success Building HTML renderer - 1.888s
info There are no new or changed html files to build.
success onPostBuild - 0.001s
info Done building in 42.3978705 sec

Environment:

  System:
    OS: Windows 10 10.0.19042
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  Binaries:
    Node: 14.16.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.4 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.14.11 - C:\Program Files\nodejs\npm.CMD
  Languages:
    Python: 3.8.8 - C:\Users\Admin\.pyenv\pyenv-win\shims\python.BAT
  Browsers:
    Chrome: 90.0.4430.212
    Edge: Spartan (44.19041.906.0), Chromium (90.0.818.66)

Also from package.json:

    "gatsby": "^3.3.1",
    "gatsby-plugin-catch-links": "^3.4.0",
    "gatsby-plugin-image": "^1.4.0",
    "gatsby-plugin-manifest": "^3.5.0",
    "gatsby-plugin-postcss": "^4.4.0",
    "gatsby-plugin-react-helmet": "^4.4.0",
    "gatsby-plugin-sharp": "^3.4.0",
    "gatsby-source-contentful": "^5.4.0",
    "gatsby-source-filesystem": "^3.4.0",
    "gatsby-transformer-remark": "^4.1.0",
    "gatsby-transformer-sharp": "^3.4.0",

[edsu]

I've definitely observed strange behavior when I:

gatsby clean
gatsby develop
# gatsby app runs fine ...
<ctrl-c>
gatsby develop
# gatsby app broken, graphql editor shows missing data

and I get a cannot query field GraphQL error. Then if I do a gatsby clean I can run gatsby develop again, but only once.

I noticed in a comment above that @keithmifsud thought that changing the property name in his JSON data from 'identifier to name helped. I had an id property in my JSON objects. Changing this to personId made the entire problem go away and I can repeatedly run gatsby develop without needing to clean the cache.

Does anyone know what might be going on in situations like this? My GraphQL query included a clause to another JSON file with an id property. Were they clashing somehow? Is there a list of reserved words that can't be used in JSON objects when inferring the schema? I spent hours, maybe days debugging this, and the fix is entirely unsatisfying.

  query($id: String!) {
    peopleJson(id: {eq: $id}) {
      id
      name
      type
    }
    wikipediaJson(personId: {eq: $id}) {
      personId
      abstract
      image {
        childImageSharp {
          gatsbyImageData(width: 300)
        }
      }
    }

[KyleAMathews]

My GraphQL query included a clause to another JSON file with an id property. Were they clashing somehow?

@edsu This is likely it. gatsby-transformer-json uses your id if you supply one but Node IDs must be universally unique as all nodes live in the same namespace — see https://www.gatsbyjs.com/docs/reference/config-files/actions/#createNode

[ibrahimaq]

Ran into this issue today. I was setting up Gatsby with Contentful and wrote some dummy text in the Rich Text Field in Contentful to fetch through GraphQL. It was running smooth until I edited the rich text/embedded assets. It through this error:

Missing onError handler for invocation 'building-schema', error was 'Invariant Violation: Encountered an error trying to infer a GraphQL type for: `blog    
post___NODE`. There is no corresponding node with the `id` field matching: "6G3Uz8VEYCJCNEsOk5z5jP".'. Stacktrace was 'Invariant Violation: Encountered an  
error trying to infer a GraphQL type for: `blog post___NODE`. There is no corresponding node with the `id` field matching: "6G3Uz8VEYCJCNEsOk5z5jP".        
    at invariant (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\invariant\invariant.js:40:15)
    at getFieldConfigFromFieldNameConvention (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:254:3)  
    at getFieldConfig (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:143:19)
    at forEach (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:88:25)
    at Array.forEach (<anonymous>)
    at addInferredFieldsImpl (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:72:28)
    at addInferredFields (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\add-inferred-fields.js:27:3)
    at addInferredType (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\index.js:107:3)
    at map (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\index.js:65:5)
    at addInferredTypes (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\infer\index.js:64:23)
    at updateSchemaComposer (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\schema.js:137:9)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processTimers (node:internal/timers:497:9)
    at build (C:\Users\Owner\Desktop\Websites\portfolio\node_modules\gatsby\src\schema\index.js:112:18)'

⠸ building schema

and building scheme never ended until I Ctrl C.

I ran gatsby clean to clear the .cache and public folder. Started dev server and it worked.
After endless digging I think I'm going to clean .cache and public folder as part of my build from now on.

[ZeldOcarina]

Issue: Import error when using component shadowing in gatsby-theme with gatsby 5.2.0

Description:

I am experiencing an import error when working with component shadowing in a gatsby-theme that uses gatsby 5.2.0. Every time I make a change in the codebase, I get the following error:

Attempted import error: '$virtual/async-requires' does not contain a default export (imported as 'asyncRequires').

The only way to resolve this error is to run gatsby clean and then gatsby start again.

Steps to reproduce:

1. Create a gatsby-theme using gatsby 5.2.0
2. Use component shadowing in the theme
3. Make any change in the codebase
4. Observe the import error

Expected behavior:

I expect the code to recompile and update the site without any errors.

Actual behavior:

I am getting the import error described above every time I make a change in the codebase.

Environment:

• Gatsby version: 5.2.0
• Operating system: M1 MacOS

Additional context:

Has anyone else encountered this issue before? Is there a fix or workaround for this problem? I would really appreciate any help or guidance on how to resolve this issue. Thank you!

[alice-sugarblock]

Issue: Import error when using component shadowing in gatsby-theme with gatsby 5.2.0

Description:

I am experiencing an import error when working with component shadowing in a gatsby-theme that uses gatsby 5.2.0. Every time I make a change in the codebase, I get the following error:

Attempted import error: '$virtual/async-requires' does not contain a default export (imported as 'asyncRequires').

The only way to resolve this error is to run gatsby clean and then gatsby start again.

Steps to reproduce:

1. Create a gatsby-theme using gatsby 5.2.0
2. Use component shadowing in the theme
3. Make any change in the codebase
4. Observe the import error

Expected behavior:

I expect the code to recompile and update the site without any errors.

Actual behavior:

I am getting the import error described above every time I make a change in the codebase.

Environment:

• Gatsby version: 5.2.0
• Operating system: M1 MacOS

Additional context:

Has anyone else encountered this issue before? Is there a fix or workaround for this problem? I would really appreciate any help or guidance on how to resolve this issue. Thank you!

This also happened to me when using a monorepo architecture with a "shared" package that my Gatsby site imported. What worked for me was to add "type": "commonjs" to my Gatsby's site package.json (while my "shared" package used "type": "module").