Topics/regroup non static files #6346

brotzky · 2018-07-08T22:51:39Z

Continuation of #6285

Removed --group-files flag and made grouping default
Cleaned up rebase problem

`/public/` output

`index.html` with no prefix

`index.html` with prefix of `/blog`

…tyles/script paths.

…/gatsby into topics/regroup-non-static-files

gatsbybot · 2018-07-08T22:58:02Z

Deploy preview for using-drupal ready!

Built with commit f1ad758

https://deploy-preview-6346--using-drupal.netlify.com

gatsbybot · 2018-07-08T22:59:57Z

Deploy preview for gatsbygram ready!

Built with commit f1ad758

https://deploy-preview-6346--gatsbygram.netlify.com

KyleAMathews · 2018-07-09T20:51:29Z

packages/gatsby/src/utils/webpack-utils.js

@@ -430,8 +430,8 @@ module.exports = async ({
   */
  plugins.extractText = options =>
    new MiniCssExtractPlugin({
-      filename: `[name].[contenthash].css`,
-      chunkFilename: `[name].[contenthash].css`,
+      filename: `../css/[name].[contenthash].css`,


why is ../css/ needed?

So, the css is generated in the stage build-javascript using the MiniCssExtractPlugin, which grabs the output path from webpack's config webpackConfig.output.path.

From what I could tell, there's no way of changing the output path (only publicPath?) using MiniCssExtractPlugin so adjusting the filenames is a workaround.

What's happening is the system things it's putting the CSS into the /js folder, but it actually goes back into the root, and creates a /css folder to place the CSS files in.

If there's a different way to handle this, such as creating build-css or a different webpackConfig.output.path during the time of CSS generation that would solve this workaround.

grega · 2018-07-11T09:52:49Z

Have tested this and it's looking great - a big benefit for us, thank you!

Jonic · 2018-07-12T08:55:19Z

This is looking like a great update! It's just what we need to get this one thing shipped :)

m-allanson · 2018-07-12T14:09:59Z

packages/gatsby/cache-dir/static-entry.js

          />
        )
      } else {
        headComponents.unshift(
          <style
-            data-href={urlJoin(pathPrefix, style.name)}
+            type="text/css"


Can you remove this line? See #6384

m-allanson · 2018-07-12T14:12:29Z

This looks great, thanks @brotzky! I added one nitpicky comment, then I think this is ready to merge.

…/gatsby into topics/regroup-non-static-files

…changes by mistkae

brotzky · 2018-07-12T15:08:04Z

packages/gatsby/src/utils/webpack-utils.js

@@ -430,8 +430,8 @@ module.exports = async ({
   */
  plugins.extractText = options =>
    new MiniCssExtractPlugin({
-      filename: `../css/[name].[contenthash].css`,
-      chunkFilename: `../css/[name].[contenthash].css`,
+      filename: `[name].[contenthash].css`,


I've added back the ../css/ back as this wasn't supposed to be in this commit. See the following commit for reference.

brotzky · 2018-07-12T15:08:28Z

packages/gatsby/cache-dir/static-entry.js

@@ -260,7 +260,6 @@ export default (pagePath, callback) => {
      } else {
        headComponents.unshift(
          <style


#6346 (comment) @m-allanson removed.

Thanks 👍

m-allanson · 2018-07-12T18:00:21Z

Trying this out on a local project and it's looking good!

However I noticed that assets don't always end up in the right place. Here's a demo repo: https://github.com/m-allanson/gatsby-pr-6346-demo. You'll need to run it using the version of Gatsby in this PR.

I have some font files in src/assets/. They're imported in the Layout component.

Once the site is built they end up in public/js/static/ instead of public/static:

ls public/js/static/
inter-ui-bold-a20157ae226e82804c971f33890925f7.woff2
inter-ui-bold-bfbca0680dd1ae394462ba9649e3c06d.woff
inter-ui-italic-0dbe24780396c3843fb2f3fc536761b1.woff2
inter-ui-italic-4f71078805ff7dba8db988b54caaf06a.woff
inter-ui-regular-5f4e2851ddf0dcdcb56ce5f4c4f13758.woff
inter-ui-regular-73a04e2380ff2e8195f3c67807de9b2a.woff2

brotzky · 2018-07-12T18:07:05Z

@m-allanson great job testing it out locally and thanks! I'll look into that bug today. Looks like an issue with file extension types not being filtered properly.

brotzky · 2018-07-13T02:04:52Z

packages/gatsby/src/utils/webpack-utils.js

+   * "build-javascript" output path of `/js`. The same technique is
+   * used for CSS
+   */
+  const assetRelativeRoot = `../static/`


@m-allanson This had to be treated the same way as the ../css/ case because of how there's only a build-javascript stage. #6346 (comment)

Here are all the build stages:

type Stage = "develop" | "develop-html" | "build-javascript" | "build-html"

I think in the future it could be a good idea to add discrete build stages for CSS (although CSS in JS makes this a question mark?) and static assets instead of relying on this current workaround.

Here's an example of the output folder of your example repository with this commit

m-allanson

Thank you @brotzky 👍 let's get this merged and released.

gatsbot · 2018-07-17T12:32:39Z

Holy buckets, @brotzky — we just merged your PR to Gatsby! 💪💜

Gatsby is built by awesome people like you. Let us say “thanks” in two ways:

We’d like to send you some Gatsby swag. As a token of our appreciation, you can go to the Gatsby Swag Store and log in with your GitHub account to get a coupon code good for one free piece of swag. (Currently we’ve got a couple t-shirts available, plus some socks that are really razzing our berries right now.)
We just invited you to join the Gatsby organization on GitHub. This will add you to our team of maintainers. You’ll receive an email shortly asking you to confirm. By joining the team, you’ll be able to label issues, review pull requests, and merge approved pull requests.

If there’s anything we can do to help, please don’t hesitate to reach out to us: tweet at @gatsbyjs and we’ll come a-runnin’.

Thanks again!

m-allanson · 2018-07-17T12:47:00Z

Published as gatsby@2.0.0-beta.41

brotzky · 2018-07-17T15:44:23Z

Awesome, thanks!

@m-allanson I have a feeling this feature can cause some breaks. Ill be scanning the issues going forward and please tag me if you think an issue is filed related to this.

m-allanson · 2018-07-17T16:01:15Z

please tag me if you think an issue is filed related to this

Will do, thank you!

m-allanson · 2018-07-19T14:53:00Z

Some netlify builds are failing since this was merged. I've not been able to replicate the failure locally though. Wondering if you have any insights on this @brotzky?

Here's an example failure when building next.gatsbyjs.org, from this repos www directory (full build log):

5:31:55 PM: error Building static HTML for pages failed
5:31:55 PM: See our docs page on debugging HTML builds for help https://goo.gl/yL9lND
5:31:55 PM: 
5:31:55 PM:   WebpackError: ENOENT: no such file or directory, open '/opt/build/repo/www/css  /0.22e0a7306140b3769866.css'
5:31:55 PM:   
5:31:55 PM:   
5:31:55 PM:   - static-entry.js:62 Object.fs.readFileSync
5:31:55 PM:     lib/.cache/static-entry.js:62:3
5:31:55 PM:   
5:31:55 PM:   - static-entry.js:267 styles.slice.reverse.forEach.style
5:31:55 PM:     lib/.cache/static-entry.js:267:26
5:31:55 PM:   
5:31:55 PM:

@pieh noticed that public is missing from the filepath in this line 5:31:55 PM: WebpackError: ENOENT: no such file or directory, open '/opt/build/repo/www/css /0.22e0a7306140b3769866.css', which makes me wonder if it's related to this issue?

It seems like this is the line that's failling https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/cache-dir/static-entry.js#L270.

Any ideas on what might be causing this? No worries if not!

jquense · 2018-07-19T19:59:17Z

ya, its not straightforward to organize the webpack output, all the loaders that provide names need to specify their names relative to their new output. see: https://github.com/jquense/webpack-atoms/blob/master/src/index.js#L257

IMO opinion its not worth the extra complexity for more organized output :/ to many odd things can break

KyleAMathews · 2018-07-19T20:02:57Z

@jquense hmmm yeah — makes sense. I'm not really sure this is worth it. Gatsby is a website compiler and its output isn't meant to be "understood" per se as you don't ever interact with it directly.

This reverts commit e401237.

KyleAMathews · 2018-07-20T07:05:56Z

I reverted this as it's causing bugs. @brotzky this would be nice to have in but looks like there's more work that'd have to be done to make it super solid.

iwilsonq · 2018-08-29T00:52:55Z

@KyleAMathews since this was reverted, I'm wondering if there will be any plans to implement something to group up non statics?

For example, my use case has two webapps (one gatsby, one react app) running on the same subdomain. We have a load balancer that listens for particular paths and tries to deliver the correct js bundle. While we can deliver our regular react app's bundle, the gatsby output js files are, as they stand in the root of the public folder, difficult to configure listeners for.

Currently we're using the gatsby output in an nginx container hosted on aws ec2, but the issue is the same for putting it on S3, since we have to configure the paths for CloudFront to deliver the right bundles too.

Any ideas?

KyleAMathews · 2018-08-29T15:01:09Z

@iwilsonq no plans — see @jquense's comment #6346 (comment)

Perhaps in your case you can set cookies that the load balancer can use.

koga73 · 2019-07-10T20:39:11Z

Any update on this? Is there currently a way to output css/js into folders in /public?

koga73 · 2019-07-16T15:04:56Z

I created a plugin for this: gatsby-plugin-tidy. It puts js/css into folders with option to name them and does some other things that makes the build output a bit nicer: https://www.gatsbyjs.org/packages/gatsby-plugin-tidy/

sonusynup · 2019-07-24T05:51:54Z

@KyleAMathews , Still I am not able to make directories for js/css files on production build. Should I update my gatsby core to avail this feature?

plasticine · 2020-06-16T03:53:52Z

Heya all 👋

I’m really hopeful that Gatsby changes their approach to this and revisits this functionality. I think the lack of partitioning of generated, static content is presents some pretty glaring issues and incompatibilities with webservers and reverse proxies in general.

By way of a concrete use-case — the lack of ability to prefix paths for generated media for a site build makes it nearly impossible to effectively serve a site where complex rules around serving are required for different types of content, as most reverse proxying load balancers (for example the GCP GLB) can only operate on path prefix. For example, even for a static site, It’s a pretty common use case to want to add additional headers to page/html responses, but delegate serving any static media from a CDN for example. Currently doing this is a nightmare given how Gatsby has elected to write it’s output.

To implement the above use-case, ideally you’d do the following (pseudo loadbalancer / routing config);

/static, /js, /css  -> CDN / backend bucket
everything else     -> Webserver that can add headers (HSTS, CSP, etc)

This really sticks out to me;

@KyleAMathews — Gatsby is a website compiler and its output isn't meant to be "understood" per se as you don't ever interact with it directly.

I can’t disagree more with this stance — you absolutely need to care about the generated output of a tool like Gatsby, and you absolutely need to interact with the output directly. The output is the value, it’s what you’re serving, and Gatsby doesn’t exist in a vaccum — it needs to integrate well with other tools that will naturally exist around it, and it’s unreasonable to expect them not to need to care about the structure of Gatsby’s output.

🤙🏼 Edit: Also, I realise that this is a ~year old, closed issue at this point, so I apologise if bumping it up again is perceived as unhelpful. However this seems like such a glaring flaw in what is otherwise a great tool, and I’d love to see this discussed.

brotzky added 13 commits July 3, 2018 17:07

First try at cleaning up build output

4803d61

Adding flag to build, ordering pathPrefix correctly, and cleaned up s…

e1b8be9

…tyles/script paths.

Removing the deletion of files as referenced by m-allanson and pieh

77371b9

Adding support for styles grouping

59cf18a

renaming output directories to js and css

366bd97

First try at cleaning up build output

972fb6b

Adding flag to build, ordering pathPrefix correctly, and cleaned up s…

4042ae1

…tyles/script paths.

Removing the deletion of files as referenced by m-allanson and pieh

695cb68

Adding support for styles grouping

699e5ca

renaming output directories to js and css

eed6616

Merge branch 'topics/regroup-non-static-files' of github.com:narative…

d8a8377

…/gatsby into topics/regroup-non-static-files

Removing config flag

87f0ada

Removing config flag logic from styles

cd834ab

KyleAMathews reviewed Jul 9, 2018

View reviewed changes

Merge branch 'master' into topics/regroup-non-static-files

414ce25

m-allanson reviewed Jul 12, 2018

View reviewed changes

brotzky added 3 commits July 12, 2018 08:05

Removing 'text/css' from CSS Link

41f80b6

Merge branch 'topics/regroup-non-static-files' of github.com:narative…

ef334d2

…/gatsby into topics/regroup-non-static-files

Woops, adding back the CSS as I was testing locally and commited the …

6f1e987

…changes by mistkae

brotzky commented Jul 12, 2018

View reviewed changes

Adding '../' to the relative static route similar to CSS

f1ad758

brotzky commented Jul 13, 2018

View reviewed changes

m-allanson approved these changes Jul 17, 2018

View reviewed changes

m-allanson merged commit e401237 into gatsbyjs:master Jul 17, 2018

m-allanson mentioned this pull request Jul 17, 2018

Regroup non-static files into a public's subdirectory #3760

Closed

KyleAMathews mentioned this pull request Jul 18, 2018

Is there a way to merge all JS build files in only one bundle? #6548

Closed

m-allanson mentioned this pull request Jul 19, 2018

onCreateWebpackConfig({ loaders }): loaders.url() is broken #6525

Closed

KyleAMathews added a commit that referenced this pull request Jul 20, 2018

Revert "Topics/regroup non static files (#6346)"

b24e976

This reverts commit e401237.

m-allanson mentioned this pull request Jul 20, 2018

gatsby-plugin-netlify generates wrong js Link #6536

Closed

m-allanson mentioned this pull request Aug 15, 2018

[v2] Pages render briefly then turn blank in non-local environments #7289

Closed

jaredsilver mentioned this pull request Oct 5, 2018

Question: Putting Assets in a Different Folder #8833

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Topics/regroup non static files #6346

Topics/regroup non static files #6346

brotzky commented Jul 8, 2018 •

edited

Loading

gatsbybot commented Jul 8, 2018 •

edited

Loading

gatsbybot commented Jul 8, 2018 •

edited

Loading

KyleAMathews Jul 9, 2018

brotzky Jul 10, 2018

grega commented Jul 11, 2018

Jonic commented Jul 12, 2018

m-allanson Jul 12, 2018

m-allanson commented Jul 12, 2018

brotzky Jul 12, 2018

brotzky Jul 12, 2018 •

edited

Loading

m-allanson Jul 12, 2018

m-allanson commented Jul 12, 2018

brotzky commented Jul 12, 2018

brotzky Jul 13, 2018 •

edited

Loading

m-allanson left a comment

gatsbot bot commented Jul 17, 2018

m-allanson commented Jul 17, 2018

brotzky commented Jul 17, 2018 •

edited

Loading

m-allanson commented Jul 17, 2018

m-allanson commented Jul 19, 2018

jquense commented Jul 19, 2018

KyleAMathews commented Jul 19, 2018

KyleAMathews commented Jul 20, 2018

iwilsonq commented Aug 29, 2018

KyleAMathews commented Aug 29, 2018

koga73 commented Jul 10, 2019

koga73 commented Jul 16, 2019

sonusynup commented Jul 24, 2019 •

edited

Loading

plasticine commented Jun 16, 2020 •

edited

Loading

Topics/regroup non static files #6346

Topics/regroup non static files #6346

Conversation

brotzky commented Jul 8, 2018 • edited Loading

/public/ output

index.html with no prefix

index.html with prefix of /blog

gatsbybot commented Jul 8, 2018 • edited Loading

gatsbybot commented Jul 8, 2018 • edited Loading

KyleAMathews Jul 9, 2018

Choose a reason for hiding this comment

brotzky Jul 10, 2018

Choose a reason for hiding this comment

grega commented Jul 11, 2018

Jonic commented Jul 12, 2018

m-allanson Jul 12, 2018

Choose a reason for hiding this comment

m-allanson commented Jul 12, 2018

brotzky Jul 12, 2018

Choose a reason for hiding this comment

brotzky Jul 12, 2018 • edited Loading

Choose a reason for hiding this comment

m-allanson Jul 12, 2018

Choose a reason for hiding this comment

m-allanson commented Jul 12, 2018

brotzky commented Jul 12, 2018

brotzky Jul 13, 2018 • edited Loading

Choose a reason for hiding this comment

m-allanson left a comment

Choose a reason for hiding this comment

gatsbot bot commented Jul 17, 2018

m-allanson commented Jul 17, 2018

brotzky commented Jul 17, 2018 • edited Loading

m-allanson commented Jul 17, 2018

m-allanson commented Jul 19, 2018

jquense commented Jul 19, 2018

KyleAMathews commented Jul 19, 2018

KyleAMathews commented Jul 20, 2018

iwilsonq commented Aug 29, 2018

KyleAMathews commented Aug 29, 2018

koga73 commented Jul 10, 2019

koga73 commented Jul 16, 2019

sonusynup commented Jul 24, 2019 • edited Loading

plasticine commented Jun 16, 2020 • edited Loading

brotzky commented Jul 8, 2018 •

edited

Loading

`/public/` output

`index.html` with no prefix

`index.html` with prefix of `/blog`

gatsbybot commented Jul 8, 2018 •

edited

Loading

gatsbybot commented Jul 8, 2018 •

edited

Loading

brotzky Jul 12, 2018 •

edited

Loading

brotzky Jul 13, 2018 •

edited

Loading

brotzky commented Jul 17, 2018 •

edited

Loading

sonusynup commented Jul 24, 2019 •

edited

Loading

plasticine commented Jun 16, 2020 •

edited

Loading