Add support for subproject and per-version sitemaps, and styled main sitemap #12249
Conversation
…sitemap This was a little morning project to expand our default sitemap to make allow projects to create more granular sitemaps. The idea here is that the main sitemap becomes a `sitemapindex` sitemap that points to multiple individual sitemap files. These include: - A sitemap of version URLs (this is the current prod sitemap) - A sitemap for each version (new) - A sitemap for each subproject version (new) The new sitemaps will 404 if the user doesn't output them. And in addition, this replaces the XML and XML comment approach, which doesn't render in many browsers, with a super basic XSLT template. The template isn't necessiarly needed but it's a nicer experience for browsers.
Are these new sitemaps auto-generated by Read the Docs if the user doesn't output them as we currently do? |
|
Does it make sense to return different content under |
Is XSLT supported for sitemaps? I've never heard of it. I checked the official and Google documentation and I didn't find any mention to it. Do we have documentation around this? |
XSLT is a feature from the late 90s for XML, not sitemaps specifically. Sitemaps are just XML so XSLT is supported through that. Crawlers don't interact with the template, they just use the XML. There are many examples of this in the wild, like: https://www.sitemap.style/
The top-level sitemap request is convention ( I mostly just didn't want to get into a dedicated view/URL just yet. We could make a separate view and URL for the list of version URLs if there is a strong reason though. We just can't mix a list of sitemaps with a list of URLs in a single sitemap.
Not at the moment. Currently, users can output these sitemap files but there are some hacks required to be able to use them instead of our sitemap with a single link to the active version base URL. We could generate these files, but I think it's also fine to require the user to use their tooling to generate these. These per-version and per-subproject-version sitemaps will 404 if they aren't output by the project. |
Yeah, I was asking about this in particular because of the
I think it's fine to require the user to generate these files for now. We can auto-generate them later if we want to.
However, I don't think we should link to them if they are dead links. I suppose this will have negative impact in SEO. Footnotes
|
Because
This could be an improvement for later, but I wouldn't block this incremental change on an end goal change like that. Our current sitemap doesn't provide projects with good SEO already, so any improvement to sitemap handling would be better for SEO. We can only guess if returning a 404 on sitemaps is even potentially a negative to SEO in comparison.
Sitemaps can be linked from robots.txt, but I started with this as a sitemap feature because it's more common for users to override robots.txt for various reasons. Overriding robots.txt will break any features we add to our default, so seemed like a place to isolate features. I don't feel robots.txt is our best option, but it's a direction we could go. |
This was a little morning project to expand our default sitemap to make
allow projects to create more granular sitemaps. The idea here is that
the main sitemap becomes a
sitemapindexsitemap that points tomultiple individual sitemap files. These include:
The new sitemaps will 404 if the user doesn't output them.
And in addition, this replaces the XML and XML comment approach, which
doesn't render in many browsers, with a super basic XSLT template. The
template isn't necessiarly needed but it's a nicer experience for
browsers.