Better integration with docgen tools

[ericnakagawa]

🚀 Feature

Add support for running Sphinx doc generation

Have you read the Contributing Guidelines on issues?

Yes

Motivation

This support would allow building docs for Python projects like PyTorch.

Pitch

Automatically allow configuration for running sphinx doc build steps when building a Docusaurus site.

Fulfills part of #78.

[jgibbarduk]

+1

[yangshun]

This won't be added for v1, but we are considering it for v2. @dmitryvinn was working on one.

[jkquijas]

What's the state of this issue?

[slorber]

@jkquijas no progress. I don't even really know what should be done, can someone give more context on what exactly should be built?

As it can apparently be done with v2 plugins, does it need to live in the core repo and have official support?

[fross123]

What would a plugin look this look like? I'm trying to solve this one at the moment as well. I haven't really found a way to get the .rst files from sphinx to build in a format thats useful to use in Docusaurus.

[JoelMarcey]

cc @brianjo @dmitryvinn-fb

[jochman]

I would like to see a mkdocs-like plugin.

[slorber]

@jochman if you have a strong opinion about how Docusaurus+Sphinx should work, please write detailed instructions on how this should work exactly. Until someone comes up with a concrete plan / RFC / API design, this issue remains a "vague wish" and it's unlikely we'll work on it. I personally have no idea about what should be done.

[gabrielcsapo]

@ericnakagawa if you want to brainstorm on what this would look like we are trying to use this more internally at Linkedin and one of the major uses of it would be python apps and libraries who are writing RST and using sphinx for their docs.

[slorber]

@gabrielcsapo I don't have any experience with Sphinx but would be happy to brainstorm with you and get a better understanding on what the expectations are

[gabrielcsapo]

@slorber I think the biggest thing is RST support over markdown.

[gabrielcsapo]

I am down to meet to talk about the use cases we have at Linkedin if you want to come up with a higher level epic or idea of how we can handle this!? @slorber

[brianjo]

Hey all,

Just FYI, I have a demo site that builds Docusaurus and Sphinx docs. It uses GitHub actions and I've deployed it in a number of projects:

https://github.com/brianjo/pydocsite
https://brianjo.github.io/pydocsite/

Here's the script: https://github.com/brianjo/pydocsite/blob/master/.github/workflows/main.yml

You can ignore that hacky little tutorial notebook thing I was doing. I'm still working through that. :)

If you have any questions, just let me know.

Cheers,

Brian

[pbk0]

please write detailed instructions on how this should work exactly. Until someone comes up with a concrete plan / RFC / API design, this issue remains a "vague wish" and it's unlikely we'll work on it. I personally have no idea about what should be done.

Thanks @brianjo this looks promising .... hope there is some official announcement for python sphinx projects

[slorber]

@brianjo example seems to be 2 sites under the same domain:

• https://brianjo.github.io/pydocsite/ => Docusaurus
• https://brianjo.github.io/pydocsite/api => Sphinx

And there's a Docusaurus navbar link to the Sphinx site

{href: 'https://brianjo.github.io/pydocsite/api/', label: 'API', position: 'left'},

Is this what you expect from a Docusaurus + Sphinx integration?

Because you can easily do that, it's out of the scope of Docusaurus or Sphinx, and is a hosting/deployment concern: you just need to merge 2 static deployments together.

I suspect users may have different expectations here

[gabrielcsapo]

My expectation of sphinx support would be being able to render a sphinx rst 100% using Docasaurus and ignore sphinx all together.

…

On Thu, Mar 24, 2022 at 8:24 AM Sébastien Lorber ***@***.***> wrote: @brianjo <https://github.com/brianjo> example seems to be 2 sites under the same domain: - https://brianjo.github.io/pydocsite/ => Docusaurus - https://brianjo.github.io/pydocsite/api => Sphinx And there's a Docusaurus navbar link to the Sphinx site {href: 'https://brianjo.github.io/pydocsite/api/', label: 'API', position: 'left'}, [image: image] <https://user-images.githubusercontent.com/749374/159950225-255877a5-dd63-4d3a-a6ae-e8b0b65178ba.png> Is this what you expect from a Docusaurus + Sphinx integration? Because you can easily do that, it's out of the scope of Docusaurus or Sphinx, and is a hosting/deployment concern: you just need to merge 2 static deployments together. I suspect users may have different expectations here — Reply to this email directly, view it on GitHub <#1059 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE2W43HZPRNJQJUNGQ7Q3VBSCKNANCNFSM4F6YFNAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[brianjo]

My expectation of sphinx support would be being able to render a sphinx rst
100% using Docasaurus and ignore sphinx all together.

Yes, that was what I was thinking I would like to see as well.

[warsaw]

My expectation of sphinx support would be being able to render a sphinx rst
100% using Docasaurus and ignore sphinx all together.

What about Sphinx extensions? There's an incredibly rich ecosystem for them, both built-in and on PyPI. I don't underestimate the challenge of supporting those, but I wonder if raw RST support w/o Sphinx extensions would get much traction. (It's like alternative implementations of Python -- if they don't support extension modules, they don't have much of a hope of being used more than just for niche applications.)

[pbk0]

BoTorch seems to use docusaurus with sphinx.
They are using python scripts to get naked HTML from sphinx and then directly serve that as static content to docusaurus. This then has no need for supporting rst parsing, as sphinx has already taken care of that. Moreover, they also have python-notebooks parsed to blogs using this python script.

So, is it possible to have something official in similar lines, as this approach will involve less maintenance??

But I would say raw HTML from sphinx does not blend well with docusaurus as can be seen here, while for python notebooks it can be seen here. Nonetheless, as of now this is better than nothing.

@slorber may be improving integration of raw HTML generated by sphinx might be right and easy direction…

[slorber]

Looks interesting thanks, never saw this integration before and not sure how it's setup 🤔

It is Docusaurus v1 though, but I guess the approach could also be used with v2

[Josh-Cena]

Typedoc integration is done in a similar fashion, iirc. Typedoc outputs HTML files that are statically served by the Docusaurus site. Example is ts-node: https://typestrong.org/ts-node/ The "API" section is external, generated with Typedoc.

[Josh-Cena]

Also, note that we now have a postBuild lifecycle (this issue was written back in 2018!), which can be used to trigger sphinx builds and copy the build output. I believe that's what the OP is asking for. Maybe we can have a way to do that more naturally.

[slorber]

@Josh-Cena look at the integration demonstrated by @pbk0 :

• https://botorch.org/api/
• https://botorch.org/tutorials/custom_botorch_model_in_ax

As far as I understand, Docusaurus v1 pages (quite similar to v2) have been generated before building the Docusaurus site from the Sphinx output, reading its generated HTML.

This is more similar to what we do to generate the blog feed (render the Docusaurus blog, and then use blog HTML in RSS feed content)

I believe that's what the OP is asking for.

This seems to be the case yes, but at the same time many users of this issue probably want more than a postBuild hook, cf the Botorch case reported by @pbk0

[Josh-Cena]

That certainly not the OP asked for, but I'm fine with keeping it open and seeing what others think. So we want to generate HTML from external tools first, and then insert the HTML into our layout? Doesn't sound unactionable then, the only problem being how hydration would work in such case

[Hecatron]

This is quite interesting, I was looking for a way to move from Sphinx to docusaurus myself
But I needed an equivalent to https://github.com/michaeljones/breathe for generating docs from doxygen
(Since breathe seems to work quite better than moxygen for non c++ langs such as python)

From the looks of things I can just use the postBuild with doxygen / Sphinx / breathe to generate the bits I need in advance
Then pass this onto docusaurus

[pbk0]

Also check pydoc-markdown
They have multiple renderer's there, one of that happens to be docusaurus.
Note that this does not use sphinx, but it has its own specification docspec
I am uncertain if docspec has any traction in python documentation. But have a look for those who are ending up here after google search.

[BenjaminYdeVintecc]

Are there any updates on this?

[pgzmnk]

Updates on this?

[slorber]

Edit: not sure why I received a notification about this issue, although the comment above is from last year 😅

Are there any updates on this?

As you can see from the above discussions it's still unclear what should be built, and even if we should build anything in the first place.

We have a plugin API, and we don't have the bandwidth to create all the plugins ourselves for integrating with tools that we don't even use (such as Sphinx, rST, Doxygen...).

If you want such integrations to happen, be precise.

• Which tool do you want to integrate and how should the integration work?
• What does the output/UI/UX look like?
• What prevents you from building a third-party plugin today?

---

This issue is currently way too broad and unactionable, so I'm closing it for now.

But I'll keep the discussion open.

If you are ready to commit time to help us figure out how a specific integration should be built, or why it should be built in Docusaurus core instead of being third-party, I'm listening, and you are free to create a dedicated issue for integrating that specific tool and link it below.

If you are willing to build a third-party integration plugin, I'll be here to help and support you. If you are stuck due to a limitation of our plugin APIs that prevents you from building such integration, please let me know.

[reduckted]

So we want to generate HTML from external tools first, and then insert the HTML into our layout?

This is probably closest to what I would like to achieve.

At the moment I have a Docusaurus site for written documentation and a Storybook site for component documentation and examples. What I would really like to do is have the Storybook pages appear within the Docusaurus site so that I have just a single site for everything.

Storybook can be built as a static website, so I think all that I would need to do is somehow render the static HTML within Docusaurus.

I've done a small amount of React development, but nothing advanced, so I don't know if this is even possible. If it is possible, I'd be happy to create a third-party plugin to do this. Id just need a pointer or two to get me started in the right direction. 😄

[slorber]

@reduckted what prevents you from embedding Storybook pages as iframes inside your Docusaurus docs markdown files directly?

In fact, I'd recommend most people to try that first, since it's often a good-enough solution.

---

https://storybook.js.org/docs/sharing/embed

# Embed storybook example

<iframe
  src="https://5ccbc373887ca40020446347-wtuhidckxo.chromatic.com/iframe.html?id=shadowboxcta--default&viewMode=story&shortcuts=false&singleStory=true"
  width="800"
  height="200"
></iframe>

<iframe
  src="https://5ccbc373887ca40020446347-wtuhidckxo.chromatic.com/?path=/story/shadowboxcta--default&full=1&shortcuts=false&singleStory=true"
  width="800"
  height="260"
></iframe>

If you build your Storybook under a base url, you might also be able to use local URLs such as /myStorybook/iframe.html

Sure, embedding the HTML natively (without an iframe) is better for UX and performance, but it's also way more challenging because it requires custom/tight integration with the tool, and also ensuring that when importing Storybook CSS in Docusaurus there is no conflict.
(Storybook CSS, or your own CSS messes up with Docusauurs CSS and produces unwanted side effects. Sandboxing styles with a Shadow DOM is an option to solve that)

[reduckted]

what prevents you from embedding Storybook pages as iframes inside your Docusaurus docs markdown files directly?

Deep-linking. There are ways to work around that, but it's pretty hacky. I'll try to put together an example to see how the UX is.