feat: Docs vNext (Docusaurus Migration)

[kamranayub]

===:clipboard: PR Checklist :clipboard:===

• 📌 issue exists in github for these changes
• 🔬 existing tests still pass
• 🙈 code conforms to the style guide
• 📐 new tests written and passing / old tests updated with new scenario(s)
• 📄 changelog entry added (or not needed)

==================

Closes excaliburjs/excaliburjs.github.io#383

Preview Site

👉 https://excaliburjs.pages.dev/

Local Dev

cd site
npm install
npm start

Changes:

• Added site/ directory to hold Docusaurus site

Task Tracking

Foundations Phase

Everything that will enable the full migration to work

• Add Typedoc API integration and API reference section
• Test linking from MDX to APIs
  • fix: initial build does not link symbols (affects first local build + deployed site)
• Migrate and test our custom MDX components
  • Note replaced with Admonitions
  • CodeSandboxEmbed
  • IFrameEmbed
  • Example storybook embed
  • embed: snippets (Live Code?)
• Move over all the previous MD/X files
• Scaffold the other static pages
• Integrate Tailwind
  • Tutorial: How to add Tailwind 3 to Docusaurus 2 in 2022 swyxio/swyxdotio#456
  • Do we "really" need it, or could we make do with Infima until official preset/theming is in?
• Upgrade to Docusaurus v3
  • TypeScript 5.x
  • Typedoc 0.25.3
• Make sure all the docs are working and links are updated
• Figure out more integrated way to provide compiled, interactive side-by-side examples
• Update community showcase
• Update donation/support page
• Migrate blog posts and come up with a workflow that will work well (+ newsletter?)
• Replace Storybook examples with new Playgrounds
  • Upgrade to Storybook 7.5.x
  • Migrate examples
• Investigate social cards
• Add product analytics (Posthog)
• Make new homepage

Docs Buildout Phase

• Organize old docs into new unified docs system
• Redo and update the Quickstart
• Update or make new diagrams (Mermaid?)
• Bring in community tutorials/resources
• Figure out old API versioning build flow
• Provide at least one recipe/learning path beyond the Quickstart

Site Buildout Phase

Notes

• Separate tooling: Right now the site directory is managed separately from the rest of the engine package building. I may explore unified tooling using a monorepo approach, but it will require moving folders around + redoing all the build tooling. If we move Docusaurus to run/build from the root of the repo, there are some conflicts with the build temp directory, as well as docs directory and src directories. We'd have to change those and I haven't looked into it yet.

• Cloudflare root directory: Since the Docusaurus build needs to reach into other parts of the repo (hello Typedoc), I was getting errors when I set the Pages build root to site. I had to remove that and then modify the build command to cd site && npm ci && npm run build so the build takes longer. It'd be great not to have to (but we may need to do that if we cannot access the repo files outside the site dir).

• Initial build doesn't link symbols: Fixed. Since the docusaurus-plugin-typedoc-api runs as a plugin lifecycle, and we need the .json Typedoc file as input into the docs.remarkPlugins config, it has to wait until the first successful build to get the JSON. I am not sure if there's a way to fix this without modifying the other plugin. 🤔

• Custom symbol link builder: I had to patch my https://github.com/kamranayub/remark-typedoc-symbol-links package to support custom link building since it was hardcoded to Typedoc paths. I could extract this into a npm package perhaps. It would be ideal to bake it into docusaurus-plugin-typedoc-api though 🤔 Not sure how to handle the remarkPlugin piece with @mdx/loader.

• Integrated Storybook build: Previously we had to run npm run visual or npm run build-storybook to work with Storybook examples. I have a POC that adds direct dev server + build integration with Docusaurus so everything happens in one "experience." This will move SB to the site directory instead of the root, keeping things tidy. It will also allow us to move the stories into the examples/ folders within docs, making story creation even more seamless. There is some uncertainty with how the SB dev server will interact with the Docusaurus dev server, and process file handles, exiting gracefully, etc.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	90a4f71
Status:	✅  Deploy successful!
Preview URL:	https://5c2dfdd5.excaliburjs.pages.dev

View logs

[kamranayub]

Notes for live code research

Ideal DX: write TS code files that are compiled alongside MDX files, and can be embedded, with live code + preview.

Approach: Storybook

I believe Storybook will get us that but the challenge is that the build pipeline is separate. However, it feels possible to integrate Storybook's build as a Docusaurus plugin.

• Builder API: https://storybook.js.org/docs/html/builders/builder-api
• Custom framework: https://storybook.js.org/docs/html/api/new-frameworks
• Live code add-on: https://github.com/hirohe/storybook-addon-jarle-monaco

Results

As of b42edc1 I got SB integrated as a plugin, with a side-by-side dev server.

I think it will be too tough right now at least to get seamless SB integration working. The issue is that the Storybook dev server and compilation pipeline is tightly coupled. Unlike Typedoc, we can't just call an easy API to get JSON back to do custom rendering. Plus I'd need to basically rebuild all the UI even if we did.

I also question whether we may need SB if we can get a playground working properly. Because then all we really "need" is the "Controls" support SB provides and we can write that ourselves or build it into the docs.

Approach: react-live

Docusaurus already provides a theme plugin for live-code via React Live. You can then use the raw-loader Webpack plugin to load the raw contents of a TS file directly into a <CodeBlock live> MDX component.

This means we would write snippets in TS, compile them, and they would be interactive and live at runtime.

As of bd7d7a0, this POC approach is working.

Downsides

• No snippet support (hiding extra context)
• No extra nice-to-haves like copying code, reporting an issue, resizing the canvas
• Embedded canvas, instead of a sandboxed iframe, which may result in conflicting engine state
• No "official" TS support (but seems to work nonetheless)

Results

Seems workable. To mitigate the embedded canvas, I believe we could house the CodeBlock theme component on a dedicate iframe route, and the GameCodeBlock can take care of embedding it properly.

Approach: Shiki Twoslash

There's a library called Shiki Twoslash which actually renders TS compiler annotations (and errors!), including snippets and highlighting. There's a docusaurus plugin as well.

Issues

• Trouble integrating with Docusaurus -- fixed with rehype-raw
• Requires fenced code blocks, doesn't interop with MDX components like GameCodeBlock
• VSCode support for MDX is missing: feat(extensions): register commands and providers also for MDX files shikijs/twoslash#173

Results

Update 1: I could not easily integrate the Docusaurus plugin with v3; I was running into errors. Even if I did, we don't get live editing. However, just having the annotations would be really nice for static snippets, so I will still play with this and see if we can't get it to work for static code.

Update 2: As of 647916a, I was able to get Shiki Twoslash working with static code blocks! There is a possible issue with removeChild when navigating between docs/API reference.

Approach: StackBlitz SDK

https://developer.stackblitz.com/platform/api/javascript-sdk

Pros

• Creates sandboxes on-demand
• Intellisense/autocomplete
• Allows us to customize and pass multiple files
• Isolated and supports live editing
• Can be wrapped in a MDX React component

Issues

• Auto-focus scrolls page: Seems like on Actors page, the collision example at the end pulls focus and scroll down to it on load 🤔 This might be an editor issue. Possible issue that used to be fixed: Prevent auto-scroll to iframe stackblitz/core#1653. Workaround: Using clickToLoad seems to work.
• Cannot use commit source code: Since Stackblitz is a standalone environment, it pulls the excalibur npm package instead of the actual current built code. It MAY be possible to pass in the raw source as a file to the SDK. I am being blocked / running into this issue with WDS + HMR.
• Only text files: In order to load assets, we would need to pass as an SVG or provide a npm package of pre-packaged royalty-free excalibur assets (would be a good idea in general).

Downsides

• Base64 Assets: Instead of statically hosting binary assets, we are passing them as base64 data string ES modules. This keeps the example code clean, and avoids having to deploy anything statically, however it increases payload size. Long-term we could look at an asset package.
• Not synced with local edits
• Dependent on Stackblitz being up and around

Results

As of 5d84603 StackBlitz is integrated and working with multiple examples on a page (Actors) as well as image assets (Animations).

This could be a very good approach, if we can figure out how to make local code edits a bit more seamless.

Approach: Sandpack

Thanks to this blog post by Josh, Sandpack could be an option -- but like most of the live playgrounds, it is optimized for JavaScript, not TypeScript.

Sandpack uses CodeMirror 5 (?) but there is an LSP integration that in theory would work with the TS LSP in the browser. I'd need to manually hook it all up.

This is what the new React docs uses.

Pros

• Sandboxed editor
• Flexible layout, styling, and customizations
• Built on CodeSandbox
• Can provide assets
• Can self-host the bundler (no dependency on CodeSandbox)

Issues

• React-based (not too different from current react-live approach)
• No native TypeScript support or LSP integration, however there is a prototype and demo of using it with TS

Approach: Bright (CodeHike)

This might need to be supported by Docusaurus directly, but:

https://bright.codehike.org/

Does support extensions, which presumably would let us inject live editing if we want (plus build other extensions specific to excalibur)

[eonarheim]

@kamranayub MERGED! 🎉 🎉 🎉