diff --git a/README.md b/README.md index 8912b131..34d1bedc 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,7 @@ npm start - **Robust Error Handling**: Docusaurus has excellent error catching, particularly for broken and missing links, reducing the need for manual testing. -![Pro Tip](./website/docs/03-other-software/images/proTip.png "Use Overture Docs repo to search across all Overture repos") +![Pro Tip](./website/docs/02-platform-tools/images/proTip.png "Use Overture Docs repo to search across all Overture repos") > [!TIP] > The Overture Docs repo contains everything, therefore finding & tracking links and content across all our repos has never been easier. diff --git a/submodules/.github b/submodules/.github index 14977cc6..e750770a 160000 --- a/submodules/.github +++ b/submodules/.github @@ -1 +1 @@ -Subproject commit 14977cc64b66f0ca87b0d12472f049e0effa6346 +Subproject commit e750770a2fc9e15a3f5389f9e4d0aa8bee949ac2 diff --git a/submodules/lectern b/submodules/lectern index f65fb1ca..08ad7ec8 160000 --- a/submodules/lectern +++ b/submodules/lectern @@ -1 +1 @@ -Subproject commit f65fb1cab599b1ec59cbe6ad390ea57211e1b07c +Subproject commit 08ad7ec8d70c8fe80ffd7789db18e0c92d4b0dc0 diff --git a/submodules/maestro b/submodules/maestro index b3267fab..e8beeb1b 160000 --- a/submodules/maestro +++ b/submodules/maestro @@ -1 +1 @@ -Subproject commit b3267fabc42fab1f6250ba212b2e7e85fdbe2e37 +Subproject commit e8beeb1b6913a986b59390d50aa4cb0a87818eb4 diff --git a/submodules/song b/submodules/song index 2e01f7bc..5ebe11fa 160000 --- a/submodules/song +++ b/submodules/song @@ -1 +1 @@ -Subproject commit 2e01f7bcf97db303fad64932e36afedc264125be +Subproject commit 5ebe11fae66679d6634ecdd3f7cbbd1e85b4defe diff --git a/submodules/stage b/submodules/stage index 3a56591f..fa63b860 160000 --- a/submodules/stage +++ b/submodules/stage @@ -1 +1 @@ -Subproject commit 3a56591fa06676836b406ca56fe92f1a722bf892 +Subproject commit fa63b86054c853374ae982c85fffc9f5c72f6924 diff --git a/libretto.sh b/symlinker.sh similarity index 100% rename from libretto.sh rename to symlinker.sh diff --git a/website/community/02-funding.md b/website/community/02-funding.md index 8e01e510..2eda4be8 100644 --- a/website/community/02-funding.md +++ b/website/community/02-funding.md @@ -6,22 +6,22 @@ We gratefully acknowledge the financial support that has made this project possi As working group 2, we are developing a federated framework for genomic data access and analysis, specifically building an enhanced Overture data submission system to facilitate the ingestion, validation, and tracking of clinical and molecular data into the PCGL platform. - - **Project Title:** Pan-Canadian Genome Library (PCGL) - - **Funding Period:** 2023-2028 +- **Project Title:** [Pan-Canadian Genome Library (PCGL)](https://genomelibrary.ca/) +- **Funding Period:** 2023-2028 ### National Cancer Institute at the US National Institutes of Health Under this grant we are making the Overture platform more accessible by breaking down barriers to adoption, integrating existing analysis tools and adding federated search functionality between distributed Overture instances using standardized data governance frameworks ([GA4GH](https://www.ga4gh.org/)). - - **Grant Number:** #U24CA253529 - - **Funding Period:** 2021-2026 +- **Grant Number:** #U24CA253529 +- **Funding Period:** 2021-2026 -### Canadian COVID-19 Genomic Data Infrastructure +### Canadian COVID-19 Genomic Data Infrastructure The [VirusSeq Data Portal](https://virusseq-dataportal.ca/) demonstrates Overture's successful expansion beyond oncology data management. This funding facilitated the platform's adaptation to support viral genomic data, while enabling the development, deployment, and ongoing maintenance of Canada's central SARS-CoV-2 sequence repository. - - **Project Title:** VirusSeq Data Portal - - **Funding Period:** 2020-2024 +- **Project Title:** VirusSeq Data Portal +- **Funding Period:** 2020-2024 ### The Government of Ontario @@ -37,4 +37,4 @@ We also thank the following organizations for their contributions: For more information about our funding or to discuss potential collaborations, please email us at contact@overture.bio. -**Last updated:** 11-19-24 \ No newline at end of file +**Last updated:** 11-19-24 diff --git a/website/docs/00-getting-started.mdx b/website/docs/00-getting-started.mdx index 323c740d..5b754d1d 100644 --- a/website/docs/00-getting-started.mdx +++ b/website/docs/00-getting-started.mdx @@ -18,13 +18,13 @@ Our documentation is the definitive resource for the Overture project. Here you' { type: 'link', label: 'Other Software', - href: '/docs/other-software/', + href: '/docs/platform-tools/', description: 'Software that complements and supports Overture.' }, { type: 'link', label: 'Standards', - href: '/docs/standards/', + href: '/docs/documentation-standards/', description: 'The formal basis of our Overture documentation and developement practices.' }, diff --git a/website/docs/01-core-software/00-Lectern b/website/docs/01-core-software/00-Lectern new file mode 120000 index 00000000..bf9c629d --- /dev/null +++ b/website/docs/01-core-software/00-Lectern @@ -0,0 +1 @@ +../../../submodules/lectern/docs/ \ No newline at end of file diff --git a/website/docs/01-core-software/01-core-software.mdx b/website/docs/01-core-software/01-core-software.mdx index d34e460a..3901907e 100644 --- a/website/docs/01-core-software/01-core-software.mdx +++ b/website/docs/01-core-software/01-core-software.mdx @@ -1,6 +1,6 @@ # Core Software -![Core Software](./images/core-software.webp 'Core Software') +![Core Software](./images/core-software.webp "Core Software") :::info Overture is a genomics data platform that combines front and back-end services. Stage handles navigation, while Arranger enables data search. Authentication runs through Keycloak/Ego, Song manages metadata, Score handles S3 file transfers, and Maestro indexes to Elasticsearch for Arranger's search functionality. Together, these components enable efficient genomics data management. @@ -10,46 +10,44 @@ Overture is a genomics data platform that combines front and back-end services. Our core software documentation is organized as follows: - ``` - . - └── /Service/ - ├── /Overview - ├── /Setup - └── /Usage - ``` +``` +. +└── /Service/ + ├── /Overview + ├── /Setup + └── /Usage +``` ### Overview -The Overview section for each service typically includes: - - High-level overview of the service - - System architecture diagram and explanation - - Key features and capabilities - - Repository structure - - Links to relevant GitHub repositories or additional resources +The Overview section for each service typically includes: - High-level overview of the service - System architecture diagram and explanation - Key features and capabilities - Repository structure - Links to relevant GitHub repositories or additional resources -### Setup +### Setup The Setup section runs you through the process of getting each service setup in a development environment. We offer guidance for the following Overture development configurations: - - ![Platform](../03-other-software/images/platform.svg 'Overture Platform') + ![Platform](../02-platform-tools/images/platform.svg 'Overture Platform') - ![Song Dev](../03-other-software/images/songDev.svg 'Song Dev Environment') + ![Song Dev](../02-platform-tools/images/songDev.svg 'Song Dev Environment') - ![Score Dev](../03-other-software/images/scoreDev.svg 'Score Dev Environment') + ![Score Dev](../02-platform-tools/images/scoreDev.svg 'Score Dev + Environment') - ![Maestro Dev](../03-other-software/images/maestroDev.svg 'Maestro Dev Environment') + ![Maestro Dev](../02-platform-tools/images/maestroDev.svg 'Maestro Dev + Environment') - ![Arranger Dev](../03-other-software/images/arrangerDev.svg 'Arranger Dev Environment') + ![Arranger Dev](../02-platform-tools/images/arrangerDev.svg 'Arranger Dev + Environment') - ![Stage Dev](../03-other-software/images/stageDev.svg 'Stage Dev Environment') + ![Stage Dev](../02-platform-tools/images/stageDev.svg 'Stage Dev + Environment') @@ -60,6 +58,7 @@ For newcomers to Overture, we recommend starting with our Platform Quickstart an ### Usage The Usage section provides detailed information on how to effectively use each service, including: + - Interacting with the API - Common use cases and best practices - Troubleshooting and FAQs @@ -69,5 +68,3 @@ The Usage section provides detailed information on how to effectively use each s ### Contents - - diff --git a/website/docs/03-other-software/01-prelude.md b/website/docs/02-platform-tools/01-prelude.md similarity index 100% rename from website/docs/03-other-software/01-prelude.md rename to website/docs/02-platform-tools/01-prelude.md diff --git a/website/docs/03-other-software/03-Other-Software.mdx b/website/docs/02-platform-tools/02-platform-tools.mdx similarity index 90% rename from website/docs/03-other-software/03-Other-Software.mdx rename to website/docs/02-platform-tools/02-platform-tools.mdx index 79d6c27f..22e0b532 100644 --- a/website/docs/03-other-software/03-Other-Software.mdx +++ b/website/docs/02-platform-tools/02-platform-tools.mdx @@ -1,7 +1,5 @@ -# Other Software +# Platfrom Tools This section provides information on supplemental software that, while not directly integrated into the Overture platform, is part of the broader Overture project. Although primarily used for internal purposes, feel free to explore the documentation for these tools that support our open-source initiative. - - - \ No newline at end of file + diff --git a/website/docs/03-other-software/02-quickstart.mdx b/website/docs/02-platform-tools/02-quickstart.mdx similarity index 100% rename from website/docs/03-other-software/02-quickstart.mdx rename to website/docs/02-platform-tools/02-quickstart.mdx diff --git a/website/docs/03-other-software/images/DevelopmentPhases.png b/website/docs/02-platform-tools/images/DevelopmentPhases.png similarity index 100% rename from website/docs/03-other-software/images/DevelopmentPhases.png rename to website/docs/02-platform-tools/images/DevelopmentPhases.png diff --git a/website/docs/03-other-software/images/arrangerDev.svg b/website/docs/02-platform-tools/images/arrangerDev.svg similarity index 100% rename from website/docs/03-other-software/images/arrangerDev.svg rename to website/docs/02-platform-tools/images/arrangerDev.svg diff --git a/website/docs/03-other-software/images/maestroDev.svg b/website/docs/02-platform-tools/images/maestroDev.svg similarity index 100% rename from website/docs/03-other-software/images/maestroDev.svg rename to website/docs/02-platform-tools/images/maestroDev.svg diff --git a/website/docs/03-other-software/images/platform.svg b/website/docs/02-platform-tools/images/platform.svg similarity index 100% rename from website/docs/03-other-software/images/platform.svg rename to website/docs/02-platform-tools/images/platform.svg diff --git a/website/docs/03-other-software/images/scoreDev.svg b/website/docs/02-platform-tools/images/scoreDev.svg similarity index 100% rename from website/docs/03-other-software/images/scoreDev.svg rename to website/docs/02-platform-tools/images/scoreDev.svg diff --git a/website/docs/03-other-software/images/songDev.svg b/website/docs/02-platform-tools/images/songDev.svg similarity index 100% rename from website/docs/03-other-software/images/songDev.svg rename to website/docs/02-platform-tools/images/songDev.svg diff --git a/website/docs/03-other-software/images/stageDev.svg b/website/docs/02-platform-tools/images/stageDev.svg similarity index 100% rename from website/docs/03-other-software/images/stageDev.svg rename to website/docs/02-platform-tools/images/stageDev.svg diff --git a/website/docs/03-other-software/tabs.css b/website/docs/02-platform-tools/tabs.css similarity index 100% rename from website/docs/03-other-software/tabs.css rename to website/docs/02-platform-tools/tabs.css diff --git a/website/docs/02-under-development/01-lectern b/website/docs/02-under-development/01-lectern deleted file mode 120000 index 49dbeda4..00000000 --- a/website/docs/02-under-development/01-lectern +++ /dev/null @@ -1 +0,0 @@ -../../../submodules/lectern/docs/overview \ No newline at end of file diff --git a/website/docs/02-under-development/02-lyric b/website/docs/02-under-development/02-lyric deleted file mode 120000 index 023ba46c..00000000 --- a/website/docs/02-under-development/02-lyric +++ /dev/null @@ -1 +0,0 @@ -../../../submodules/lyric/docs/ \ No newline at end of file diff --git a/website/docs/02-under-development/02-under-development.md b/website/docs/02-under-development/02-under-development.md deleted file mode 100644 index 163342d4..00000000 --- a/website/docs/02-under-development/02-under-development.md +++ /dev/null @@ -1,17 +0,0 @@ -# Under Development - -As part of our work on the [Pan-Canadian Genome Library](https://oicr.on.ca/first-ever-national-library-of-genomic-data-will-help-personalize-cancer-treatment-in-canada-and-around-the-world/) we are currently developing a **new data submission system** to better -support tabular (clinical) data. - -This new system will see two new components integrated into the Overture Platform: - - - [**Lectern**](/docs/under-development/lectern/): A version-controlled Data Dictionary Schema Manager that defines, validates, and manages schemas used for data submissions, supporting data integrity across evolving data management systems. - - [**Lyric**](/docs/under-development/lyric/): Validate, store, query, and re-validate tabular data against evolving Lectern Dictionary schemas. - -Additionally, we are updating our indexing service: - - - [**Maestro v5**](https://github.com/overture-stack/maestro/tree/M5) will see multiple updates including the ability to index data from Song and Lyric into one elasticsearch index. - -Combined together the new Overture Platfrom architecture will look as follows: - -![Submission System Architecture](./images/submission-system.svg 'Updated Overture Submission System') diff --git a/website/docs/03-other-software/03-libretto.md b/website/docs/03-other-software/03-libretto.md deleted file mode 100644 index 25e813b3..00000000 --- a/website/docs/03-other-software/03-libretto.md +++ /dev/null @@ -1,230 +0,0 @@ -# Libretto - -Libretto is a single, easy-to-navigate hub that beautifully renders all our product documentation from the `/docs` folder of all our GitHub repositories. - -## Documentation Framework - -Overture has three user profiles: - -- **Software Engineers**: They build and customize Overture components for their software stack. - -- **IT specialist**: Deploying and configuring Overture platforms. - -- **Informaticians**: Use Overture platforms to gather, organize, explore and share data. - -Our documentation is split up as follows: - -| Documentation | User Profile | Description | -| ----------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------- | -| Product Documentation (Housed here) | Software Engineers & Developers | Technical resources for those working on or contributing to the project. | -| Platform Guides (Overture.bio) | IT Specialists & Informaticians | Instructive guides covering platform setup, maintenance and usage for end-users and administrators. | - -## How Overture Docs Works - -- **Docusaurus**: We use Docusaurus to render the site, providing a sleek and navigable interface for our documentation. - -- **Markdown Files**: All documentation content is stored as markdown files in the `/website/docs` directory. - -- **Git Submodules**: We use Git submodules to store and track all our GitHub repositories within one main repository. All submodules can be found in the `submodule` folder. - -- **Symlinks**: Only the necessary documentation files are symlinked from the submodules in the `submodule` folder to the `/website/docs` directory. This allows us to import only the required markdown content that Docusaurus needs. - -## Benefits of this Setup - -- **A Centralized Resource for Decentralized Documentation**: A single, easy-to-navigate hub displaying all our developer documentation while keeping all documentation markdown files within their respective repositories. - -- **Consistent**: Enables us to easily ensure all documentation follows the same standards across different projects. - -- **Easy to Maintain**: Updates to any of the individual project repositories `/docs` folders are automatically reflected here. - -- **Robust Error Handling**: Docusaurus has excellent error catching, particularly for broken and missing links, reducing the need for manual testing. - -![Pro Tip](./images/proTip.png "Use Overture Docs repo to search across all Overture repos") - -:::tip Pro Tip -The Overture Docs repo contains everything, therefore finding & tracking links and content across all our repos has never been easier. -::: - -## Getting Started - -### Running it Locally - -To clone the repository with the files in the submodules: - -```bash -git clone --recurse-submodules https://github.com/overture-stack/docs.git -``` - -Install required dependencies: - -``` -npm ci -``` - -Start the server - -``` -npm start -``` - -> **Note:** Docusaurus requires node version 18 or higher - -### Adding Submodules - -Use the following command from the submodules directory to add a new submodule: - -```bash -git submodule add -b module_name -``` - -This will automatically update the `.gitmodules` file located in the root directory. - -### Updating Submodules - -To pull the latest changes for all submodules, including any newly added ones run: - -```bash -git submodule update --remote -``` - -### Using Symlinks - -All documentation content is stored in markdown files located within `/website/docs` while the original repositories for these markdown files are in the `submodule` folder. - -Symlinks are used here to link only the necessary folders containing markdown files from the submodule directories. - -The script for this is in the root directory, titled `Libretto.sh`: - -```bash -ln -s ../../submodules/song/docs website/docs/Song -ln -s ../../submodules/score/docs website/docs/Score -``` - -This allows us to import only the necessary markdown files that Docusaurus needs. Changes to either directory are linked and reflected in both. - -#### Getting it to run with Docusaurus - -To run this without errors, I needed to create a plugin found in `website/docsPlugin.ts`: - -```typescript -module.exports = function (context, options) { - return { - name: "custom-docusaurus-plugin", - configureWebpack(config, isServer, utils) { - return { - resolve: { - symlinks: false, - }, - }; - }, - }; -}; -``` - -This plugin is imported on line 32 of the `docusaurus.config.ts`: - -```typescript -plugins: ['./docsPlugin.ts'], -``` - -The source of this fix can be found [here](https://github.com/facebook/docusaurus/issues/3272#issuecomment-688409489). - -## Globally imported Mdx components - -We can import components by default across all our mdx files reducing the head matter when using these components in mdx documents. This can be configured from the `/src/theme/MDXComponents.ts`: - -```ts title="MDXComponents.ts" -import type { ComponentType } from "react"; -import MDXComponents from "@theme-original/MDXComponents"; -import Tabs from "@theme/Tabs"; -import TabItem from "@theme/TabItem"; - -const components: typeof MDXComponents & { - Tabs: ComponentType; - TabItem: ComponentType; - DocCardList: ComponentType; -} = { - ...MDXComponents, - Tabs, - TabItem, - DocCardList, -}; - -export default components; -``` - -## Custom Components - -There are three custom components built for this site all located in the components directory as follows: - -``` -. -└── /src/ - └── components/ - ├── FundingStatment - ├── SiteMap - └── SwaggerAPIDoc -``` - -### Site Map - -- The sitemap component renders the frontpage navigation of the website, organized in a mosaic layout with left and right columns -- Categories can be added by extending the `const categories:` object (lines 24-45) with new entries containing: - - `title`: Category display name - - `description`: Brief category description -- Products are defined in `const products:` array (lines 47-61) with each product requiring: - - `title`: Product name - - `link`: URL path - - `description`: Brief description - - `category`: Must match a category key - - `image`: Optional icon path - -The layout groups products into their respective categories and automatically distributes them between the left column (`core`, `development`) and right column (`platform`, `misc`, `standards`). - -This component also includes our funding statement, the copy used can be updated directly from the component on line 90. - -:::tip Improvement Consideration -The current implementation uses hardcoded arrays (leftColumnCategories and rightColumnCategories) for column distribution. This is not ideal however it is the simplest method to organize our site content in its ideal layout. -::: - -### Sidebar Funding Statement - -A simple React component that displays funding attribution information on the bottom left of all sidebars. It uses CSS Modules for scoped styling (styles.module.css). - -:::tip Improvement Consideration -Consider making the content configurable by accepting props rather than hardcoding the grant information, allowing reuse for different funding sources. -::: - -### Swagger API Embed - -The `SwaggerAPIDoc` component renders API documentation completely client-side by: - -1. Importing local JSON specification files (`songAPI.json`, `scoreAPI.json`, `maestroAPI.json`) -2. Using these static JSON files to generate the documentation UI, meaning: - - No actual API calls are made - - Documentation works offline - - No backend connectivity required - - Safe for internal/private API documentation - - Specifications can be version controlled alongside the code - -Usage example with local JSON: - -```jsx -// Your JSON spec file (e.g., songAPI.json) -{ - "openapi": "3.0.0", - "info": { - "title": "Song API", - // ... rest of your API specification - } -} - -// Component usage remains the same - -``` - -The `tryItOutEnabled={false}` setting further reinforces this by preventing any attempts to make actual API calls from the documentation interface. - -:::tip Improvement Consideration -Consider making the content configurable by accepting a path prop rather than having to add the spec to the component itself. -::: diff --git a/website/docs/03-other-software/images/proTip.png b/website/docs/03-other-software/images/proTip.png deleted file mode 100644 index cbfb7f0f..00000000 Binary files a/website/docs/03-other-software/images/proTip.png and /dev/null differ diff --git a/website/docs/03-under-development/01-Lyric.md b/website/docs/03-under-development/01-Lyric.md new file mode 100644 index 00000000..7e3ddc66 --- /dev/null +++ b/website/docs/03-under-development/01-Lyric.md @@ -0,0 +1,61 @@ +# Lyric + +Lyric is a tabular data management service designed to handle structured clinical and research data. Built on top of [Lectern's](https://docs.overture.bio/docs/under-development/lectern/) dictionary framework, it provides a system for organizations to submit, validate, and manage structured data according to predefined schemas. While primarily used for clinical data management, Lyric's architecture remains domain-agnostic, allowing it to handle any type of structured data that can be defined within a Lectern dictionary. + +## Key Features + +- **Schema-driven validation:** Uses Lectern dictionaries to enforce data structure and relationships, validating submissions of tabular data files against predefined schemas. +- **Flexible submission workflow:** Provides a staged submission process where users can iteratively update and validate their data before committing to the database. +- **Comprehensive data management:** Offers complete CRUD operations (Create, Read, Update, Delete) through a RESTful API documented in Swagger. +- **Detailed change history:** Maintains a complete audit trail of all data modifications, tracking changes from committed submissions and updates ensuring data governance and accountability. +- **SQON Query Endpoint:** Provides an endpoint for SQON (Structured Query Object Notation) based queries allowing complex search operations through combinations of simple field operations (`in`, `<=`, `>=`) and logic (`and`, `or`, `not`). This allows complex queries to be expressed in a simple JSON format. +- **Multi-dictionary support:** Handles multiple Lectern dictionaries simultaneously, allowing organizations to manage different data categories with distinct schemas while maintaining data integrity and relationships + +## System Architecture + +Lyric manages the submission of tabular data through its API, validating submissions based on Lectern dictionary schemas stored within Lyric and specified on submission. Song will interact with Lyric to confirm the presence of the data in Lyric's database that corresponds to the file metadata being submitted to Song. All Lyric data is stored on the backend within a PostgreSQL database that will be indexed on publication by [Maestro](https://docs.overture.bio/docs/core-software/Maestro/overview) into Elasticsearch documents. + +![Submission System Architecture](./images/submission-system.svg "Updated Overture Submission System") + +:::info Why Elasticsearch? +Storing data in Elasticsearch allows us to build powerful search UI components linked to a flexible GraphQL API facilitated by the Arranger Server. This approach delivers enhanced query performance, advanced full-text search capabilities, and flexible filtering options compared to querying the database directly. +::: + +As part of the Overture, Lyric is typically used with additional integrations, including: + +- **[Lectern](https://docs.overture.bio/docs/under-development/lectern/):** Validates, stores and manages dictionary schemas fetched, stored, and used by Lyric +- **[Maestro](https://docs.overture.bio/docs/core-software/Maestro/overview):** Handles the indexing of data into elasticsearch on publication events +- **[Song](https://docs.overture.bio/docs/core-software/Song/overview) & [Score](https://docs.overture.bio/docs/core-software/Score/overview):** Facilitates the submission, validation and management of file data in object storage (Score) and the corresponding file metadata (Song) in a database + +## Development Roadmap + +1. **Authentication System** + - Implement authentication rules + - Integrate with Keycloak for identity and access management +2. **Song Integration** + - Enable Song to validate data registration with Lyric + - Implement pre-submission file validation checks +3. **Publication Control** + - Integrate indexing functionality with Maestro V5 + - Enable Lyric & Song data searchability through Arranger interface + +## Repository Structure + +The structure of this monorepo is app centric having `apps/` folder to keep deployable applications while `packages/` folder to keep shared libraries. + +``` +. +├── apps/ +│ └── server +├── packages + ├── data-model + └── data-provider +``` + +[Click here to view the Lyric repository on GitHub](https://github.com/overture-stack/lyric) + +- `app/`: Explanation of directory + - `server/`: Explanation of directory +- `packages`: Explanation of directory + - `data-model`: Explanation of directory + - `data-provider`: Explanation of directory diff --git a/website/docs/03-under-development/02-lecternViewer.md b/website/docs/03-under-development/02-lecternViewer.md new file mode 100644 index 00000000..d99e6e2a --- /dev/null +++ b/website/docs/03-under-development/02-lecternViewer.md @@ -0,0 +1,164 @@ +# Lectern Viewer + +The Lectern data dictionary viewer is a React-based component library that will provide user interfaces for viewing and interacting with Lectern data dictionaries. It aims to transform technical JSON schema specifications into accessible, organized displays that make data models and data requirements clear and understandable for researchers and data submitters. + +The Lectern UI is being developed as a standalone component library that can be integrated into any data platform to provide dictionary viewing capabilities. It will consume [Lectern dictionary schemas](../01-core-software/00-Lectern/03-dictionaryReference.md) and render them through two primary interface modes: tabular views for detailed field-by-field exploration and a diagram view for visualizing schema relationships. + +The viewer serves as the presentation layer in the data submission workflow, providing researchers with clear visualization of data requirements before they begin the submission process through integrated platforms. + +The viewer is being developed initially to serve the [Pan-Canadian Genome Library (PCGL)](https://genomelibrary.ca/), a national initiative that will integrate the viewer for communicating data requirements to researchers across Canada's genomics community. The PCGL implementation will serve as a reference for how other research platforms can integrate the viewer components. + +## Field Viewer + +![SchemaViewer](./images/lecternviewer.png) + +The tabular view provides comprehensive field-by-field documentation with expandable sections for detailed validation rules, examples, and metadata. Interactive elements allow users to filter, search, and navigate through complex schema definitions efficiently. + +**Key Features:** + +- **Interactive Schema Tables:** Detailed tabular views showing field definitions, data types, validation rules, and examples with expandable sections for complex restrictions +- **Version Management:** Built-in support for switching between different dictionary versions with change tracking and comparison capabilities +- **Template Downloads:** Generate CSV templates based on schema definitions to streamline data submission workflows +- **Responsive Design:** Modern, accessible interface that works across desktop and mobile devices with support for custom theming + +### Conditional Logic Modal + +![ConditionalLogicModal](./images/clm.png) + +Complex conditional validation rules are presented through modal overlays that break down if-then-else logic into digestible, structured displays that clearly communicate field dependencies and requirements. + +**Key Features:** + +- **Conditional Logic Visualization:** Clear presentation of complex conditional validation rules through modal overlays and structured displays + +## Entity Relationship Viewer + +![ERD Viewer](./images/erdviewer.png) + +The diagram view visualizes schema relationships using interactive flowcharts that demonstrate how different schemas connect through primary and foreign key relationships, helping users understand the overall data model structure. + +**Key Features:** + +- **Entity Relationship Diagrams:** Visual representations of schema relationships using interactive flowcharts that show primary and foreign key connections +- **Component-Driven Architecture:** Built with Storybook for isolated development, testing, and documentation of reusable components + +## Integration Points + +As part of the larger Overture ecosystem, the Lectern UI will be used with additional integrations, including: + +- **Lectern Server:** The REST API service that manages and serves dictionary schemas, providing version control and validation +- **Lyric:** Overture's tabular data submission service that uses Lectern schemas for data validation and quality assurance + +## Development Approach + +The Lectern UI is built using **component-driven development** with [Storybook](https://storybook.js.org/) as the primary development environment. This approach enables: + +- **Isolated Component Development:** Each UI component is developed and tested in isolation, ensuring reliability and reusability +- **Interactive Documentation:** Living documentation that serves both developers and stakeholders with real examples and use cases +- **Design System Integration:** Consistent theming and styling across all components with easy theme switching and testing +- **Collaborative Development:** Non-technical stakeholders can review and provide feedback on components before integration + +## Developer Quick Start + +The codebase leverages [Storybook](https://storybook.js.org/) as the primary development environment for building, testing, and documenting components. This component-driven development approach ensures each piece of the UI is developed in isolation and thoroughly tested before integration. + +To run Storybook: + +1. Install all dependencies: + +```sh +pnpm install +``` + +2. Run Storybook: + +```sh +pnpm storybook +``` + +Storybook will run on port 6006 by default: [http://localhost:6006/](http://localhost:6006/) + +:::tip Why Storybook? +Storybook enables rapid development and testing of UI components in isolation, provides living documentation for the design system, and facilitates collaboration between developers, designers, and stakeholders by offering a visual interface for component exploration. +::: + +## Editing Stories + +All stories are located within the `/stories` directory. + +### Theme Decorator + +All Lectern Components rely on the `LecternThemeProvider`. If this provider is not present, they will render using the default `LecternTheme`. + +In order to help testing Lectern-UI Components with alternate themes, a `themeDecorator` is provided to be used in stories. The `themeDecorator` can be added to any component story as follows: + +```ts +import themeDecorator from "../themeDecorator"; // import from stories/themeDecorator.tsx + +const meta = { + component: MyComponent, + title: "Category Name/My Component", + tags: ["autodocs"], + // INCLUDE THE DECORATOR HERE: + decorators: [themeDecorator()], +} satisfies Meta; +``` + +To help with testing, a global property named `theme` has been set, and a selector has been added to the Storybook Toolbar: + +![Storybook Global Theme Selector Toolbar](./images/global-theme-selector.png) + +The theme selected from the toolbar theme selection tool will be applied to all stories with the `themeDecorator`. + +At the moment there is only one alternate theme provided. It is defined inside the `themeDecorator` file - to test alternate stylings please update this custom theme. + +#### Adding Additional Themes + +To add additional themes in the drop down for the `themeDecorator` to use, you must add an option for the storybook `globalType.theme` property. This is done in the `/storybook/preview.ts`) file: + +```ts +const preview: Preview = { + globalTypes: { + theme: { + description: + "Display theme used for all components with the themeDecorator.", + toolbar: { + icon: "paintbrush", + items: [ + { value: "default", right: "✅", title: "Default Theme" }, + { value: "custom", right: "🎨", title: "Custom Theme" }, + { + value: "newTheme", + right: "{choose an emoji}", + title: "Your New Theme", + }, + ], + }, + }, + }, + // ... +}; +``` + +Now that we have an additional option, the `themeDecorator` must be provided the specified theme. The logic for providing the selected theme to the decorator is handled in the `themeDecorator` file in the function `function getGlobalTheme(globalTheme: string)`. Add a new case for the `globalType.theme` value that you added to `./storybook/preview.ts`. + +```ts +function getGlobalTheme(globalTheme: string): PartialTheme { + switch (globalTheme) { + case "custom": { + return customTheme; + } + case "newTheme": { + // Return your new theme + return {}; + } + default: { + return {}; + } + } +} +``` + +:::note Development Status +The Lectern UI is currently in active development. Features and interfaces are subject to change as we iterate based on community feedback and user testing with research platforms like the Pan-Canadian Genome Library (PCGL). +::: diff --git a/website/docs/03-under-development/03-under-development.md b/website/docs/03-under-development/03-under-development.md new file mode 100644 index 00000000..9e1e471d --- /dev/null +++ b/website/docs/03-under-development/03-under-development.md @@ -0,0 +1,22 @@ +# Under Development + +As part of our work on the [Pan-Canadian Genome Library](https://oicr.on.ca/first-ever-national-library-of-genomic-data-will-help-personalize-cancer-treatment-in-canada-and-around-the-world/) we are currently developing a **new data submission system** to better +support tabular (clinical) data. + +This new system will see two new components integrated into the Overture Platform: + +- [**Lectern**](https://docs.overtue.bio/docs/core-software/lectern/overview): A version-controlled Data Dictionary Schema Manager that defines, validates, and manages schemas used for data submissions, supporting data integrity across evolving data management systems. +- [**Lyric**](https://docs.overture.bio/docs/development/lyric/): Validate, store, query, and re-validate tabular data against evolving Lectern Dictionary schemas. + +Additionally, we are updating our indexing service: + + - [**Maestro v5**](https://github.com/overture-stack/maestro/tree/M5) will see multiple updates including the ability to index data from Song and Lyric into one elasticsearch index. + +Combined together the new Overture Platfrom architecture will look as follows: + +![Submission System Architecture](./images/submission-system.svg "Updated Overture Submission System") + +We have also began development on a series of user interfaces to help communicate data in human readable formats: + +- [Lectern UI](./02-lecternViewer.md) +- Arranger Charts diff --git a/website/docs/03-under-development/images/clm.png b/website/docs/03-under-development/images/clm.png new file mode 100644 index 00000000..3191209a Binary files /dev/null and b/website/docs/03-under-development/images/clm.png differ diff --git a/website/docs/03-under-development/images/erdviewer.png b/website/docs/03-under-development/images/erdviewer.png new file mode 100644 index 00000000..637f122e Binary files /dev/null and b/website/docs/03-under-development/images/erdviewer.png differ diff --git a/website/docs/03-under-development/images/global-theme-selector.png b/website/docs/03-under-development/images/global-theme-selector.png new file mode 100644 index 00000000..1867c238 Binary files /dev/null and b/website/docs/03-under-development/images/global-theme-selector.png differ diff --git a/website/docs/03-under-development/images/lecternviewer.png b/website/docs/03-under-development/images/lecternviewer.png new file mode 100644 index 00000000..bd38b51a Binary files /dev/null and b/website/docs/03-under-development/images/lecternviewer.png differ diff --git a/website/docs/02-under-development/images/submission-system.svg b/website/docs/03-under-development/images/submission-system.svg similarity index 100% rename from website/docs/02-under-development/images/submission-system.svg rename to website/docs/03-under-development/images/submission-system.svg diff --git a/website/docs/04-standards b/website/docs/04-documentation-standards similarity index 100% rename from website/docs/04-standards rename to website/docs/04-documentation-standards diff --git a/website/docusaurus.config.ts b/website/docusaurus.config.ts index 7f5fdddf..b454572a 100644 --- a/website/docusaurus.config.ts +++ b/website/docusaurus.config.ts @@ -187,6 +187,7 @@ const config: Config = { prism: { theme: prismThemes.github, darkTheme: prismThemes.dracula, + additionalLanguages: ["json"], }, algolia: { // application ID provided by Algolia diff --git a/website/src/components/SiteMap/index.tsx b/website/src/components/SiteMap/index.tsx index 0b7e3554..cb89d888 100644 --- a/website/src/components/SiteMap/index.tsx +++ b/website/src/components/SiteMap/index.tsx @@ -25,8 +25,12 @@ interface Category { const categories: Record = { platform: { - title: "Platform Guides", - description: "Bringing it all together with platform focused guides", + title: "Platform Tools", + description: "Bringing it all together", + }, + guides: { + title: "Platform Documentation", + description: "Platform focused guides", }, core: { title: "Developer Documentation", @@ -36,10 +40,6 @@ const categories: Record = { title: "Under Development", description: "New components not quite ready for production", }, - supporting: { - title: "Supporting Software", - description: "Supporting tools within the Overture project", - }, standards: { title: "Documentation Standards", description: @@ -49,105 +49,116 @@ const categories: Record = { const products: Product[] = [ { - title: "Deployment Guides", + title: "Platform Development Toolkit", + link: "/docs/platform-tools/Prelude", + description: "Prelude", + category: "platform", + }, + { + title: "Local Demo Portal", + link: "/docs/platform-tools/Quickstart", + description: "Quickstart", + category: "platform", + }, + { + title: "Deployment Docs", link: "/guides/deployment-guide/", description: "Deploying to Production", - category: "platform", + category: "guides", }, { title: "Administration Guides", link: "/guides/administration-guides/", - description: "Management and Customization", - category: "platform", + description: "Management & Customization", + category: "guides", }, { title: "User Guides", link: "/guides/user-guides/", description: "Interacting with the platform", - category: "platform", + category: "guides", }, { title: "API Reference", link: "/guides/api-reference", description: "Interacting with the platform's APIs", - category: "platform", + category: "guides", + }, + { + title: "Dictionary Management", + link: "/docs/core-software/lectern/overview", + image: iconLectern, + description: "Lectern", + category: "core", }, { - title: "Song", + title: "File Management", link: "docs/core-software/song/overview", image: iconSong, - description: "Metadata Management Service", + description: "Song", category: "core", }, { - title: "Score", + title: "File Transfer", link: "/docs/core-software/score/overview", image: iconScore, - description: "File Transfer Service", + description: "Score", category: "core", }, { - title: "Maestro", + title: "Indexing Service", link: "/docs/core-software/maestro/overview", image: iconMaestro, - description: "Metadata Indexing Service", + description: "Maestro", category: "core", }, { - title: "Arranger", + title: "Search API", link: "/docs/core-software/arranger/overview", image: iconArranger, - description: "Search API", + description: "Arranger", category: "core", }, { - title: "Stage", + title: "Web Portal", link: "/docs/core-software/stage/overview", image: iconStage, - description: "Web Portal Scaffolding", + description: "Stage", category: "core", }, { - title: "Lectern", - link: "/docs/under-development/lectern/", - image: iconLectern, - description: "Schema Management Service", + title: "Dictionary Viewers", + link: "/docs/under-development/lecternViewer/", + description: "Lectern Viewer", category: "development", }, + // { + // title: "Arranger Charts", + // link: "/docs/development/arrangercharts/", + // description: "Visualized Cohort Browsing", + // category: "development", + // }, { - title: "Lyric", - link: "/docs/under-development/lyric/", - image: iconLyric, - description: "Tabular Data Submission Service", + title: "Tabular Data Submission", + link: "/docs/development/lyric/", + description: "Lyric", category: "development", }, { - title: "Quickstart", - link: "https://docs.overture.bio/docs/other-software/Quickstart", - description: "Locally Deployed Demo Portal", - category: "supporting", - }, - { - title: "Libretto", - link: "/docs/other-software/Libretto", - description: "Documentation Site", - category: "supporting", - }, - { - title: "Prelude", - link: "https://docs.overture.bio/docs/other-software/Prelude", - description: "Platform planning & development toolkit.", - category: "supporting", + title: "Docs Site", + link: "/docs/documentation-standards/docsSite", + description: "Documentation website info", + category: "standards", }, { title: "Documenting Projects", - link: "/docs/Standards/github", + link: "/docs/documentation-standards/github", description: "Organization Standards", category: "standards", }, { title: "Documenting Software", - link: "/docs/Standards/Software/", + link: "/docs/documentation-standards/Software/", description: "Software Standards", category: "standards", }, @@ -207,7 +218,7 @@ const SiteMap = () => { return acc; }, {}); - const rightColumnCategories = ["platform", "supporting", "standards"]; + const rightColumnCategories = ["platform", "guides"]; const leftColumnCategories = ["core", "development"]; return ( diff --git a/website/src/components/SiteMap/styles.module.css b/website/src/components/SiteMap/styles.module.css index dfba0583..4a22fa70 100644 --- a/website/src/components/SiteMap/styles.module.css +++ b/website/src/components/SiteMap/styles.module.css @@ -46,7 +46,7 @@ .card { --card-shadow: 0 1px 2px rgba(0, 0, 0, 0.1); --card-shadow-hover: 0 1px 2px rgba(0, 0, 0, 0.15); - + flex: 1 1 calc(50% - 0.5rem); min-width: 100px; display: flex; @@ -110,10 +110,6 @@ gap: 2rem; } -.mosaicLayout .rightColumn .card { - height: 5.86rem; -} - .mosaicLayout .rightColumn .cardTitle { padding-top: 0.5rem; } @@ -180,8 +176,9 @@ width: 50px; height: 50px; } - - .cardTitle, .cardDescription { + + .cardTitle, + .cardDescription { width: 100%; text-align: center; } @@ -189,7 +186,7 @@ .fundingBadge { padding: 1.5rem; } - + .fundingTitle { font-size: 1.5rem; } @@ -203,4 +200,4 @@ text-align: left; align-items: flex-start; } -} \ No newline at end of file +}