Monorepo

Repo https://github.com/tgandrews/ons-monorepo

Current Problems

1. Making a change

Currently when working on author it often requires changes across a number of components to complete a change. This can create a dependency tree of pull requests which creates a number of problems:

• constant context switching
• need reviewers to look at lots of PRs
• reviewers need to pull all the changes to get it running locally
• PR CI is often red as the CI is using already released version of dependencies

2. Sharing is tricky

We have a few components that need to be shared (linting rules and schema) and currently we have a relatively complex process to make changes to these including releasing npm packages.

Proposed Solution

Overview

Import all the existing dependencies into one git repository like here: https://github.com/tgandrews/ons-monorepo

What is a monorepo?

1. https://danluu.com/monorepo/
2. https://hackernoon.com/one-vs-many-why-we-moved-from-multiple-git-repos-to-a-monorepo-and-how-we-set-it-up-f4abb0cfe469

Examples

1. https://github.com/withspectrum/spectrum This is quite interesting as they have hoisted all their dependencies to one top level package.json and then their components are all children inheriting them. This would make their docker images larger as they would contain unnecessary dependencies but would make managing the dependencies easier.
2. https://github.com/babel/babel This is not such a useful example as it is a project focussed on releasing lots of npm packages which does not align with our use case well.

How does this help?

1. A change can be made across all the necessary components in one go without repository overhead.
2. Shared resources can easily be extracted with no unnecessary overhead of having to release npm packages.
3. PR should always be green as they contain everything they are changing.
4. A reviewer can checkout one PR and run everything locally.
5. A single release version. Moving to a monorepo would help when we configure to deploy on release to pre-prod and prod environments. A release would group together versions of the projects that have been proven to work together. With a mult-repo setup, this becomes more difficult and error prone, different repositories could, and would, have different version numbers and the coordination of versions that work together would cause be an additional activity for the development team to have to maintain.

Costs

1. Loss of git history - looks like some tools might help with this.
2. Bigger PRs - but fewer of them 3 PRs x 100 changes vs 1 PR x 300 changes.

Spike investigation

CI

Currently we run all the tests for a given repo in a PR but this is not ideal if we have all components in the project. So implemented a travis matrix in combination with a shell script to determine if a component is in the changeset for the change. This works for the simple set up but will need to get more complex if a shared folder is changed as this could impact all components. Additionally we could run the end-to-end tests for every change as well the component tests to give us greater level of confidence.

• .travis.yml
• build-conditions.sh

Netlify

For author UI changes we spin up netlify environments for both storybook and the app. This makes it easier for reviewers (including designers) to review the changes so we need it to work in a monorepo environment. Netlify allows you to provide a .toml configuration file where we can specify the location of the project and the command to run.

• netlify.toml

CD

Currently our CD pipeline is triggered on a commit to master of a repository. We would have to change this to all look at the same repository then run docker build in the necessary component folder.

Tooling

• Lerna - This is a tool used by babel to manage their huge number of packages and the dependencies between them. However, it is not appropriate for us as our release artefacts are docker images and not npm packages. We are only using them because of our current repo structure at the moment.
• Yarn workspaces - Again this looks like a great tool for managing projects releasing lots of npm packages so is not appropriate as it currently creates just one top level yarn.lock which does not align well with our current docker build.

Feedback

Received the following feedback from Andrew M.

Hey Sam, I had a think about this yesterday and I can definitely see why it’s being considered. Some things we need to consider:

• How do we version and release services?
• How are github issues raised against particular services?
• Do github security checks work with multiple package.json files?

Here is current thinking about how we would address these concerns.

How do we version and release services?

The plan would be to have a single version for “Author” the product for release purposes. The individual services will inherit this version number and would be deployed in step with one another. What we’ve noticed is that the API, GraphQL schema, UI and Publisher all have to change in order to deliver a “feature”. Therefore we’ve concluded that a consistent version that transcends these projects makes sense to ensure that a version proven to work together is released into preprod and prod environments. If we have to make a bugfix then we would use semver system to indicate so by releasing a patch version increment.

How are github issues raised against particular services?

I think the way we would deal with this is by raising issues against the author repo. We could use labels to distinguish between the projects. If we had separate teams around each of the “services” then I think this would be more of a problem. But the reality is that it’s a single team covering each of Author’s components so an issue with Publisher is really an issue with Author and someone on the team would investigate and develop a fix on whichever component that needs to be changed.

Do github security checks work with multiple package.json files?

Issue raised to investigate. https://github.com/tgandrews/ons-monorepo/issues/3