Compile multiple packages in a mono-repo in a single compilation step

[bajtos]

In https://github.com/strongloop/loopback-next, we are developing a set of Node.js modules using TypeScript as the language and lerna convention for the monorepo layout.

I am not able to figure out how to configure TypeScript to compile all our packages in a single step, also allowing us to use --watch mode. As a result, whenever we need to transpile our code, we have to run tsc for each package, which adds a lot of friction to our red-green-refactor loop. (I suspect tsc is wasting a lot of time by re-reading type definitions produced by previous tsc runs, which would have been avoided if everything was compiled in one tsc run.)

Let me show a small example to illustrate our project setup.

Source files committed to git:

packages/
  helper/
    src/
      helper.ts
      util.ts

  core/
    src/
      app.ts
      core.ts

Transpiled files that will be eventually published to npmjs.org in two packages (foo and bar):

packages/
  helper/
    lib/
      helper.js
      util.js

  core/
    lib/
      app.js
      core.js

Essentially, I need to configure TypeScript compiler to use outDir=packages/helper/lib when compiling files from packages/helper/src, but use a different outDir value packages/core/lib when compiling files from packages/core/src.

Additional caveats:

• We have interdependencies between our packages, e.g. core depends on helper. lerna is taking care of running the build command in such order that dependencies are built first, dependants second.
• I considered starting multiple tsc --watch processes (one for each package), however think this will not trigger rebuild when a dependency was changed (and rebuilt by another tsc --watch process).

The proposal

Allow outDir option to take a map value configuring different output directories for different source file locations. Alternatively, define a new option (e.g. outDirs or outDirMap).

{
  "outDir": {
    "packages/foo/src/**": "packages/foo/lib",
    "packages/bar/src/**": "packages/bar/lib",
  }
}

Another alternative that comes to my mind is to allow rootDir and outDir to contain a pattern matcher, where the value of the matcher from rootDir will be applied to outDir.

{
  "rootDir": "packages/*/src",
  "outDir": "packages/*/lib",
}

I understand this may be out of scope of TypeScript project (as mentioned in #7752). In which case, is there a compiler API allowing users to control the mapping from source to output files in such fine-grained way?

[ksjogo]

We use webpack and it's entrypoint functionality to compile multiple packages from one repo (and have one watch process). I am not sure if/how types are cached with multiple entrypoints though. The created bundles are consumed by the server, so no third-party is requiring them, so we are fine with bundling everything. You might need to look into the externals configuration option if you want to exclude further things.

[breathe]

I'm pondering this same issue ... I want to convert a monorepo project to use lerna, but I don't want to have a separate compilation context for each package (less project setup, more type checking, more better!).

I'm considering just turning off outDir and letting the typescript compiler generate the .js files adjacent to the source .ts files ...

The only thing I don't like about that solution (that I can think of) is not being able to do a 'safe clean' ... I could define 'safe clean' as remove all .js files from all src/ directories -- but that would prevent incrementally converting js -> ts when necessary ...

Other than that issue -- do think this would work? I haven't thought through @types/ discovery -- not sure if that would magically work right ...

[aluanhaddad]

@breathe if you are incrementally converting, why not leave --outDir enabled and add --allowJs which will be safe. TypeScript will then process both kinds of sources.

[breathe]

@aluanhaddad -- I don't necessarily want to hijack this issue -- but I normally do use --outDir. My above contemplation was about turning off --outDir so that compilation artifacts from child projects in a mono-repo would end up in the correct place.

E.g. -- instead of the parent's desire:

packages/
  helper/
    src/
      helper.ts
      util.ts

  core/
    src/
      app.ts
      core.ts

With transpiled files that will be eventually published to npmjs.org in two packages (foo and bar):

packages/
  helper/
    lib/
      helper.js
      util.js

  core/
    lib/
      app.js
      core.js

Not using --outDir and organizing as follows:

packages/
  helper/
    lib/
      helper.ts
      helper.js
      util.ts
      util.js

  core/
    lib/
      app.ts
      app.js
      core.ts
      core.js

The benefit being a (maybe?) workable strategy for getting a single compilation context for a mono-repo -- with the drawback of making '--allowJs' less safe -- or at least making clean operations less safe.

[DanielRosenwasser]

Hey @bajtos, sorry for the delay here. We've been trying to revisit this sort of scenario a little bit more closely. #3469 documents some of the ideas in play here. While we don't have a concrete solution at this time, we're definitely trying to scope out how to solve this scenario.

[bajtos]

Hello @DanielRosenwasser, thank you for chiming in! I think project references should address our needs pretty well. I am proposing to keep this issue open until #3469 is released and I can verify that it does support what we need. Thoughts?

[jonaskello]

I think a simple solution for monorepo compiles and watch would be to allow multiple -p path/to/tsconfig.json on tsc.

You can have a base tsconfig.json in the root and one in each package that is empty except it extends the one in the root.

If -p would also support globbing it would be really simple to compile and watch all packages with at single invocation of tsc:

tsc --watch -p ./packages/*/tsconfig.json

In case of globbing there is a need to establish compilation order, this could be done by looking for a package.json for each tsconfig.json and then sort out which package is a dependency on another package.

[harrysolovay]

Any progress on this? It would be great to simplify monorepo packages' ts compilation by having a single root-level tsconfig, and dev and build scripts.

[dvdzkwsk]

First-class support would be wonderful, but in case it helps anybody else, here's a workaround we implemented recently:

A project I'm working on (a fairly large monorepo) is not yet able to upgrade to use project references, and until very recently we were running tsc in each package (parallelized, to a degree). This was, as you've all pointed out, quite slow.

Because all of the packages extend same tsconfig, what we did was update the root tsconfig to build to a dist folder in the root, using rootDir so that it mirrored the monorepo directory structure. Once built, a simple script traveresed the directory and copied the emitted files into their respective workspace's own dist folder.

.
├── dist
│   └── packages
│       ├── foo            # this moves to packages/foo/dist
│       │  └── index.js
│       └── bar            # this moves to packages/bar/dist
│          └── index.js
└── packages
        ├── foo
        |   └── index.ts
        └── bar
            └── index.ts

This change dropped build times from 13 minutes for all packages to 1.5 minutes. It required some rewriting of sourcemap paths, but that was a fairly similar process (read all sourcemaps and rewrite sources to be relative to the workspace instead of rootDir). Unfortunately I'm not able to share the exact code because it's for internal build systems, but hopefully that helps someone.

[harrysolovay]

@davezuko that's a neat workaround! I'm surprised though that it resulted in a major built-time reduction––it seems like it's still having to watch each directory and recompile all packages upon any single package change(?).

Unrelated: have you used ts-node's register at all? I'm thinking that I might ditch pre-compilation all-together & keep my TS closer to the wire. According to what I've read, it's just as fast. Getting rid of the build step is really appealing to me. Was wondering if you had any thoughts? Thanks!

[dvdzkwsk]

@harrysolovay so our use-case may be a bit unique. Our standard workflow involves only a subset of our repo's packages at a time (through webpack). Because of this, a change to one package doesn't necessitate rebuilding all others.

Where the strategy I mentioned above comes into play is in full monorepo builds, such as in PR or CI environments. In these cases, we have to validate a developer's changes against all other packages in the repo, and not just those their package was bundling. It was this stage that was quite slow, mostly because each package was being treated individually, leading to a lot of duplicated type checking and time spent re-initializing the TypeScript compiler.

It was a solution to our needs, but it may not be to yours. I'd been struggling with this problem for a while, and having not seen much out there I figured I'd at least share our idea :).

--

Regarding ts-node, not in any serious work. However, I have dealt with its counterpart babel-register quite a bit, so I can maybe offer a related perspective.

Complicating your development workflow with a pre-compilation stage is never fun. I personally dislike having to produce intermediary files just to run the code, and so I'd use ts-node for the development cycle if nothing else. When it comes to deploying your code to the server, even then I'd argue a build step isn't worth the complication until you can prove that the ts-node overhead is a legitimate bottleneck.

[harrysolovay]

@davezuko by "intermediary file", do you mean the file that calls babel-register and requires the entry point? If not for the intermediary file, would you feel differently? For servers and SSRed clients, the initial build time shouldn't matter too much. It kind of seems like these "pirated" runtimes get rid of a development stage (no need to trigger watch mode or builds, you get to ship your source code, and you don't need to worry about sub-directories to differentiate src vs. dist). Is there a change to the babel-register or ts-node DXs that would make it worth using in your next project? I would love to hear more of your thoughts on this :)

[dvdzkwsk]

@harrysolovay, by "intermediary files" I was referring to development workflows that don't use on-the-fly transpilation; i.e. ones that compile the source to new files on disk (src/index.js -> dist/index.js) and then run those. The file you are thinking of — the one that loads babel-register — can be omitted by running node -r babel-register src/index.ts, for example.

We seem to be in agreement about the workflow. I would use ts-node or a require hook until it proved to be a true bottleneck. I'd bet it would be a while before that becomes the case.

In the interest of keeping this issue on track, you're welcome to shoot me an email if you want to discuss these workflows more :) (listed on my GitHub profile).

[harrysolovay]

@davezuko thank you so much for the insight –– gonna email you about an idea :)

[RyanCavanaugh]

I think project references and their associated features are either the solution to this, or as close as we're going to get. Customizing the emit location logic any further should happen at a compiler-hosting level.

[chanaksha]

Just so it may be useful for newbies like me -- you may choose to write your own wrapper invoking the typescript compiler cli as separate process to make it work in your development setup (or use something like concurrently.

The wrapper I wrote looks like this (as concurrently would be lots of cd and npm scripts if there are too many packages)

export class TypeScriptBuildWatcher {
  private tscProcess: ChildProcessWithoutNullStreams;
  private lastOutput: string;
  private projectCode: string;

  constructor(private readonly projectPath) {
    this.projectCode = path.basename(this.projectPath);
  }

  watch(callback: Function): void {
    console.log(`[MONITOR] Watching ${this.projectCode}`);
    const args = 'tsc -w --preserveWatchoutput';

    this.tscProcess = spawn('npx', args.split(' '), {
      cwd: this.projectPath
    });

    this.tscProcess.stdout.on('data', data => {
      this.lastOutput = data.toString();
      process.stdout.write(this.toLine(this.lastOutput));
      callback();
    });

    this.tscProcess.stderr.on('data', data => {
      process.stdout.write(this.toLine(data.toString()));
    });
  }

  private toLine(data: string): string {
    return `[${this.projectCode}] ${data.trim()}\n`;
  }

  get isReady(): boolean {
    return this.lastOutput && this.lastOutput.indexOf('Found 0 errors') > -1;
  }
}

You could create multiple instances of TypeScriptBuildWatcher as buildWatchers in some separate class and pass callback to it for acting on change events (no nodemon, no chokidar)

  public watch(): void {
    this.buildWatchers.forEach(buildWatcher =>
      buildWatcher.watch(() => this.act())
    );
  }

  private act(): void {
    if (this.buildWatchers.every(buildWatcher => buildWatcher.isReady)) {
      console.log('[MONITOR] Starting server as everything looks good.');
      this.server.start();
    } else {
      if (this.server.isRunning) {
        console.log('[MONITOR] Stopping server as changes detected.');
        this.server.stop();
      }
    }
  }

[dandv]

I think project references and their associated features are either the solution to this, or as close as we're going to get.

Thanks @RyanCavanaugh for the sample project-references repo. I found it more useful than the docs.

Is there a way to use project references to import package.json from the parent directory of src? The use case is to import the version from package.json and log it when scripts start.