Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way to validate the memory usage of your dts files #34

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 31 additions & 2 deletions source/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,49 @@ const cli = meow(`
Usage
$ tsd [path]

Options
--verify-size, -s Keep on top of the memory usage with a data snapshot
--write, -w Overwrite existing memory usage JSON fixture
--size-delta, -d What percent of change is allow in memory usage, default is 10
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kinda seems like these should be in a subcommand, like tsd memory-usage --write. For example, tsd --write is way too generic naming for the top-level interface. However, we can't really do subcommands as we expect paths. Alternatively, we could namespace the flags: --mem-write, --mem-verify-size, ...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sindresorhus We could however do tsd --cwd=path instead of tsd path so we can still accept subcommands?

Or tsd verify path or something? Not sure, just thinking out loud.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to ideally make it run at the same time as normal tests (rather than have to run it as a 2nd command) - which is why I opted for using flags on the main command.

I copied --write from jest, but I'm open to having it more verbose and specific too 👍


Examples
$ tsd /path/to/project

$ tsd

index.test-d.ts
✖ 10:20 Argument of type string is not assignable to parameter of type number.
`);
`,
{
flags: {
verifySize: {
type: 'boolean',
alias: 'u'
},
write: {
type: 'boolean',
alias: 'w'
},
sizeDelta: {
type: 'string',
alias: 'd',
default: '10'
}
}
}
);

(async () => { // tslint:disable-line:no-floating-promises
updateNotifier({pkg: cli.pkg}).notify();

try {
const options = cli.input.length > 0 ? {cwd: cli.input[0]} : undefined;
const cwd = cli.input.length > 0 ? cli.input[0] : process.cwd();
const options = {
cwd,
verify: cli.flags.verifySize,
writeSnapshot: cli.flags.write,
sizeDelta: parseInt(cli.flags.sizeDelta, 10)
};

const diagnostics = await tsd(options);

Expand Down
54 changes: 38 additions & 16 deletions source/lib/compiler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ import {
Program,
SourceFile,
Node,
forEachChild
forEachChild,
sys
} from 'typescript';
import {Diagnostic, DiagnosticCode, Context, Location} from './interfaces';
import {Diagnostic, DiagnosticCode, Context, Location, RunResults} from './interfaces';

// List of diagnostic codes that should be ignored
const ignoredDiagnostics = new Set<number>([
Expand All @@ -22,7 +23,8 @@ const diagnosticCodesToIgnore = new Set<DiagnosticCode>([
DiagnosticCode.CannotAssignToReadOnlyProperty,
DiagnosticCode.TypeIsNotAssignableToOtherType,
DiagnosticCode.GenericTypeRequiresTypeArguments,
DiagnosticCode.ExpectedArgumentsButGotOther
DiagnosticCode.ExpectedArgumentsButGotOther,
DiagnosticCode.NoOverloadMatches
]);

/**
Expand Down Expand Up @@ -95,19 +97,15 @@ const ignoreDiagnostic = (diagnostic: TSDiagnostic, expectedErrors: Map<Location
};

/**
* Get a list of TypeScript diagnostics within the current context.
* Get a list of TypeScript diagnostics for a program
*
* @param context - The context object.
* @returns List of diagnostics
* @param program - A set-up TypeScript Program
* @returns An array of diagnostics
*/
export const getDiagnostics = (context: Context): Diagnostic[] => {
const fileNames = context.testFiles.map(fileName => path.join(context.cwd, fileName));

const result: Diagnostic[] = [];

const program = createProgram(fileNames, context.config.compilerOptions);
export const getDiagnostics = (program: Program) => {
const results: Diagnostic[] = [];

const diagnostics = program
const diagnostics = program
.getSemanticDiagnostics()
.concat(program.getSyntacticDiagnostics());

Expand All @@ -120,7 +118,7 @@ export const getDiagnostics = (context: Context): Diagnostic[] => {

const position = diagnostic.file.getLineAndCharacterOfPosition(diagnostic.start as number);

result.push({
results.push({
fileName: diagnostic.file.fileName,
message: flattenDiagnosticMessageText(diagnostic.messageText, '\n'),
severity: 'error',
Expand All @@ -130,12 +128,36 @@ export const getDiagnostics = (context: Context): Diagnostic[] => {
}

for (const [, diagnostic] of expectedErrors) {
result.push({
results.push({
...diagnostic,
message: 'Expected an error, but found none.',
severity: 'error'
});
}

return result;
return results;
};

/**
* Get a list of TypeScript diagnostics, and memory size within the current context.
*
* @param context - The context object.
* @returns An object with memory stats, and diagnostics
*/
export const getTypeScriptResults = (context: Context): RunResults => {
const fileNames = context.testFiles.map(fileName => path.join(context.options.cwd, fileName));

const program = createProgram(fileNames, context.config.compilerOptions);
const diagnostics = getDiagnostics(program);

// These are private as of 3.6, but will be public in 3.7 - microsoft/TypeScript#33400
const programAny = program as any;
const sysAny = sys as any;
const stats = {
typeCount: programAny.getTypeCount && programAny.getTypeCount(),
memoryUsage: sysAny.getMemoryUsage && sysAny.getMemoryUsage(),
relationCacheSizes: programAny.getRelationCacheSizes && programAny.getRelationCacheSizes(),
};

return {diagnostics , stats};
};
31 changes: 20 additions & 11 deletions source/lib/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,11 @@ import * as path from 'path';
import * as readPkgUp from 'read-pkg-up';
import * as pathExists from 'path-exists';
import globby from 'globby';
import {getDiagnostics as getTSDiagnostics} from './compiler';
import {getTypeScriptResults as getTSDiagnostics} from './compiler';
import loadConfig from './config';
import getCustomDiagnostics from './rules';
import {Context, Config} from './interfaces';

interface Options {
cwd: string;
}
import {Context, Config, Options} from './interfaces';
import {verifyMemorySize} from './memory-validator';

const findTypingsFile = async (pkg: any, options: Options) => {
const typings = pkg.types || pkg.typings || 'index.d.ts';
Expand Down Expand Up @@ -43,12 +40,21 @@ const findTestFiles = async (typingsFile: string, options: Options & {config: Co
return testFiles;
};

// The default options if you were to run tsd on its own
const defaultOptions: Options = {
cwd: process.cwd(),
verify: false,
writeSnapshot: false,
sizeDelta: 10
};

/**
* Type check the type definition of the project.
*
* @returns A promise which resolves the diagnostics of the type definition.
* @returns A promise which resolves the run results with diagnostics and stats for the type definitions.
*/
export default async (options: Options = {cwd: process.cwd()}) => {
export default async (inputOptions: Partial<Options> = defaultOptions) => {
const options = {...defaultOptions, ...inputOptions};
const {pkg} = await readPkgUp({cwd: options.cwd});

if (!pkg) {
Expand All @@ -66,15 +72,18 @@ export default async (options: Options = {cwd: process.cwd()}) => {
});

const context: Context = {
cwd: options.cwd,
options,
pkg,
typingsFile,
testFiles,
config
config,
};

const tsResults = getTSDiagnostics(context);

return [
...getCustomDiagnostics(context),
...getTSDiagnostics(context)
...tsResults.diagnostics,
...await verifyMemorySize(context, tsResults.stats)
];
};
23 changes: 21 additions & 2 deletions source/lib/interfaces.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,18 @@ export interface Config<Options = CompilerOptions> {
export type RawConfig = Partial<Config<RawCompilerOptions>>;

export interface Context {
cwd: string;
pkg: any;
typingsFile: string;
testFiles: string[];
config: Config;
options: Options
}

export interface Options {
cwd: string;
verify: boolean;
writeSnapshot: boolean;
sizeDelta: number;
}

export enum DiagnosticCode {
Expand All @@ -26,7 +33,8 @@ export enum DiagnosticCode {
PropertyDoesNotExistOnType = 2339,
ArgumentTypeIsNotAssignableToParameterType = 2345,
CannotAssignToReadOnlyProperty = 2540,
ExpectedArgumentsButGotOther = 2554
ExpectedArgumentsButGotOther = 2554,
NoOverloadMatches = 2769
}

export interface Diagnostic {
Expand All @@ -37,6 +45,17 @@ export interface Diagnostic {
column?: number;
}

export interface ProjectSizeStats {
typeCount: number;
memoryUsage: number;
relationCacheSizes: object;
}

export interface RunResults {
diagnostics: Diagnostic[];
stats: ProjectSizeStats;
}

export interface Location {
fileName: string;
start: number;
Expand Down
85 changes: 85 additions & 0 deletions source/lib/memory-validator.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
import * as path from 'path';
import {ProjectSizeStats, Diagnostic, Context} from './interfaces';
import * as pathExists from 'path-exists';
import {writeFileSync, readFileSync} from 'fs';

const statsFilename = '.tsd-memory-check-results.json';

const findStatsFile = async (context: Context) => {
const rootFileExists = await pathExists(path.join(context.options.cwd, statsFilename));
if (rootFileExists) {
return path.join(context.options.cwd, statsFilename);
}

const subfolderFileExists = await pathExists(path.join(context.options.cwd, '.tsd', statsFilename));
if (subfolderFileExists) {
return path.join(context.options.cwd, '.tsd', statsFilename);
}

return;
};

/**
* Get a list of TypeScript diagnostics, and memory size within the current context.
*
* @param context - The context object.
* @param runStats - An object which has the memory stats from a run.
* @returns An array of diagnostics
*/
export const verifyMemorySize = async (context: Context, runStats: ProjectSizeStats): Promise<Diagnostic[]> => {
const existingStatsPath = await findStatsFile(context);

const writeJSON = (message: string) => {
const rootFilePath = path.join(context.options.cwd, statsFilename);
writeFileSync(rootFilePath, JSON.stringify(runStats, null, ' '), 'utf8');
return [{
fileName: rootFilePath,
message,
severity: 'warning' as 'warning'
}];
};

if (!existingStatsPath) {
return writeJSON('Recording the stats for memory size in your types');
}

if (context.options.writeSnapshot) {
return writeJSON('Updated the stats for memory size in your types');
}

const existingResults = JSON.parse(readFileSync(existingStatsPath, 'utf8')) as ProjectSizeStats;
const validate = (prev: number, current: number) => {
const largerPrev = (prev / 100) * (context.options.sizeDelta + 100);
return current < largerPrev;
};

const names: string[] = [];

// This is an approximation, and would likely change between versions, so the number is
// conservative
const typescriptTypes = 8500;

if (!validate(existingResults.typeCount - typescriptTypes, runStats.typeCount - typescriptTypes)) {
names.push(`- Type Count raised by ${runStats.typeCount - existingResults.typeCount}`);
}

if (!validate(existingResults.memoryUsage, runStats.memoryUsage)) {
names.push(`- Memory usage raised by ${runStats.typeCount / existingResults.typeCount * 100}%`);
}

if (names.length) {
const messages = [
'Failed due to memory changes for types being raised. Higher numbers',
'can slow down the editor experience for consumers.',
'If you\'d like to update the fixtures to keep these changes re-run tsd with --write',
'', ...names];

return [{
fileName: existingStatsPath,
message: messages.join('\n '),
severity: 'error' as 'error'
}];
}

return [];
};
2 changes: 1 addition & 1 deletion source/lib/rules/files-property.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ export default (context: Context): Diagnostic[] => {
return [];
}

const content = fs.readFileSync(path.join(context.cwd, 'package.json'), 'utf8');
const content = fs.readFileSync(path.join(context.options.cwd, 'package.json'), 'utf8');

return [
{
Expand Down
2 changes: 1 addition & 1 deletion source/lib/rules/types-property.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ export default (context: Context): Diagnostic[] => {
const {pkg} = context;

if (!pkg.types && pkg.typings) {
const content = fs.readFileSync(path.join(context.cwd, 'package.json'), 'utf8');
const content = fs.readFileSync(path.join(context.options.cwd, 'package.json'), 'utf8');

return [
{
Expand Down
2 changes: 1 addition & 1 deletion source/test/test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ test('support setting a custom test directory', async t => {
test('expectError for functions', async t => {
const diagnostics = await m({cwd: path.join(__dirname, 'fixtures/expect-error/functions')});

t.true(diagnostics.length === 1);
t.true(diagnostics.length === 1, `Diagnostics: ${diagnostics.map(d => d.message)}`);

t.true(diagnostics[0].column === 0);
t.true(diagnostics[0].line === 5);
Expand Down