Enable definition for generated interpreter cases to be composed from multiple files

[jbower-fb]

Feature or enhancement

Allow specifying multiple input files to the generate_cases.py script, making it behave mostly as if the input files were concatenated. Additionally allow existing definitions of instructions to be explicitly overridden using a new override keyword.

I attach a PR with a proposed initial implementation.

Pitch

In Cinder we add a number of new instructions to support our features like Static Python. We also currently have a few tweaks to existing instructions. When we migrate to the new upstream generated interpreter it would be preferable if we could avoid having to make changes to the core bytecodes.c and keep our own definitions/changes separate. As well as easing upstream merges, this would also avoid us having to copy/fork more than we need for Cinder features in a standalone module.

I've made an initial implementation which allows extra files to be passed to generate_cases.py by repeated use of the -i argument. E.g.:

$ generate_cases.py -i bytecodes.c -i cinder-bytecodes.c -o generated_cases.c.h

This mostly behaves as if the input files are concatenated but parsing only takes place between the BEGIN/END BYTECODES markers in each file. We also take advantage of mostly existing book-keeping features to track which input files definitions come from when producing errors.

I've also added a new override keyword which can prefix instruction definitions to explicitly express the intent to override an existing definition. E.g.:

inst(NOP, (--)) {
}

// This is the definition which ends up being used in generation.
override inst(NOP, (--)) {
  magic();
}

// Error - previous definition of NOP exists and "override" not specified.
inst(NOP, (--)) {
}

// Error - requested override but no previous definition of ZOP exists.
override inst(ZOP, (--)) {
}

The goal of explicitly calling out overrides is to quickly reveal if either: something we modify is removed from upstream, or if a new opcode we add ends up with a name clash with a new upstream opcode.

Previous discussion

The idea of having multiple input files for interpreter generation was briefly discussed around the faster-cpython project.

Linked PRs

• gh-102021 : Allow multiple input files for interpreter loop generator #102022

[gvanrossum]

Thanks!