The code generated for the interpreter(s) is too fragile

[markshannon]

Changing the name of an instruction, or slight changes in cache size can cause the generated code to be incorrect in odd ways.
This is a result of there being too many special cases and undocumented assumptions built in to the code generator.

This was a natural result of getting the thing working and was, until now, acceptable technical debt.
However, as we want to generate abstract interpreters, a new combiner tier-1/tier-2 interpreter and possible tail-calling interprtet/JIT compiler pair, things are too fragile.

We need to:

• Reduce the number of ad-hoc special cases in the tools to zero.
• Minimize and very clearly specify any special markers and variables we use in bytecodes.c
• Clarify the in-memory state and transitions of execution.
• Document the representation of that state in the various tiers and components.

Linked PRs

• GH-111485: Increment next_instr consistently at the start of the instruction. #111486
• GH-111485: Remove some special cases from the code generator and bytecodes.c #111540
• GH-111485: Use micro-ops to split specialization code from base action #111547
• GH-111485: Use micro-ops to split specialization code from base action #111561
• GH-111485: Silence warnings in Python/executor_cases.c.h #111619
• GH-111485: Allow arbitrary annotations on instructions and micro-ops. #111697
• GH-111485: Make BEFORE_WITH a uop #111812
• GH-111485: Separate out parsing, analysis and code-gen phases #112299
• GH-111485: Sort cases #112315
• GH-111485: Factor out opcode ID generator. #112831
• GH-111485: Factor out generation of uop IDs from cases generator. #112877
• GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator #112968
• GH-111485: Sort metadata tables for easier checking of future diffs #113101
• GH-111485: Mark some instructions as TIER_ONE_ONLY #113155
• GH-111485: Break up instructions with unused cache entries into component micro-ops #113169
• GH-111485: Test new cases generator #113252
• GH-111485: Generate instruction and uop metadata #113287
• GH-111485: Generate TARGET table for computed goto dispatch. #113319
• GH-111485: Delete the old generator code. #113321
• GH-111485: Fix DEFAULT_OUTPUT in opcode_metadata_generator.py #113324
• GH-111485: Fix handling of FOR_ITER in tier2 #113394

[neonene]

Maybe INSTRUMENTED_RESUME has an unintentional edit (mixed replacement) since d27acd4?

cpython/Python/bytecodes.c

Lines 187 to 189 in d27acd4

	if (frame->instr_ptr != this_instr) {
	/* Instrumentation has jumped */
	next_instr = this_instr;

Previously, line 189 was next_instr = frame->instr_ptr;

[gvanrossum]

@markshannon ^^

[markshannon]

Indeed. That doesn't look right.