Allow sys.settrace() to inspect locals() variables including (optional) name resolution.

[Josverl]

Summary

debugpy running on an ESP32 connected over wifi

This enhances the debugpy/pdb support to enable debuggers and profiling tools to access local variable names and values in function frames, which is essential for interactive debugging, breakpoint inspection, and runtime analysis. The implementation includes both basic local variable access and advanced variable name preservation with correct slot mapping.

Key Features Added

1. Basic Frame Local Variables Access (frame.f_locals)

The initial implementation provides access to local variables in stack frames through the frame.f_locals property. This foundational feature enables debuggers to inspect variable values at runtime.

Key Components:

• frame_f_locals() function - Core implementation in py/profile.c
• Memory-safe access - Includes GC lock checking and state validation
• Generic fallback names - Uses local_01, local_02, etc. when variable names unavailable
• Robust error handling - Graceful handling of invalid state or memory allocation failures

Implementation Features:

• Pre-allocates dictionary size based on frame state count for efficiency
• Validates code state and state array before access
• Skips NULL values in the state array
• Safe qstr creation with error checking

2. Advanced Variable Name Storage (MICROPY_SAVE_LOCAL_VARIABLE_NAMES)

An enhanced compile-time feature that preserves actual local variable names in compiled bytecode for professional debugging capabilities.

Configuration:

#define MICROPY_SAVE_LOCAL_VARIABLE_NAMES (1)  // Save local variable names for debugging

Note: The implementation also supports a conditional flag MICROPY_PY_SYS_SETTRACE_SAVE_NAMES for more granular control when MICROPY_PY_SYS_SETTRACE is enabled.

Components:

• py/localnames.h - Data structures for variable name mapping
• py/localnames.c - Functions to manage variable name storage and retrieval
• py/debug_locals.h - Debug utilities header
• py/debug_locals.c - Debug utilities implementation

3. Enhanced Frame Local Variables Access with Real Names

Enhanced the frame.f_locals property to return actual local variable names and values instead of generic placeholder names.

Evolution:

# Basic implementation returns:
{'local_01': value1, 'local_02': value2, ...}

# Enhanced implementation returns:
{'foo': 'hello debugger', 'bar': 123, 'my_var': [1, 2, 3], ...}

4. Reverse Slot Assignment Fix

Fixed a critical bug in variable-to-slot mapping where variables were incorrectly mapped to runtime slots.

Problem: Variables were being assigned sequentially from low slots up, but the runtime was expecting them from high slots down.

Solution: Implemented reverse slot assignment where:

• First variable in source order → Highest available slot
• Last variable in source order → Lowest available slot (after parameters)

5. Debug Configuration Options

Added optional compiler optimization controls for enhanced debugging experience:

// Disable compiler optimizations for debugging (optional)
#define MICROPY_COMP_CONST                 (0)
#define MICROPY_COMP_MODULE_CONST          (0)
#define MICROPY_COMP_DOUBLE_TUPLE_ASSIGN   (0)
#define MICROPY_COMP_TRIPLE_TUPLE_ASSIGN   (0)

Data Structures

mp_local_names_t

typedef struct _mp_local_names_t {
    uint16_t num_locals;                      // Total number of local variables with names
    qstr local_names[MICROPY_PY_SYS_SETTRACE_NAMES_MAX];     // Array of variable names, indexed by local_num
    uint16_t local_nums[MICROPY_PY_SYS_SETTRACE_NAMES_MAX];  // Reverse mapping: name index → local_num
    uint16_t order_count;                     // Number of variables stored in order they were defined
    uint16_t runtime_slots[MICROPY_PY_SYS_SETTRACE_NAMES_MAX]; // Mapping of local_num to runtime slots
} mp_local_names_t;

Note: MICROPY_PY_SYS_SETTRACE_NAMES_MAX defaults to 32 and can be configured at compile time.

Enhanced mp_raw_code_t

typedef struct _mp_raw_code_t {
    // ... existing fields ...
    #if MICROPY_PY_SYS_SETTRACE_SAVE_NAMES
    mp_local_names_t local_names;  // Maps local variable indices to names
    #endif
} mp_raw_code_t;

API Functions

Core Functions

• mp_local_names_init() - Initialize local names structure
• mp_local_names_add() - Add variable name mapping
• mp_local_names_get_name() - Get variable name by index
• mp_local_names_get_local_num() - Get local number by order index
• mp_local_names_get_runtime_slot() - Get runtime slot mapping

Debug Functions

• mp_debug_print_local_variables() - Print variable mappings for debugging
• mp_debug_locals_info() - Exposed as sys.debug_locals_info() for runtime debugging

Compilation Integration

Enhanced Compiler (py/compile.c)

#if MICROPY_PY_SYS_SETTRACE_SAVE_NAMES
// Save the local variable names in the raw_code for debugging
if (SCOPE_IS_FUNC_LIKE(scope->kind) && scope->num_locals > 0) {
    // Populate with variable names - examining the assignment order
    for (int i = 0; i < scope->id_info_len; i++) {
        id_info_t *id = &scope->id_info[i];
        if ((id->kind == ID_INFO_KIND_LOCAL || id->kind == ID_INFO_KIND_CELL) && 
            id->local_num < scope->num_locals && 
            id->local_num < MICROPY_PY_SYS_SETTRACE_NAMES_MAX) {
            
            mp_local_names_add(&scope->raw_code->local_names, id->local_num, id->qst);
        }
    }
}
#endif

Frame Locals Implementation (py/profile.c)

The implementation of frame_f_locals() has evolved through two phases:

Phase 1: Basic Implementation

The initial version provides fundamental local variable access:

static mp_obj_t frame_f_locals(mp_obj_t self_in) {
    // Memory safety: Cannot create locals dict when GC is locked
    if (gc_is_locked()) {
        return MP_OBJ_NULL;
    }
    
    mp_obj_frame_t *frame = MP_OBJ_TO_PTR(self_in);
    mp_obj_dict_t *locals_dict = mp_obj_new_dict(frame->code_state->n_state);
    const mp_code_state_t *code_state = frame->code_state;

    // Validate state array before access
    if (code_state == NULL || code_state->state == NULL) {
        return MP_OBJ_FROM_PTR(locals_dict);
    }

    // Generate generic variable names for all non-NULL values
    for (size_t i = 0; i < code_state->n_state; ++i) {
        if (code_state->state[i] == NULL) {
            continue; // Skip invalid values
        }
        
        char var_name[16];
        snprintf(var_name, sizeof(var_name), "local_%02d", (int)(i + 1));
        qstr var_name_qstr = qstr_from_str(var_name);
        
        if (var_name_qstr == MP_QSTR_NULL) {
            continue; // Skip if qstr creation fails
        }
        
        mp_obj_dict_store(locals_dict, MP_OBJ_NEW_QSTR(var_name_qstr), code_state->state[i]);
    }
    
    return MP_OBJ_FROM_PTR(locals_dict);
}

Phase 2: Enhanced Implementation

The enhanced version adds comprehensive variable name support:

1. Handles Parameters - Maps function arguments to correct slots (0 to n_args-1)
2. Implements Reverse Slot Assignment - Maps local variables using reverse order
3. Provides Fallback - Uses generic names when variable names unavailable
4. Validates State - Ensures safe access to frame state arrays

// REVERSE SLOT ASSIGNMENT: Variables assigned from highest available slot down
uint16_t total_locals = code_state->n_state;
uint16_t reverse_slot = total_locals - 1 - order_idx;

Key Safety Features

• GC Lock Protection: Prevents dictionary creation during garbage collection
• State Validation: Checks for NULL code_state and state arrays
• Memory Allocation Checking: Validates qstr creation before use
• NULL Value Skipping: Ignores uninitialized or cleared variables

Configuration Changes

Unix Standard Port (ports/unix/variants/standard/mpconfigvariant.h)

The configuration enables the debugging features. The current implementation uses different configuration options:

// Enable sys.settrace support (required)
#define MICROPY_PY_SYS_SETTRACE (1)

// Enable local variable name preservation (currently commented out by default)
// #define MICROPY_PY_SYS_SETTRACE_SAVE_NAMES (1)

// Alternative configuration for broader variable name support
#define MICROPY_SAVE_LOCAL_VARIABLE_NAMES (1)

Configuration Levels:

1. Basic Debugging (minimum):

   #define MICROPY_PY_SYS_SETTRACE (1)

   • Enables sys.settrace() with generic variable names
   • frame.f_locals returns {'local_01': value1, 'local_02': value2, ...}

2. Enhanced Debugging (settrace-specific):

   #define MICROPY_PY_SYS_SETTRACE (1)
   #define MICROPY_PY_SYS_SETTRACE_SAVE_NAMES (1)

   • Enables sys.settrace() with actual variable names using settrace-specific storage
   • Conditional compilation only when settrace is enabled

3. Advanced Debugging (broader support):

   #define MICROPY_PY_SYS_SETTRACE (1)
   #define MICROPY_SAVE_LOCAL_VARIABLE_NAMES (1)

   • Enables variable name preservation across broader MicroPython functionality
   • frame.f_locals returns {'foo': value1, 'bar': value2, ...}

4. Debug-Optimized Build (for intensive debugging):

   #define MICROPY_PY_SYS_SETTRACE (1)
   #define MICROPY_SAVE_LOCAL_VARIABLE_NAMES (1)
   #define MICROPY_COMP_CONST (0)
   #define MICROPY_COMP_MODULE_CONST (0)
   #define MICROPY_COMP_DOUBLE_TUPLE_ASSIGN (0)
   #define MICROPY_COMP_TRIPLE_TUPLE_ASSIGN (0)

   • All debugging features enabled
   • Compiler optimizations disabled for more predictable variable behavior

Global Configuration (py/mpconfig.h)

// If not explicitly enabled, disable settrace variable name saving
#ifndef MICROPY_PY_SYS_SETTRACE
#define MICROPY_PY_SYS_SETTRACE_SAVE_NAMES (0)
#endif
#ifndef MICROPY_PY_SYS_SETTRACE_SAVE_NAMES
#define MICROPY_PY_SYS_SETTRACE_SAVE_NAMES (0)
#endif

Note: The broader MICROPY_SAVE_LOCAL_VARIABLE_NAMES configuration is not currently implemented in the global config, but is used at the implementation level.

Build System Integration

Makefile Changes (py/py.mk)

PY_CORE_O_BASENAME = $(addprefix py/,\
    # ... existing files ...
    localnames.o \
    debug_locals.o \
    # ... rest of files ...

Note: The build system automatically includes the new object files for local variable name support and debug utilities.

Usage Examples

Basic sys.settrace Usage

import sys
from collections import OrderedDict

def tracer(frame, event, arg):
    if event == 'line':
        print(f"Line {frame.f_lineno}: {frame.f_locals}")
    return tracer

def test_function():
    foo = "hello debugger"
    bar = 123
    my_list = [1, 2, 3]
    print("Debug me!")

sys.settrace(tracer)
test_function()
sys.settrace(None)

Expected Output

Line 11: {'foo': 'hello debugger'}
Line 12: {'foo': 'hello debugger', 'bar': 123}
Line 13: {'foo': 'hello debugger', 'bar': 123, 'my_list': [1, 2, 3]}
Debug me!
Line 14: {'foo': 'hello debugger', 'bar': 123, 'my_list': [1, 2, 3]}

Debug Output

When MICROPY_PY_SYS_SETTRACE_SAVE_NAMES is enabled, detailed debug information is printed:

DEBUG: Processing frame with 16 state slots
DEBUG: Parameters: 0 positional + 0 keyword-only = 0 total
DEBUG: Variable 'foo' (order 0) -> REVERSE slot 15
SUCCESS: 'foo' mapped to state[15] = 'hello debugger'
DEBUG: Variable 'bar' (order 1) -> REVERSE slot 14  
SUCCESS: 'bar' mapped to state[14] = 123

Memory Considerations

• Slot Limit: Maximum of MICROPY_PY_SYS_SETTRACE_NAMES_MAX (32) local variables per function
• Memory Overhead: Approximately 4 bytes per local variable name mapping
• Compile-time: Only enabled when MICROPY_PY_SYS_SETTRACE_SAVE_NAMES is defined

Compatibility

Backward Compatibility

• When MICROPY_PY_SYS_SETTRACE_SAVE_NAMES is disabled (default), basic functionality still works
• When only MICROPY_PY_SYS_SETTRACE is enabled, frame.f_locals uses generic names (local_01, local_02, etc.)
• Existing code continues to work without modification
• Progressive enhancement: more features become available as more flags are enabled

Platform Support

• Currently tested on
  • Unix standard
  • ESP32_GENERIC
• Should work on any platform with sys.settrace() support
• Requires MICROPY_PERSISTENT_CODE_SAVE to be enabled

Debugging Commands

Runtime Debug Information

import sys
sys.debug_locals_info()  # Print detailed variable mapping information

This command prints comprehensive information about:

• Current code state and frame details
• Variable name to slot mappings
• Complete state array contents
• Parameter vs. local variable assignments

Testing

Trade-offs and Alternatives

Currently the max number of local names preservers is 32 , to limit the memory impact
if you inspect a scope/closure with more local vars - the names are mapped incorrectly
Not quite sure yet about the memory impact if it is possible to increase this number, or there is a need to fall-back to the local_<nn> format. (which also takes memory to allocate the names)

[andrewleech]

I'm not entirely versed in the internals enough to be sure; but the existing locals implementation is just a normal C array of obj's isn't it? (pointer and len). The same array holds a couple of other pieces of stack data at the start before the locals? (old memory of how it works).

The current implementation of storing locals feels a bit heavy; Can it be simplified along the lines of "in the compiler where it currently determines how many local variables are in scope to create / set the size of the locals array, also define the array of QSTR references of matching length to the number of local variables. Then it can be populated with the QSTR references at the same time. Also, elsewhere in the compiler names of objects like classes, functions, global variables are all captured as QSTR references; ensure this section is updated (if the #define is enabled) to also capture local variable names as QSTRS's"

[Josverl]

Thanks for the suggestion,
There must indeed be existing logic that may be possible to re-use.

I'm also wondering if the current 2 different ways to store/generate local names actually compare wrt to memory usage.

I'm not surprised to need a few more iterations. But this seems like another good stepping-stone.

[andrewleech]

I'm not surprised to need a few more iterations. But this seems like another good stepping-stone.

Oh definitely yes, it's an amazing stepping stone that effectively demonstrates just what is possible!

This is what I'm loving about the current crop of AI tools. Even if the first pass implementations aren't great they let you try things so quickly, the code written is "cheap" in that is easy to throw it away and try again!

My favourite (by a long margin) is Claude Code; I pointed it at this branch to review your working implantation, then wrote a summary of everything I thought I knew about this area and set it loose.

That's the new linked pr just above here, another attempt at local variables (two different options in the one PR), as well as unit tests for both your original commit and the new stuff. Those tests are the only testing I've done so far, the branch was "written" from my phone, I haven't had time at a computer to test it in vscode yet.