From 96e1e786244925766da34f20a781837f05cfedd7 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 16 Jul 2024 10:12:34 -0600 Subject: [PATCH 01/19] Add the multiple interpreters howto doc. --- Doc/howto/index.rst | 2 + Doc/howto/multiple-interpreters.rst | 796 ++++++++++++++++++++++++++ Doc/library/interpreters.channels.rst | 150 +++++ Doc/library/interpreters.queues.rst | 148 +++++ Doc/library/interpreters.rst | 235 ++++++++ 5 files changed, 1331 insertions(+) create mode 100644 Doc/howto/multiple-interpreters.rst create mode 100644 Doc/library/interpreters.channels.rst create mode 100644 Doc/library/interpreters.queues.rst create mode 100644 Doc/library/interpreters.rst diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index f350141004c2db..abc2e48b44ed2f 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -32,6 +32,7 @@ Python Library Reference. isolating-extensions.rst timerfd.rst mro.rst + multiple-interpreters.rst free-threading-python.rst free-threading-extensions.rst remote_debugging.rst @@ -57,6 +58,7 @@ Advanced development: * :ref:`freethreading-python-howto` * :ref:`freethreading-extensions-howto` * :ref:`isolating-extensions-howto` +* :ref:`multiple-interpreters-howto` * :ref:`python_2.3_mro` * :ref:`socket-howto` * :ref:`timerfd-howto` diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst new file mode 100644 index 00000000000000..4f319742af0a79 --- /dev/null +++ b/Doc/howto/multiple-interpreters.rst @@ -0,0 +1,796 @@ +.. _multiple-interpreters-howto: + +*************************** +Multiple Interpreters HOWTO +*************************** + +When it comes to concurrency and parallelism, +Python provides a number of options, each with various tradeoffs. +This includes threads, async, and multiprocessing. +There's one more option: multiple interpreters (AKA "subinterpreters"). + +.. (PyPI packages expand that with "greenlets", and distributed computing.) + +Multiple interpreters in the same process provide conceptual isolation +like processes but more efficiently, like threads. In fact, you can +think of them like threads with opt-in sharing. + +That provides a concurrency model which is easier to reason about, +similar to CSP or the actor model. +Furthermore, each interpreter has its own GIL, so they provide +true multi-core parallelism. + +This HOWTO document has 2 parts: first a tutorial and then a set +of recipes and practical examples. + +The tuturial covers the basics of using multiple interpreters +(:ref:`interp-tutorial-basics`), as well as how to communicate between +interpreters (:ref:`interp-tutorial-communicating`). +The examples section (:ref:`interp-recipes`) is focused on providing +effective solutions for real concurrency use cases. This includes +comparisons with Python's various existing concurrency models. + +.. currentmodule:: interpreters + +.. seealso:: + + :ref:`execcomponents` + :mod:`interpreters` + :mod:`interpreters.queues` + :mod:`interpreters.channels` + +.. note:: + + This page is focused on *using* multiple interpreters. + For information about changing an extension module to work + with multiple interpreters, see :ref:`isolating-extensions-howto`. + + + + + + +Why Use Multiple Interpreters? +------------------------------ + +This provides a similar + +We can take advantage of the isolation and independence of multiple +processes, by using :mod:`multiprocessing` or :mod:`subprocess` +(with sockets and pipes) +In the same process, We have threads, + + +For concurrent code, imagine combining the efficiency of threads +with the isolation +You can run multiple isolated Python execution environments +in the same process + + + + +Python can actually run multiple interpreters at the same time, +which lets you have independent, isolated execution environments +in the same process. This has been true since Python 2.2, but +the feature was only available through the C-API and not well known; +and the isolation was incomplete. However... + +As of Python 3.12, isolation has been fixed, +to the point that interpreters in the same process no longer +even share the GIL (global interpreter lock). That means you can + +has had the ability to run multiple copies + +Benefits: + +Downsides: + +... + + +When to Use Multiple Interpreters +--------------------------------- + +... + ++-------------------------------------+--------------------------------------+ +| Task you want to perform | The best tool for the task | ++=====================================+======================================+ +| ... | ... | ++-------------------------------------+--------------------------------------+ +| ... | ... | ++-------------------------------------+--------------------------------------+ + + +.. _interp-tutorial-basics: + +Tutorial: Basics +================ + +First of all, keep in mind that using multiple interpreters is like +using multiple processes. They are isolated and independent from each +other. The main difference is that multiple interpreters live in the +same process, which makes it all more efficient. + +Each interpreter has its own ``__main__`` module, its own +:func:`sys.modules`, and, in fact, its own set of *all* runtime state. +Interpreters pretty much never share objects and very little data. + +Running Code in an Interpreter +------------------------------ + +Running code in an interpreter is basically equivalent to running the +buitlin :func:`exec` using that interpreter:: + + import interpreters + + interp = interpreters.create() + + interp.exec('print("spam!")') + interp.exec(""" + print('spam!') + """) + +Calling a function in an interpreter works the same way:: + + import interpreters + + interp = interpreters.create() + + def script(): + print('spam!') + interp.call(script) + +When it runs, the code is executed using the interpreter's ``__main__`` +module, just like a Python process normally does:: + + import interpreters + + print(__name__) + # __main__ + + interp = interpreters.create() + interp.exec(""" + print(__name__) + """) + # __main__ + +In fact, a comparison with ``python -c`` is quite direct:: + + import interpreters + + ############################ + # python -c 'print("spam!")' + ############################ + + interp = interpreters.create() + interp.exec('print("spam!")') + +It's also fairly easy to simulate the other forms of the Python CLI:: + + import interpreters + + with open('script.py', 'w') as outfile: + outfile.write(""" + print('spam!') + """) + + ################## + # python script.py + ################## + + interp = interpreters.create() + with open('script.py') as infile: + script = infile.read() + interp.exec(script) + + ################## + # python -m script + ################## + + interp = interpreters.create() + interp.exec(""" + import runpy + runpy.run_module('script') + """) + +That's more or less what the ``python`` executable is doing for each +of those cases. + +Preparing and Reusing an Interpreter +------------------------------------ + +When you use the Python REPL, it doesn't reset after each command. +That would make the REPL much less useful. Likewise, when you use +the builtin :func:`exec`, it doesn't reset the namespace it uses +to run the code. + +In the same way, running code in an interpreter does not reset it. +The next time you run code in that interpreter, the ``__main__`` module +will be in exactly the state in which you left it:: + + import interpreters + + interp = interpreters.create() + interp.exec(""" + answer = 42 + """) + interp.exec(""" + assert answer == 42 + """) + +You can take advantage of this to prepare an interpreter ahead of time +or use it incrementally:: + + import interpreters + + interp = interpreters.create() + + # Prepare the interpreter. + interp.exec(""" + # We will need this later. + import math + + # Initialize the value. + value = 1 + + def double(val): + return val + val + """) + + # Do the work. + for _ in range(10): + interp.exec(""" + assert math.factorial(value + 1) >= double(value) + value = double(value) + """) + + # Show the result. + interp.exec(""" + print(value) + """) + +In case you're curious, in a little while we'll look at how to pass +data in and out of an interpreter (instead of just printing things). + +Sometimes you might also have values that you want to use in another +interpreter. We'll look more closely at that case in a little while: +:ref:`interp-script-args`. + +Running in a Thread +------------------- + +A key point is that, in a new Python process, the runtime doesn't +create a new thread just to run the target script. It uses the main +thread the process started with. + +In the same way, running code in another interpreter doesn't create +(or need) a new thread. It uses the current OS thread, switching +to the interpreter long enough to run the code. + +To actually run in a new thread, you do it explicitly:: + + import interpreters + import threading + + interp = interpreters.create() + + def run(): + interp.exec("print('spam!')") + t = threading.Thread(target=run) + t.start() + t.join() + + def run(): + interp.call(script) + t = threading.Thread(target=run) + t.start() + t.join() + +There's also a helper method for that:: + + import interpreters + + interp = interpreters.create() + + def script(): + print('spam!') + t = interp.call_in_thread(script) + t.join() + +Handling Uncaught Exceptions +---------------------------- + +Consider what happens when you run a script at the commandline +and there's an unhandled exception. In that case, Python will print +the traceback and the process will exit with a failure code. + +The behavior is very similar when code is run in an interpreter. +The traceback get printed and, rather than a failure code, +an :class:`ExecutionFailed` exception is raised:: + + import interpreters + + interp = interpreters.create() + + try: + interp.exec('1/0') + except interpreters.ExecutionFailed: + print('oops!') + +The exception also has some information, which you can use to handle +it with more specificity:: + + import interpreters + + interp = interpreters.create() + + try: + interp.exec('1/0') + except interpreters.ExecutionFailed as exc: + if exc.excinfo.type.__name__ == 'ZeroDivisionError': + ... + else: + raise # re-raise + +You can handle the exception in the interpreter as well, like you might +do with threads or a script:: + + import interpreters + + interp = interpreters.create() + + interp.exec(""" + try: + 1/0 + except ZeroDivisionError: + ... + """) + +At the moment there isn't an easy way to "re-raise" an unhandled +exception from another interpreter. Here's one approach that works for +exceptions that can be pickled:: + + import interpreters + + interp = interpreters.create() + + try: + interp.exec(""" + try: + 1/0 + except Exception as exc: + import pickle + data = pickle.dumps(exc) + class PickledException(Exception): + pass + raise PickledException(data) + """) + except interpreters.ExecutionFailed as exc: + if exc.excinfo.type.__name__ == 'PickledException': + import pickle + raise pickle.loads(exc.excinfo.msg) + else: + raise # re-raise + +Managing Interpreter Lifetime +----------------------------- + +Every interpreter uses resources, particularly memory, that should be +cleaned up as soon as you're done with the interpreter. While you can +wait until the interpreter is cleaned up when the process exits, you +can explicitly clean it up sooner:: + + import interpreters + + interp = interpreters.create() + try: + interp.exec(""" + print('spam!') + """) + finally: + interp.close() + +concurrent.futures.InterpreterPoolExecutor +------------------------------------------ + +The :mod:`concurrent.futures` module is a simple, popular concurrency +framework that wraps both threading and multiprocessing. You can also +use it with multiple interpreters:: + + from concurrent.futures import InterpreterPoolExecutor + + with InterpreterPoolExecutor(max_workers=5) as executor: + executor.submit('print("spam!")') + +Gotchas +------- + +... + +Keep in mind that there is a limit to the kinds of functions that may +be called by an interpreter. Specifically, ... + + +* limited callables +* limited shareable objects +* not all PyPI packages support use in multiple interpreters yet +* ... + + +.. _interp-tutorial-communicating: + +Tutorial: Communicating Between Interpreters +============================================ + +Multiple interpreters are useful in the convenience and efficiency +they provide for isolated computing. However, their main strength is +in running concurrent code. Much like with any concurrency tool, +there has to be a simple and efficient way to communicate +between interpreters. + +In this half of the tutorial, we explore various ways you can do so. + +.. _interp-script-args: + +Passing Values to an Interpreter +-------------------------------- + +When you call a function in Python, sometimes that function requires +arguments and sometimes it doesn't. In the same way, sometimes a +script you want to run in another interpreter requires some values. +Providing such values to the interpreter, for the script to use, +is the simplest kind of communication between interpreters. + +Perhaps the simplest approach: you can use an f-string or format string +for the script and interpolate the required values:: + + import interpreters + + interp = interpreters.create() + + def run_interp(interp, name, value): + interp.exec(f""" + {name} = {value!r} + ... + """) + run_interp(interp, 'spam', 42) + + try: + infile = open('spam.txt') + except FileNotFoundError: + pass + else: + with infile: + interp.exec(f""" + import os + for line in os.fdopen({infile.fileno())): + print(line) + """)) + +This works especially well for intrinsic values. For complex values +you'll usually want to reach for other solutions. + +There's one convenient alternative that works for some objects: +:func:`Interpreter.prepare_main`. It binds values to names in the +interpreter's ``__main__`` module, which makes them available to any +scripts that run in the interpreter after that:: + + import interpreters + + interp = interpreters.create() + + def run_interp(interp, name, value): + interp.prepare_main(**{name: value}) + interp.exec(f""" + ... + """) + run_interp(interp, 'spam', 42) + + try: + infile = open('spam.txt') + except FileNotFoundError: + pass + else: + with infile: + interp.prepare_main(fd=infile.fileno()) + interp.exec(f""" + import os + for line in os.fdopen(fd): + print(line) + """)) + +Note that :func:`Interpreter.prepare_main` only works with "shareable" +objects. (See :ref:`interp-shareable-objects`.) In a little while +also see how shareable objects make queues and channels especially +convenient and efficient. + +For anything more complex that isn't shareable, you'll probably need +something more than f-strings or :func:`Interpreter.prepare_main`. + +That brings us the next part: 2-way communication, passing data back +and forth between interpreters. + +Pipes, Sockets, etc. +-------------------- + +We'll start off with an approach to duplex communication between +interpreters that you can implement using OS-provided utilities. +The examples will use :func:`os.pipe`, but the approach applies equally +well to regular files and sockets. + +Keep in mind that pipes, sockets, and files work with :class:`bytes`, +not other objects, so this may involve at least some serialization. +Aside from that inefficiency, sending data through the OS will +also slow things down. + +After this we'll look at using queues and channels for simpler and more +efficient 2-way communication. The contrast should be clear then. + +First, let's use a pipe to pass a message from one interpreter +to another:: + + import interpreters + import os + + interp1 = interpreters.create() + interp2 = interpreters.create() + + rpipe, spipe = os.pipe() + try: + def task(): + interp.exec(""" + import os + msg = os.read({rpipe}, 20) + print(msg) + """) + t = threading.thread(target=task) + t.start() + + # Sending the message: + os.write(spipe, b'spam!') + t.join() + finally: + os.close(rpipe) + os.close(spipe) + +One interesting part of that is how the subthread blocked until +we wrote to the pipe. In addition to delivering the message, the +read end of the pipe acted like a lock. + +We can actually make use of that to synchronize +execution between the interpreters:: + + import interpreters + + interp = interpreters.create() + + r1, s1 = os.pipe() + r2, s2 = os.pipe() + try: + def task(): + interp.exec(""" + # Do stuff... + + # Synchronize! + msg = os.read(r1, 1) + os.write(s2, msg) + + # Do other stuff... + """) + t = threading.thread(target=task) + t.start() + + # Synchronize! + os.write(s1, '') + os.read(r2, 1) + finally: + os.close(r1) + os.close(s1) + os.close(r2) + os.close(s2) + +You can also close the pipe ends and join the thread to synchronize. + +Again, using :func:`os.pipe`, etc. to communicate between interpreters +is a little awkward, as well as inefficient. We'll look at some of +the alternatives next. + +Using Queues +------------ + +:class:`interpreters.queues.Queue` is an implementation of the +:class:`queue.Queue` interface that supports safely and efficiently +passing data between interpreters. + +:: + + import interpreters.queues + + queue = interpreters.queues.create() + interp = interpreters.create() + interp.prepare_main(queue=queue) + + queue.put('spam!') + interp.exec(""" + msg = queue.get() + print(msg) + """) + +:: + + queue = interpreters.queues.create() + interp = interpreters.create() + interp.prepare_main(queue=queue) + + queue.put('spam!') + interp.exec(""" + obj = queue.get() + print(msg) + """) + +:: + + queue1 = interpreters.queues.create() + queue2 = interpreters.queues.create(syncobj=True) + interp = interpreters.create() + interp.prepare_main(queue1=queue1, queue2=queue2) + + queue1.put('spam!') + queue1.put('spam!', syncobj=True) + queue2.put('spam!') + queue2.put('spam!', syncobj=False) + interp.exec(""" + msg1 = queue1.get() + msg2 = queue1.get() + msg3 = queue2.get() + msg4 = queue2.get() + print(msg) + """) + + +Using Channels +-------------- + +... + +:: + + import interpreters.channels + + rch, sch = interpreters.channels.create() + interp = interpreters.create() + interp.prepare_main(rch=rch) + + sch.send_nowait('spam!') + interp.exec(""" + msg = rch.recv() + print(msg) + """) + +:: + + rch, sch = interpreters.channels.create() + interp = interpreters.create() + interp.prepare_main(rch=rch) + + data = bytearray(100) + + sch.send_buffer_nowait(data) + interp.exec(""" + data = rch.recv() + for i in range(len(data)): + data[i] = i + """) + assert len(data) == 100 + for i in range(len(data)): + assert data[i] == i + +Capturing an interpreter's stdout +--------------------------------- + +:: + + interp = interpreters.create() + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + interp.exec(tw.dedent(""" + print('spam!') + """)) + assert(stdout.getvalue() == 'spam!') + + # alternately: + interp.exec(tw.dedent(""" + import contextlib, io + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + print('spam!') + captured = stdout.getvalue() + """)) + captured = interp.get_main_attr('captured') + assert(captured == 'spam!') + +:func:`os.pipe()` could be used similarly. + +Capturing Unhandled Exceptions +---------------------------- + +... + +Falling Back to Pickle +---------------------- + +For other objects you can use pickle:: + + import interpreters + import pickle + + SCRIPT = """ + import pickle + data = pickle.loads(_pickled) + setattr(data['name'], data['value']) + """ + data = dict(name='spam', value=42) + + interp = interpreters.create() + interp.prepare_main(_pickled=pickle.dumps(data)) + interp.exec(SCRIPT) + +Passing objects via pickle:: + + interp = interpreters.create() + r, s = os.pipe() + interp.exec(tw.dedent(f""" + import os + import pickle + reader = {r} + """)) + interp.exec(tw.dedent(""" + data = b'' + c = os.read(reader, 1) + while c != b'\x00': + while c != b'\x00': + data += c + c = os.read(reader, 1) + obj = pickle.loads(data) + do_something(obj) + c = os.read(reader, 1) + """)) + for obj in input: + data = pickle.dumps(obj) + os.write(s, data) + os.write(s, b'\x00') + os.write(s, b'\x00') + +Gotchas +------- + +... + + +.. _interp-recipes: + +Recipes (Practical Examples) +============================ + +Concurrency-Related Python Workloads +------------------------------------ + +... + +Comparisons With Other Concurrency Models +----------------------------------------- + +... + +Concurrency +----------- + +(incl. minimal comparisons with multiprocessing, threads, and async) + +... + +Isolation +--------- + +... diff --git a/Doc/library/interpreters.channels.rst b/Doc/library/interpreters.channels.rst new file mode 100644 index 00000000000000..0b9b1e76bf1bfd --- /dev/null +++ b/Doc/library/interpreters.channels.rst @@ -0,0 +1,150 @@ +:mod:`!interpreters.channels` -- A cross-interpreter "channel" implementation +================================================================== + +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + +.. module:: interpreters.channels + :synopsis: Multiple Interpreters in the Same Process + +.. moduleauthor:: Eric Snow +.. sectionauthor:: Eric Snow + +.. versionadded:: 3.14 + +**Source code:** :source:`Lib/interpreters/channels.py` + +-------------- + + +Introduction +------------ + +This module constructs higher-level interfaces on top of the lower +level ``_interpchannels`` module. + +.. seealso:: + + :ref:`execcomponents` + ... + + :mod:`interpreters` + provides details about communicating between interpreters. + + :ref:`multiple-interpreters-howto` + demonstrates how to use channels + +.. include:: ../includes/wasm-notavail.rst + + +API Summary +----------- + ++-------------------------------------------------------------------+----------------------------------------------+ +| signature | description | ++===================================================================+==============================================+ +| ``list_all() -> [(RecvChannel, SendChannel)]`` | Get all existing cross-interpreter channels. | ++-------------------------------------------------------------------+----------------------------------------------+ +| ``create(*, unbounditems=UNBOUND) -> (RecvChannel, SendChannel)`` | Initialize a new cross-interpreter channel. | ++-------------------------------------------------------------------+----------------------------------------------+ + +| + ++-------------------------------------------+---------------------------------------------------+ +| signature | description | ++===========================================+===================================================+ +| ``class RecvChannel`` | The receiving end of a cross-interpreter channel. | ++-------------------------------------------+---------------------------------------------------+ +| ``.id`` | The channel's ID (read-only). | ++-------------------------------------------+---------------------------------------------------+ +| ``.is_closed`` | If the channel has been closed (read-only). | ++-------------------------------------------+---------------------------------------------------+ +| ``.recv(timeout=None)`` | Pop off the next item in channel. | ++-------------------------------------------+---------------------------------------------------+ +| ``.recv_nowait(default=None)`` | Pop off the next item in channel, if any. | ++-------------------------------------------+---------------------------------------------------+ +| ``.close()`` | Stop receiving more items. | ++-------------------------------------------+---------------------------------------------------+ + +| + ++------------------------------------------------------------+--------------------------------------------------+ +| signature | description | ++============================================================+==================================================+ +| ``class SendChannel`` | The sending end of a cross-interpreter channel. | ++------------------------------------------------------------+--------------------------------------------------+ +| ``.id`` | The channel's ID (read-only). | ++------------------------------------------------------------+--------------------------------------------------+ +| ``.is_closed`` | If the channel has been closed (read-only). | ++------------------------------------------------------------+--------------------------------------------------+ +| ``.send(obj, timeout=None, *, unbound=None)`` | Add an item to the back of the channel's queue. | ++------------------------------------------------------------+--------------------------------------------------+ +| ``.send_nowait(obj, *, unbound=None)`` | Add an item to the back of the channel's queue. | ++------------------------------------------------------------+--------------------------------------------------+ +| ``.send_buffer(obj, timeout=None, *, unbound=None)`` | Add an item to the back of the channel's queue. | ++------------------------------------------------------------+--------------------------------------------------+ +| ``.send_buffer_nowait(obj, *, unbound=None)`` | Add an item to the back of the channel's queue. | ++------------------------------------------------------------+--------------------------------------------------+ +| ``.close()`` | Stop sending more items. | ++------------------------------------------------------------+--------------------------------------------------+ + +Exceptions: + ++--------------------------+--------------+---------------------------------------------------+ +| class | base class | description | ++==========================+==============+===================================================+ +| ChannelError | Exception | A channel-relaed error happened. | ++--------------------------+--------------+---------------------------------------------------+ +| ChannelNotFoundError | ChannelError | The targeted channel no longer exists. | ++--------------------------+--------------+---------------------------------------------------+ +| ChannelClosedError | ChannelError | The targeted channel has been closed. | ++--------------------------+--------------+---------------------------------------------------+ +| ChannelEmptyError | ChannelError | The targeted channel is empty. | ++--------------------------+--------------+---------------------------------------------------+ +| ChannelNotEmptyError | ChannelError | The targeted channel should have been empty. | ++--------------------------+--------------+---------------------------------------------------+ +| ItemInterpreterDestroyed | ChannelError | The interpreter that added the next item is gone. | ++--------------------------+--------------+---------------------------------------------------+ + + +Basic Usage +----------- + +:: + + import interpreters.channels + + rch, sch = interpreters.channels.create() + interp = interpreters.create() + interp.prepare_main(rch=rch) + + sch.send_nowait('spam!') + interp.exec(""" + msg = rch.recv() + print(msg) + """) + +:: + + rch, sch = interpreters.channels.create() + interp = interpreters.create() + interp.prepare_main(rch=rch) + + data = bytearray(100) + + sch.send_buffer_nowait(data) + interp.exec(""" + data = rch.recv() + for i in range(len(data)): + data[i] = i + """) + assert len(data) == 100 + for i in range(len(data)): + assert data[i] == i + + +Functions +--------- + +This module defines the following functions: + + diff --git a/Doc/library/interpreters.queues.rst b/Doc/library/interpreters.queues.rst new file mode 100644 index 00000000000000..405ed924948325 --- /dev/null +++ b/Doc/library/interpreters.queues.rst @@ -0,0 +1,148 @@ +:mod:`!interpreters.queues` -- A cross-interpreter queue implementation +================================================================== + +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + +.. module:: interpreters.queues + :synopsis: Multiple Interpreters in the Same Process + +.. moduleauthor:: Eric Snow +.. sectionauthor:: Eric Snow + +.. versionadded:: 3.14 + +**Source code:** :source:`Lib/interpreters/queues.py` + +-------------- + + +Introduction +------------ + +This module constructs higher-level interfaces on top of the lower +level ``_interpqueues`` module. + +.. seealso:: + + :ref:`execcomponents` + ... + + :mod:`interpreters` + provides details about communicating between interpreters. + + :ref:`multiple-interpreters-howto` + demonstrates how to use queues + +.. include:: ../includes/wasm-notavail.rst + + +API Summary +----------- + ++-------------------------------------------------------------+--------------------------------------------+ +| signature | description | ++=============================================================+============================================+ +| ``list_all() -> [Queue]`` | Get all existing cross-interpreter queues. | ++-------------------------------------------------------------+--------------------------------------------+ +| ``create(*, syncobj=False, unbounditems=UNBOUND) -> Queue`` | Initialize a new cross-interpreter queue. | ++-------------------------------------------------------------+--------------------------------------------+ + +| + ++------------------------------------------------------------+---------------------------------------------------+ +| signature | description | ++============================================================+===================================================+ +| ``class Queue`` | A single cross-interpreter queue. | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.id`` | The queue's ID (read-only). | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.maxsize`` | The queue's capacity, if applicable (read-only). | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.qsize() -> bool`` | The queue's current item count. | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.empty() -> bool`` | Does the queue currently have no items it it? | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.full() -> bool`` | Has the queue reached its maxsize, if any? | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.put(obj, timeout=None, *, syncobj=None, unbound=None)`` | Add an item to the back of the queue. | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.put_nowait(obj, *, syncobj=None, unbound=None)`` | Add an item to the back of the queue. | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.get(timeout=None) -> object`` | Pop off the next item in queue. | ++------------------------------------------------------------+---------------------------------------------------+ +| ``.get_nowait() -> object`` | Pop off the next item in queue. | ++------------------------------------------------------------+---------------------------------------------------+ + +Exceptions: + ++--------------------------+---------------+---------------------------------------------------+ +| class | base class | description | ++==========================+==============+====================================================+ +| QueueError | Exception | A queue-relaed error happened. | ++--------------------------+---------------+---------------------------------------------------+ +| QueueNotFoundError | QueueError | The targeted queue no longer exists. | ++--------------------------+---------------+---------------------------------------------------+ +| QueueEmptyError | | QueueError | The targeted queue is empty. | +| | | queue.Empty | | ++--------------------------+---------------+---------------------------------------------------+ +| QueueFullError | | QueueError | The targeted queue is full. | +| | | queue.Full | | ++--------------------------+---------------+---------------------------------------------------+ +| ItemInterpreterDestroyed | QueueError | The interpreter that added the next item is gone. | ++--------------------------+---------------+---------------------------------------------------+ + + +Basic Usage +----------- + +:: + + import interpreters.queues + + queue = interpreters.queues.create() + interp = interpreters.create() + interp.prepare_main(queue=queue) + + queue.put('spam!') + interp.exec(""" + msg = queue.get() + print(msg) + """) + +:: + + queue = interpreters.queues.create() + interp = interpreters.create() + interp.prepare_main(queue=queue) + + queue.put('spam!') + interp.exec(""" + obj = queue.get() + print(msg) + """) + +:: + + queue1 = interpreters.queues.create() + queue2 = interpreters.queues.create(syncobj=True) + interp = interpreters.create() + interp.prepare_main(queue1=queue1, queue2=queue2) + + queue1.put('spam!') + queue1.put('spam!', syncobj=True) + queue2.put('spam!') + queue2.put('spam!', syncobj=False) + interp.exec(""" + msg1 = queue1.get() + msg2 = queue1.get() + msg3 = queue2.get() + msg4 = queue2.get() + print(msg) + """) + + +Functions +--------- + +This module defines the following functions: + diff --git a/Doc/library/interpreters.rst b/Doc/library/interpreters.rst new file mode 100644 index 00000000000000..056a42ec6bc6e6 --- /dev/null +++ b/Doc/library/interpreters.rst @@ -0,0 +1,235 @@ +:mod:`!interpreters` --- Multiple Interpreters in the Same Process +================================================================== + +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + +.. module:: interpreters + :synopsis: Multiple Interpreters in the Same Process + +.. moduleauthor:: Eric Snow +.. sectionauthor:: Eric Snow + +.. versionadded:: 3.14 + +**Source code:** :source:`Lib/interpreters/__init__.py` + +-------------- + + +Introduction +------------ + +This module constructs higher-level interfaces on top of the lower +level ``_interpreters`` module. + +.. seealso:: + + :ref:`execcomponents` + ... + + :mod:`interpreters.queues` + cross-interpreter queues + + :mod:`interpreters.channels` + cross-interpreter channels + + :ref:`multiple-interpreters-howto` + how to use multiple interpreters + + :ref:`isolating-extensions-howto` + how to update an extension module to support multiple interpreters + + :pep:`554` + + :pep:`734` + + :pep:`684` + +.. include:: ../includes/wasm-notavail.rst + + +Key Details +----------- + +Before we dive into examples, there are a small number of details +to keep in mind about using multiple interpreters: + +* isolated, by default +* no implicit threads +* limited callables +* limited shareable objects +* not all PyPI packages support use in multiple interpreters yet +* ... + + +API Summary +----------- + ++----------------------------------+----------------------------------------------+ +| signature | description | ++==================================+==============================================+ +| ``list_all() -> [Interpreter]`` | Get all existing interpreters. | ++----------------------------------+----------------------------------------------+ +| ``get_current() -> Interpreter`` | Get the currently running interpreter. | ++----------------------------------+----------------------------------------------+ +| ``get_main() -> Interpreter`` | Get the main interpreter. | ++----------------------------------+----------------------------------------------+ +| ``create() -> Interpreter`` | Initialize a new (idle) Python interpreter. | ++----------------------------------+----------------------------------------------+ + +| + ++------------------------------------+---------------------------------------------------+ +| signature | description | ++====================================+===================================================+ +| ``class Interpreter`` | A single interpreter. | ++------------------------------------+---------------------------------------------------+ +| ``.id`` | The interpreter's ID (read-only). | ++------------------------------------+---------------------------------------------------+ +| ``.whence`` | Where the interpreter came from (read-only). | ++------------------------------------+---------------------------------------------------+ +| ``.is_running() -> bool`` | Is the interpreter currently executing code? | ++------------------------------------+---------------------------------------------------+ +| ``.close()`` | Finalize and destroy the interpreter. | ++------------------------------------+---------------------------------------------------+ +| ``.prepare_main(**kwargs)`` | Bind "shareable" objects in ``__main__``. | ++------------------------------------+---------------------------------------------------+ +| ``.exec(src_str, /, dedent=True)`` | | Run the given source code in the interpreter | +| | | (in the current thread). | ++------------------------------------+---------------------------------------------------+ +| ``.call(callable, /)`` | | Run the given function in the interpreter | +| | | (in the current thread). | ++------------------------------------+---------------------------------------------------+ +| ``.call_in_thread(callable, /)`` | | Run the given function in the interpreter | +| | | (in a new thread). | ++------------------------------------+---------------------------------------------------+ + +Exceptions: + ++--------------------------+------------------+---------------------------------------------------+ +| class | base class | description | ++==========================+==================+===================================================+ +| InterpreterError | Exception | An interpreter-relaed error happened. | ++--------------------------+------------------+---------------------------------------------------+ +| InterpreterNotFoundError | InterpreterError | The targeted interpreter no longer exists. | ++--------------------------+------------------+---------------------------------------------------+ +| ExecutionFailed | InterpreterError | The running code raised an uncaught exception. | ++--------------------------+------------------+---------------------------------------------------+ +| NotShareableError | Exception | The object cannot be sent to another interpreter. | ++--------------------------+------------------+---------------------------------------------------+ + +ExecutionFailed + excinfo + type + (builtin) + __name__ + __qualname__ + __module__ + msg + formatted + errdisplay + + +For communicating between interpreters: + ++-------------------------------------------------------------------+--------------------------------------------+ +| signature | description | ++===================================================================+============================================+ +| ``is_shareable(obj) -> Bool`` | | Can the object's data be passed | +| | | between interpreters? | ++-------------------------------------------------------------------+--------------------------------------------+ +| ``create_queue(*, syncobj=False, unbounditems=UNBOUND) -> Queue`` | | Create a new queue for passing | +| | | data between interpreters. | ++-------------------------------------------------------------------+--------------------------------------------+ +| ``create_channel(*, unbounditems=UNBOUND) -> (RecvChannel, SendChannel)`` | | Create a new channel for passing | +| | | data between interpreters. | ++-------------------------------------------------------------------+--------------------------------------------+ + +Also see :mod:`interpreters.queues` and :mod:`interpreters.channels`. + + +.. _interp-examples: + +Basic Usage +----------- + +Creating an interpreter and running code in it: + +:: + + import interpreters + + interp = interpreters.create() + + # Run in the current OS thread. + + interp.exec('print("spam!")') + + interp.exec(""" + print('spam!') + """) + + def run(): + print('spam!') + + interp.call(run) + + # Run in new OS thread. + + t = interp.call_in_thread(run) + t.join() + +For additional examples, see :ref:`interp-queue-examples`, +:ref:`interp-channel-examples`, and :ref:`multiple-interpreters-howto`. + + +Functions +--------- + +This module defines the following functions: + +... + + +.. _interp-shareable-objects + +Shareable Objects +----------------- + +A "shareable" object is one with a class that specifically supports +passing between interpreters. + +The primary characteristic of shareable objects is that their values +are guaranteed to always be in sync when "shared" between interpreters. +Effectively, they will be strictly equivalent and functionally +identical. + +That means one of the following is true: + +* the object is actually shared by the interpreters + (e.g. :const:`None`) +* the underlying data of the object is actually shared + (e.g. :class:`memoryview`) +* for immutable objects, the underlying data was copied + (e.g. :class:`str`) +* the underlying data of the corresponding object in each intepreter +is effectively always identical. + + +Passing a non-shareable object where one is expected results in a +:class:`NotShareableError` exception. You can use +:func:`interpreters.is_shareable` to know ahead of time if an +object can be passed between interpreters. + +By default, the following types are shareable: + +* :const:`None` +* :class:`bool` (:const:`True` and :const:`False`) +* :class:`bytes` +* :class:`str` +* :class:`int` +* :class:`float` +* :class:`tuple` (of shareable objects) +* :class:`memoryview` +* :class:`interpreters.queues.Queue` +* :class:`interpreters.channels.Channel` From e5e980ee0b2d132bc5c34b52c587d177fdea33ee Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 19 Jun 2025 10:53:19 -0600 Subject: [PATCH 02/19] Drop extraneous changes. --- Doc/library/concurrent.interpreters.rst | 5 +- Doc/library/interpreters.channels.rst | 150 --------------- Doc/library/interpreters.queues.rst | 148 --------------- Doc/library/interpreters.rst | 235 ------------------------ 4 files changed, 4 insertions(+), 534 deletions(-) delete mode 100644 Doc/library/interpreters.channels.rst delete mode 100644 Doc/library/interpreters.queues.rst delete mode 100644 Doc/library/interpreters.rst diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index 524d505bcf144f..ae2364139f1494 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -31,7 +31,10 @@ Actual concurrency is available separately through :class:`~concurrent.futures.InterpreterPoolExecutor` combines threads with interpreters in a familiar interface. - .. XXX Add references to the upcoming HOWTO docs in the seealso block. + :ref:`multiple-interpreters-howto` + how to use multiple interpreters + + .. XXX Add a reference to the upcoming concurrency HOWTO doc. :ref:`isolating-extensions-howto` how to update an extension module to support multiple interpreters diff --git a/Doc/library/interpreters.channels.rst b/Doc/library/interpreters.channels.rst deleted file mode 100644 index 0b9b1e76bf1bfd..00000000000000 --- a/Doc/library/interpreters.channels.rst +++ /dev/null @@ -1,150 +0,0 @@ -:mod:`!interpreters.channels` -- A cross-interpreter "channel" implementation -================================================================== - -XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - -.. module:: interpreters.channels - :synopsis: Multiple Interpreters in the Same Process - -.. moduleauthor:: Eric Snow -.. sectionauthor:: Eric Snow - -.. versionadded:: 3.14 - -**Source code:** :source:`Lib/interpreters/channels.py` - --------------- - - -Introduction ------------- - -This module constructs higher-level interfaces on top of the lower -level ``_interpchannels`` module. - -.. seealso:: - - :ref:`execcomponents` - ... - - :mod:`interpreters` - provides details about communicating between interpreters. - - :ref:`multiple-interpreters-howto` - demonstrates how to use channels - -.. include:: ../includes/wasm-notavail.rst - - -API Summary ------------ - -+-------------------------------------------------------------------+----------------------------------------------+ -| signature | description | -+===================================================================+==============================================+ -| ``list_all() -> [(RecvChannel, SendChannel)]`` | Get all existing cross-interpreter channels. | -+-------------------------------------------------------------------+----------------------------------------------+ -| ``create(*, unbounditems=UNBOUND) -> (RecvChannel, SendChannel)`` | Initialize a new cross-interpreter channel. | -+-------------------------------------------------------------------+----------------------------------------------+ - -| - -+-------------------------------------------+---------------------------------------------------+ -| signature | description | -+===========================================+===================================================+ -| ``class RecvChannel`` | The receiving end of a cross-interpreter channel. | -+-------------------------------------------+---------------------------------------------------+ -| ``.id`` | The channel's ID (read-only). | -+-------------------------------------------+---------------------------------------------------+ -| ``.is_closed`` | If the channel has been closed (read-only). | -+-------------------------------------------+---------------------------------------------------+ -| ``.recv(timeout=None)`` | Pop off the next item in channel. | -+-------------------------------------------+---------------------------------------------------+ -| ``.recv_nowait(default=None)`` | Pop off the next item in channel, if any. | -+-------------------------------------------+---------------------------------------------------+ -| ``.close()`` | Stop receiving more items. | -+-------------------------------------------+---------------------------------------------------+ - -| - -+------------------------------------------------------------+--------------------------------------------------+ -| signature | description | -+============================================================+==================================================+ -| ``class SendChannel`` | The sending end of a cross-interpreter channel. | -+------------------------------------------------------------+--------------------------------------------------+ -| ``.id`` | The channel's ID (read-only). | -+------------------------------------------------------------+--------------------------------------------------+ -| ``.is_closed`` | If the channel has been closed (read-only). | -+------------------------------------------------------------+--------------------------------------------------+ -| ``.send(obj, timeout=None, *, unbound=None)`` | Add an item to the back of the channel's queue. | -+------------------------------------------------------------+--------------------------------------------------+ -| ``.send_nowait(obj, *, unbound=None)`` | Add an item to the back of the channel's queue. | -+------------------------------------------------------------+--------------------------------------------------+ -| ``.send_buffer(obj, timeout=None, *, unbound=None)`` | Add an item to the back of the channel's queue. | -+------------------------------------------------------------+--------------------------------------------------+ -| ``.send_buffer_nowait(obj, *, unbound=None)`` | Add an item to the back of the channel's queue. | -+------------------------------------------------------------+--------------------------------------------------+ -| ``.close()`` | Stop sending more items. | -+------------------------------------------------------------+--------------------------------------------------+ - -Exceptions: - -+--------------------------+--------------+---------------------------------------------------+ -| class | base class | description | -+==========================+==============+===================================================+ -| ChannelError | Exception | A channel-relaed error happened. | -+--------------------------+--------------+---------------------------------------------------+ -| ChannelNotFoundError | ChannelError | The targeted channel no longer exists. | -+--------------------------+--------------+---------------------------------------------------+ -| ChannelClosedError | ChannelError | The targeted channel has been closed. | -+--------------------------+--------------+---------------------------------------------------+ -| ChannelEmptyError | ChannelError | The targeted channel is empty. | -+--------------------------+--------------+---------------------------------------------------+ -| ChannelNotEmptyError | ChannelError | The targeted channel should have been empty. | -+--------------------------+--------------+---------------------------------------------------+ -| ItemInterpreterDestroyed | ChannelError | The interpreter that added the next item is gone. | -+--------------------------+--------------+---------------------------------------------------+ - - -Basic Usage ------------ - -:: - - import interpreters.channels - - rch, sch = interpreters.channels.create() - interp = interpreters.create() - interp.prepare_main(rch=rch) - - sch.send_nowait('spam!') - interp.exec(""" - msg = rch.recv() - print(msg) - """) - -:: - - rch, sch = interpreters.channels.create() - interp = interpreters.create() - interp.prepare_main(rch=rch) - - data = bytearray(100) - - sch.send_buffer_nowait(data) - interp.exec(""" - data = rch.recv() - for i in range(len(data)): - data[i] = i - """) - assert len(data) == 100 - for i in range(len(data)): - assert data[i] == i - - -Functions ---------- - -This module defines the following functions: - - diff --git a/Doc/library/interpreters.queues.rst b/Doc/library/interpreters.queues.rst deleted file mode 100644 index 405ed924948325..00000000000000 --- a/Doc/library/interpreters.queues.rst +++ /dev/null @@ -1,148 +0,0 @@ -:mod:`!interpreters.queues` -- A cross-interpreter queue implementation -================================================================== - -XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - -.. module:: interpreters.queues - :synopsis: Multiple Interpreters in the Same Process - -.. moduleauthor:: Eric Snow -.. sectionauthor:: Eric Snow - -.. versionadded:: 3.14 - -**Source code:** :source:`Lib/interpreters/queues.py` - --------------- - - -Introduction ------------- - -This module constructs higher-level interfaces on top of the lower -level ``_interpqueues`` module. - -.. seealso:: - - :ref:`execcomponents` - ... - - :mod:`interpreters` - provides details about communicating between interpreters. - - :ref:`multiple-interpreters-howto` - demonstrates how to use queues - -.. include:: ../includes/wasm-notavail.rst - - -API Summary ------------ - -+-------------------------------------------------------------+--------------------------------------------+ -| signature | description | -+=============================================================+============================================+ -| ``list_all() -> [Queue]`` | Get all existing cross-interpreter queues. | -+-------------------------------------------------------------+--------------------------------------------+ -| ``create(*, syncobj=False, unbounditems=UNBOUND) -> Queue`` | Initialize a new cross-interpreter queue. | -+-------------------------------------------------------------+--------------------------------------------+ - -| - -+------------------------------------------------------------+---------------------------------------------------+ -| signature | description | -+============================================================+===================================================+ -| ``class Queue`` | A single cross-interpreter queue. | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.id`` | The queue's ID (read-only). | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.maxsize`` | The queue's capacity, if applicable (read-only). | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.qsize() -> bool`` | The queue's current item count. | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.empty() -> bool`` | Does the queue currently have no items it it? | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.full() -> bool`` | Has the queue reached its maxsize, if any? | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.put(obj, timeout=None, *, syncobj=None, unbound=None)`` | Add an item to the back of the queue. | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.put_nowait(obj, *, syncobj=None, unbound=None)`` | Add an item to the back of the queue. | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.get(timeout=None) -> object`` | Pop off the next item in queue. | -+------------------------------------------------------------+---------------------------------------------------+ -| ``.get_nowait() -> object`` | Pop off the next item in queue. | -+------------------------------------------------------------+---------------------------------------------------+ - -Exceptions: - -+--------------------------+---------------+---------------------------------------------------+ -| class | base class | description | -+==========================+==============+====================================================+ -| QueueError | Exception | A queue-relaed error happened. | -+--------------------------+---------------+---------------------------------------------------+ -| QueueNotFoundError | QueueError | The targeted queue no longer exists. | -+--------------------------+---------------+---------------------------------------------------+ -| QueueEmptyError | | QueueError | The targeted queue is empty. | -| | | queue.Empty | | -+--------------------------+---------------+---------------------------------------------------+ -| QueueFullError | | QueueError | The targeted queue is full. | -| | | queue.Full | | -+--------------------------+---------------+---------------------------------------------------+ -| ItemInterpreterDestroyed | QueueError | The interpreter that added the next item is gone. | -+--------------------------+---------------+---------------------------------------------------+ - - -Basic Usage ------------ - -:: - - import interpreters.queues - - queue = interpreters.queues.create() - interp = interpreters.create() - interp.prepare_main(queue=queue) - - queue.put('spam!') - interp.exec(""" - msg = queue.get() - print(msg) - """) - -:: - - queue = interpreters.queues.create() - interp = interpreters.create() - interp.prepare_main(queue=queue) - - queue.put('spam!') - interp.exec(""" - obj = queue.get() - print(msg) - """) - -:: - - queue1 = interpreters.queues.create() - queue2 = interpreters.queues.create(syncobj=True) - interp = interpreters.create() - interp.prepare_main(queue1=queue1, queue2=queue2) - - queue1.put('spam!') - queue1.put('spam!', syncobj=True) - queue2.put('spam!') - queue2.put('spam!', syncobj=False) - interp.exec(""" - msg1 = queue1.get() - msg2 = queue1.get() - msg3 = queue2.get() - msg4 = queue2.get() - print(msg) - """) - - -Functions ---------- - -This module defines the following functions: - diff --git a/Doc/library/interpreters.rst b/Doc/library/interpreters.rst deleted file mode 100644 index 056a42ec6bc6e6..00000000000000 --- a/Doc/library/interpreters.rst +++ /dev/null @@ -1,235 +0,0 @@ -:mod:`!interpreters` --- Multiple Interpreters in the Same Process -================================================================== - -XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - -.. module:: interpreters - :synopsis: Multiple Interpreters in the Same Process - -.. moduleauthor:: Eric Snow -.. sectionauthor:: Eric Snow - -.. versionadded:: 3.14 - -**Source code:** :source:`Lib/interpreters/__init__.py` - --------------- - - -Introduction ------------- - -This module constructs higher-level interfaces on top of the lower -level ``_interpreters`` module. - -.. seealso:: - - :ref:`execcomponents` - ... - - :mod:`interpreters.queues` - cross-interpreter queues - - :mod:`interpreters.channels` - cross-interpreter channels - - :ref:`multiple-interpreters-howto` - how to use multiple interpreters - - :ref:`isolating-extensions-howto` - how to update an extension module to support multiple interpreters - - :pep:`554` - - :pep:`734` - - :pep:`684` - -.. include:: ../includes/wasm-notavail.rst - - -Key Details ------------ - -Before we dive into examples, there are a small number of details -to keep in mind about using multiple interpreters: - -* isolated, by default -* no implicit threads -* limited callables -* limited shareable objects -* not all PyPI packages support use in multiple interpreters yet -* ... - - -API Summary ------------ - -+----------------------------------+----------------------------------------------+ -| signature | description | -+==================================+==============================================+ -| ``list_all() -> [Interpreter]`` | Get all existing interpreters. | -+----------------------------------+----------------------------------------------+ -| ``get_current() -> Interpreter`` | Get the currently running interpreter. | -+----------------------------------+----------------------------------------------+ -| ``get_main() -> Interpreter`` | Get the main interpreter. | -+----------------------------------+----------------------------------------------+ -| ``create() -> Interpreter`` | Initialize a new (idle) Python interpreter. | -+----------------------------------+----------------------------------------------+ - -| - -+------------------------------------+---------------------------------------------------+ -| signature | description | -+====================================+===================================================+ -| ``class Interpreter`` | A single interpreter. | -+------------------------------------+---------------------------------------------------+ -| ``.id`` | The interpreter's ID (read-only). | -+------------------------------------+---------------------------------------------------+ -| ``.whence`` | Where the interpreter came from (read-only). | -+------------------------------------+---------------------------------------------------+ -| ``.is_running() -> bool`` | Is the interpreter currently executing code? | -+------------------------------------+---------------------------------------------------+ -| ``.close()`` | Finalize and destroy the interpreter. | -+------------------------------------+---------------------------------------------------+ -| ``.prepare_main(**kwargs)`` | Bind "shareable" objects in ``__main__``. | -+------------------------------------+---------------------------------------------------+ -| ``.exec(src_str, /, dedent=True)`` | | Run the given source code in the interpreter | -| | | (in the current thread). | -+------------------------------------+---------------------------------------------------+ -| ``.call(callable, /)`` | | Run the given function in the interpreter | -| | | (in the current thread). | -+------------------------------------+---------------------------------------------------+ -| ``.call_in_thread(callable, /)`` | | Run the given function in the interpreter | -| | | (in a new thread). | -+------------------------------------+---------------------------------------------------+ - -Exceptions: - -+--------------------------+------------------+---------------------------------------------------+ -| class | base class | description | -+==========================+==================+===================================================+ -| InterpreterError | Exception | An interpreter-relaed error happened. | -+--------------------------+------------------+---------------------------------------------------+ -| InterpreterNotFoundError | InterpreterError | The targeted interpreter no longer exists. | -+--------------------------+------------------+---------------------------------------------------+ -| ExecutionFailed | InterpreterError | The running code raised an uncaught exception. | -+--------------------------+------------------+---------------------------------------------------+ -| NotShareableError | Exception | The object cannot be sent to another interpreter. | -+--------------------------+------------------+---------------------------------------------------+ - -ExecutionFailed - excinfo - type - (builtin) - __name__ - __qualname__ - __module__ - msg - formatted - errdisplay - - -For communicating between interpreters: - -+-------------------------------------------------------------------+--------------------------------------------+ -| signature | description | -+===================================================================+============================================+ -| ``is_shareable(obj) -> Bool`` | | Can the object's data be passed | -| | | between interpreters? | -+-------------------------------------------------------------------+--------------------------------------------+ -| ``create_queue(*, syncobj=False, unbounditems=UNBOUND) -> Queue`` | | Create a new queue for passing | -| | | data between interpreters. | -+-------------------------------------------------------------------+--------------------------------------------+ -| ``create_channel(*, unbounditems=UNBOUND) -> (RecvChannel, SendChannel)`` | | Create a new channel for passing | -| | | data between interpreters. | -+-------------------------------------------------------------------+--------------------------------------------+ - -Also see :mod:`interpreters.queues` and :mod:`interpreters.channels`. - - -.. _interp-examples: - -Basic Usage ------------ - -Creating an interpreter and running code in it: - -:: - - import interpreters - - interp = interpreters.create() - - # Run in the current OS thread. - - interp.exec('print("spam!")') - - interp.exec(""" - print('spam!') - """) - - def run(): - print('spam!') - - interp.call(run) - - # Run in new OS thread. - - t = interp.call_in_thread(run) - t.join() - -For additional examples, see :ref:`interp-queue-examples`, -:ref:`interp-channel-examples`, and :ref:`multiple-interpreters-howto`. - - -Functions ---------- - -This module defines the following functions: - -... - - -.. _interp-shareable-objects - -Shareable Objects ------------------ - -A "shareable" object is one with a class that specifically supports -passing between interpreters. - -The primary characteristic of shareable objects is that their values -are guaranteed to always be in sync when "shared" between interpreters. -Effectively, they will be strictly equivalent and functionally -identical. - -That means one of the following is true: - -* the object is actually shared by the interpreters - (e.g. :const:`None`) -* the underlying data of the object is actually shared - (e.g. :class:`memoryview`) -* for immutable objects, the underlying data was copied - (e.g. :class:`str`) -* the underlying data of the corresponding object in each intepreter -is effectively always identical. - - -Passing a non-shareable object where one is expected results in a -:class:`NotShareableError` exception. You can use -:func:`interpreters.is_shareable` to know ahead of time if an -object can be passed between interpreters. - -By default, the following types are shareable: - -* :const:`None` -* :class:`bool` (:const:`True` and :const:`False`) -* :class:`bytes` -* :class:`str` -* :class:`int` -* :class:`float` -* :class:`tuple` (of shareable objects) -* :class:`memoryview` -* :class:`interpreters.queues.Queue` -* :class:`interpreters.channels.Channel` From 2fcaeddff2abc35e373717d99227db7369c94756 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 20 Jun 2025 10:25:19 -0600 Subject: [PATCH 03/19] Finish howto doc, minus recipes. --- Doc/howto/multiple-interpreters.rst | 1084 +++++++++++++++++---------- 1 file changed, 696 insertions(+), 388 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 4f319742af0a79..340565920f1f24 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -4,102 +4,90 @@ Multiple Interpreters HOWTO *************************** -When it comes to concurrency and parallelism, -Python provides a number of options, each with various tradeoffs. -This includes threads, async, and multiprocessing. -There's one more option: multiple interpreters (AKA "subinterpreters"). +In this HOWTO document we'll look at how to take advantage of +multiple interpreters in a Python program. We will focus on doing +so in Python code, through the stdlib :mod:`concurrent.interpreters` +module. -.. (PyPI packages expand that with "greenlets", and distributed computing.) +This document has 3 parts: first a brief introduction, then a tutorial, +and then a set of recipes and practical examples. -Multiple interpreters in the same process provide conceptual isolation -like processes but more efficiently, like threads. In fact, you can -think of them like threads with opt-in sharing. - -That provides a concurrency model which is easier to reason about, -similar to CSP or the actor model. -Furthermore, each interpreter has its own GIL, so they provide -true multi-core parallelism. - -This HOWTO document has 2 parts: first a tutorial and then a set -of recipes and practical examples. - -The tuturial covers the basics of using multiple interpreters +The tutorial covers the basics of using multiple interpreters (:ref:`interp-tutorial-basics`), as well as how to communicate between interpreters (:ref:`interp-tutorial-communicating`). The examples section (:ref:`interp-recipes`) is focused on providing effective solutions for real concurrency use cases. This includes comparisons with Python's various existing concurrency models. -.. currentmodule:: interpreters +.. currentmodule:: concurrent.interpreters .. seealso:: + :mod:`concurrent.interpreters` + :ref:`execcomponents` - :mod:`interpreters` - :mod:`interpreters.queues` - :mod:`interpreters.channels` + more details about how interpreters fit into + Python's execution model + + .. XXX Add a reference to the upcoming concurrency HOWTO doc. .. note:: This page is focused on *using* multiple interpreters. For information about changing an extension module to work with multiple interpreters, see :ref:`isolating-extensions-howto`. - +Introduction +============ +You can find a thorough explanation about what interpreters are and +what they're for in the :mod:`concurrent.interpreters ` +docs. +In summary, think of an interpreter as the Python runtime's execution +context. You can have multiple interpreters in a single process, +switching between them at any time. Each interpreter is almost +completely isolated from the others. -Why Use Multiple Interpreters? ------------------------------- - -This provides a similar - -We can take advantage of the isolation and independence of multiple -processes, by using :mod:`multiprocessing` or :mod:`subprocess` -(with sockets and pipes) -In the same process, We have threads, - - -For concurrent code, imagine combining the efficiency of threads -with the isolation -You can run multiple isolated Python execution environments -in the same process - - - - -Python can actually run multiple interpreters at the same time, -which lets you have independent, isolated execution environments -in the same process. This has been true since Python 2.2, but -the feature was only available through the C-API and not well known; -and the isolation was incomplete. However... +.. note:: -As of Python 3.12, isolation has been fixed, -to the point that interpreters in the same process no longer -even share the GIL (global interpreter lock). That means you can + Interpreters in the same process can technically never be strictly + isolated from one another since there are few restrictions on memory + access within the same process. The Python runtime makes a best + effort at isolation but extension modules may easily violate that. + Therefore, do not use multiple interpreters in security-senstive + situations, where they shouldn't have access to each other's data. -has had the ability to run multiple copies +That isolation facilitates a concurrency model based an independent +logical threads of execution, like CSP or the actor model. -Benefits: +Each actual thread in Python, even if you're only running in the main +thread, has its own *current* execution context. Multiple threads can +use the same interpreter or different ones. -Downsides: +Why Use Multiple Interpreters? +------------------------------ -... +These are the main benefits: +* isolation supports a human-friendly concurrency model +* isolation supports full multi-core parallelism +* avoids the data races that make threads so challenging normally -When to Use Multiple Interpreters ---------------------------------- +There are some downsides and temporary limitations: -... +* the concurrency model requires extra rigor; it enforces higher + discipline about how the isolated components in your program interact +* not all PyPI extension modules support multiple interpreters yet +* the existing tools for passing data between interpreters safely + is still relatively inefficient and limited +* actually *sharing* data safely is tricky (true for free-threading too) +* all necessary modules must be imported separately in each interpreter +* relatively slow startup time per interpreter +* non-trivial extra memory usage per interpreter -+-------------------------------------+--------------------------------------+ -| Task you want to perform | The best tool for the task | -+=====================================+======================================+ -| ... | ... | -+-------------------------------------+--------------------------------------+ -| ... | ... | -+-------------------------------------+--------------------------------------+ +Nearly all of these should improve in subsequent Python releases. .. _interp-tutorial-basics: @@ -110,54 +98,64 @@ Tutorial: Basics First of all, keep in mind that using multiple interpreters is like using multiple processes. They are isolated and independent from each other. The main difference is that multiple interpreters live in the -same process, which makes it all more efficient. +same process, which makes it all more efficient and use fewer +system resources. -Each interpreter has its own ``__main__`` module, its own -:func:`sys.modules`, and, in fact, its own set of *all* runtime state. +Each interpreter has its own :mod:`!__main__` module, its own +:data:`sys.modules`, and, in fact, its own set of *all* runtime state. Interpreters pretty much never share objects and very little data. Running Code in an Interpreter ------------------------------ Running code in an interpreter is basically equivalent to running the -buitlin :func:`exec` using that interpreter:: +builtin :func:`exec` using that interpreter:: - import interpreters + from concurrent import interpreters interp = interpreters.create() interp.exec('print("spam!")') - interp.exec(""" + # prints: spam! + + interp.exec("""if True: + ... print('spam!') + ... """) + # prints: spam! -Calling a function in an interpreter works the same way:: +(See :meth:`Interpreter.exec`.) - import interpreters +Calling a simple function in an interpreter works the same way:: + + from concurrent import interpreters interp = interpreters.create() def script(): print('spam!') interp.call(script) + # prints: spam! + +(See :meth:`Interpreter.call`.) When it runs, the code is executed using the interpreter's ``__main__`` -module, just like a Python process normally does:: +module, just like a Python process normally does when invoked from +the command-line:: - import interpreters + from concurrent import interpreters print(__name__) - # __main__ + # prints: __main__ interp = interpreters.create() - interp.exec(""" - print(__name__) - """) - # __main__ + interp.exec('print(__name__)') + # prints: __main__ In fact, a comparison with ``python -c`` is quite direct:: - import interpreters + from concurrent import interpreters ############################ # python -c 'print("spam!")' @@ -168,12 +166,15 @@ In fact, a comparison with ``python -c`` is quite direct:: It's also fairly easy to simulate the other forms of the Python CLI:: - import interpreters + from textwrap import dedent + from concurrent import interpreters - with open('script.py', 'w') as outfile: - outfile.write(""" + SCRIPT = """ print('spam!') - """) + """ + + with open('script.py', 'w') as outfile: + outfile.write(SCRIPT) ################## # python script.py @@ -189,14 +190,230 @@ It's also fairly easy to simulate the other forms of the Python CLI:: ################## interp = interpreters.create() - interp.exec(""" + interp.exec(dedent(""" import runpy runpy.run_module('script') - """) + """)) That's more or less what the ``python`` executable is doing for each of those cases. +Calling a Function in an Interpreter +------------------------------------ + +You can just as easily call a function in another interpreter:: + + from concurrent import interpreters + + interp = interpreters.create() + + def spam(): + print('spam!') + interp.call(spam) + # prints: spam! + +In fact, nearly all Python functions and callables are supported, +with the notable exception of closures. Support includes arguments +and return values, which we'll explore soon. + +Builtin functions always execute in the target interpreter's +:mod:`!__main__` module:: + + from concurrent import interpreters + + interp = interpreters.create() + + interp.call(exec, 'print(__name__)') + # prints: __main__ + + interp.call(eval, 'print(__name__)') + # prints: __main__ + +The same is true for Python functions that don't use any globals:: + + from concurrent import interpreters + + interp = interpreters.create() + + def spam(): + # globals is a builtin. + print(globals()['__name__']) + interp.call(spam) + # prints: __main__ + +There are very few cases where that matters, though. + +Otherwise, functions defined in modules other than :mod:`!__main__` run +in that module:: + + from concurrent import interpreters + from mymod import spam + + interp = interpreters.create() + interp.call(spam) + # prints: mymod + + ########## + # mymod.py + ########## + + def spam(): + print(__name__) + +For a function actually defined in the :mod:`!__main__` module, +it executes in a separate dummy module, in order to not pollute +the :mod:`!__main__` module:: + + from concurrent import interpreters + + interp = interpreters.create() + + def spam(): + print(__name__) + interp.call(spam) + # prints: '' + +This means global state used in such a function won't be reflected +in the interpreter's :mod:`!__main__` module:: + + from textwrap import dedent + from concurrent import interpreters + + interp = interpreters.create() + + total = 0 + + def inc(): + global total + total += 1 + def show_total(): + print(total) + + interp.call(show_total) + # prints: 0 + interp.call(inc) + interp.call(show_total) + # prints: 1 + interp.exec(dedent("""' + try: + print(total) + except NameError: + pass + else: + raise AssertionError('expected NameError') + """)) + + interp.exec('total = -1') + interp.exec('print(total)') + # prints: -1 + interp.call(show_total) + # prints: 1 + interp.call(inc) + interp.call(show_total) + # prints: 2 + interp.exec('print(total)') + # prints: -1 + + print(total) + # prints: 0 + + +Calling Methods and Other Objects in an Interpreter +--------------------------------------------------- + +Pretty much any callable object may be run in an interpreter, following +the same rules as functions:: + + from concurrent import interpreters + from mymod import Spam + + interp = interpreters.create() + + spam = Spam() + interp.call(Spam.modname) + # prints: mymod + interp.call(spam) + # prints: mymod + interp.call(spam.spam) + # prints: mymod + + class Eggs: + @classmethod + def modname(cls): + print(__name__) + def __call__(self): + print(__name__) + def eggs(self): + print(__name__) + + eggs = Eggs() + res = interp.call(Eggs.modname) + # prints: + res = interp.call(eggs) + # prints: + res = interp.call(eggs.eggs) + # prints: + + ########## + # mymod.py + ########## + + class Spam: + @classmethod + def modname(cls): + print(__name__) + def __call__(self): + print(__name__) + def spam(self): + print(__name__) + +Mutable State is not Shared +--------------------------- + +Just be be clear, the underlying data of very few mutable objects is +actually shared between interpreters. The notable exceptions are +:class:`Queue` and :class:`memoryview`, which we will explore in a +little while. In nearly every case, the raw data is copied in +the other interpreter and never automatically synchronized:: + + from concurrent import interpreters + + interp = interpreters.create() + + class Counter: + def __init__(self, initial=0): + self.value = initial + def inc(self): + self.value += 1 + def dec(self): + self.value -= 1 + def show(self): + print(self.value) + counter = Counter(17) + + interp.call(counter.show) + # prints: 17 + counter.show() + # prints: 17 + + interp.call(counter.inc) + interp.call(counter.show) + # prints: 18 + counter.show() + # prints: 17 + + interp.call(counter.inc) + interp.call(counter.show) + # prints: 18 + counter.show() + # prints: 17 + + interp.call(counter.inc) + interp.call(counter.show) + # prints: 18 + counter.show() + # prints: 17 + Preparing and Reusing an Interpreter ------------------------------------ @@ -205,29 +422,35 @@ That would make the REPL much less useful. Likewise, when you use the builtin :func:`exec`, it doesn't reset the namespace it uses to run the code. -In the same way, running code in an interpreter does not reset it. -The next time you run code in that interpreter, the ``__main__`` module -will be in exactly the state in which you left it:: +In the same way, running code in an interpreter does not reset that +interpreter.. The next time you run code in that interpreter, the +:mod:`!__main__` module will be in exactly the state in which you +left it:: - import interpreters + from concurrent import interpreters interp = interpreters.create() - interp.exec(""" + interp.exec("""if True answer = 42 """) - interp.exec(""" + interp.exec("""if True assert answer == 42 """) + def script(): + assert answer == 42 + interp.call(script) + You can take advantage of this to prepare an interpreter ahead of time or use it incrementally:: - import interpreters + from textwrap import dedent + from concurrent import interpreters interp = interpreters.create() # Prepare the interpreter. - interp.exec(""" + interp.exec(dedent(""" # We will need this later. import math @@ -236,26 +459,25 @@ or use it incrementally:: def double(val): return val + val - """) + """)) # Do the work. - for _ in range(10): - interp.exec(""" + for _ in range(9): + interp.exec(dedent(""" assert math.factorial(value + 1) >= double(value) value = double(value) - """) + """)) # Show the result. - interp.exec(""" - print(value) - """) + interp.exec('print(value)') + # prints: 1024 In case you're curious, in a little while we'll look at how to pass data in and out of an interpreter (instead of just printing things). -Sometimes you might also have values that you want to use in another -interpreter. We'll look more closely at that case in a little while: -:ref:`interp-script-args`. +Sometimes you might also have values that you want to use in a later +script in another interpreter. We'll look more closely at that +case in a little while: :ref:`interp-script-args`. Running in a Thread ------------------- @@ -268,9 +490,9 @@ In the same way, running code in another interpreter doesn't create (or need) a new thread. It uses the current OS thread, switching to the interpreter long enough to run the code. -To actually run in a new thread, you do it explicitly:: +To actually run in a new thread, you can do it explicitly:: - import interpreters + from concurrent import interpreters import threading interp = interpreters.create() @@ -282,6 +504,8 @@ To actually run in a new thread, you do it explicitly:: t.join() def run(): + def script(): + print('spam!') interp.call(script) t = threading.Thread(target=run) t.start() @@ -289,7 +513,7 @@ To actually run in a new thread, you do it explicitly:: There's also a helper method for that:: - import interpreters + from concurrent import interpreters interp = interpreters.create() @@ -309,7 +533,7 @@ The behavior is very similar when code is run in an interpreter. The traceback get printed and, rather than a failure code, an :class:`ExecutionFailed` exception is raised:: - import interpreters + from concurrent import interpreters interp = interpreters.create() @@ -321,7 +545,7 @@ an :class:`ExecutionFailed` exception is raised:: The exception also has some information, which you can use to handle it with more specificity:: - import interpreters + from concurrent import interpreters interp = interpreters.create() @@ -336,27 +560,29 @@ it with more specificity:: You can handle the exception in the interpreter as well, like you might do with threads or a script:: - import interpreters + from textwrap import dedent + from concurrent import interpreters interp = interpreters.create() - interp.exec(""" + interp.exec(dedent(""" try: 1/0 except ZeroDivisionError: ... - """) + """)) At the moment there isn't an easy way to "re-raise" an unhandled exception from another interpreter. Here's one approach that works for exceptions that can be pickled:: - import interpreters + from textwrap import dedent + from concurrent import interpreters interp = interpreters.create() try: - interp.exec(""" + interp.exec(dedent(""" try: 1/0 except Exception as exc: @@ -365,7 +591,7 @@ exceptions that can be pickled:: class PickledException(Exception): pass raise PickledException(data) - """) + """)) except interpreters.ExecutionFailed as exc: if exc.excinfo.type.__name__ == 'PickledException': import pickle @@ -381,13 +607,11 @@ cleaned up as soon as you're done with the interpreter. While you can wait until the interpreter is cleaned up when the process exits, you can explicitly clean it up sooner:: - import interpreters + from concurrent import interpreters interp = interpreters.create() try: - interp.exec(""" - print('spam!') - """) + interp.exec('print("spam!")') finally: interp.close() @@ -396,26 +620,18 @@ concurrent.futures.InterpreterPoolExecutor The :mod:`concurrent.futures` module is a simple, popular concurrency framework that wraps both threading and multiprocessing. You can also -use it with multiple interpreters:: +use it with multiple interpreters, via +:class:`~concurrent.futures.InterpreterPoolExecutor`:: from concurrent.futures import InterpreterPoolExecutor - with InterpreterPoolExecutor(max_workers=5) as executor: - executor.submit('print("spam!")') - -Gotchas -------- - -... - -Keep in mind that there is a limit to the kinds of functions that may -be called by an interpreter. Specifically, ... - + def script(): + return 'spam!' -* limited callables -* limited shareable objects -* not all PyPI packages support use in multiple interpreters yet -* ... + with InterpreterPoolExecutor(max_workers=5) as executor: + fut = executor.submit(script) + res = fut.result() + # res: 'spam!' .. _interp-tutorial-communicating: @@ -431,9 +647,186 @@ between interpreters. In this half of the tutorial, we explore various ways you can do so. +Call Args and Return Values +--------------------------- + +As already noted, :meth:`Interpreter.call` supports most callables, +as well as arguments and return values. + +Arguments provide a way to send information to an interpreter:: + + from concurrent import interpreters + + interp = interpreters.create() + + def show(arg): + print(arg) + interp.call(show, 'spam!') + # prints: spam! + + def ident_full(a, /, b, c=42, *args, d, e='eggs', **kwargs): + print([a, b, c, d, e], args, kwargs) + interp.call(ident_full, 1, 2, 3, 4, 5, d=6, e=7, f=8, g=9) + # prints: [1, 2, 3, 6, 7] (4, 5) {'f': 8, 'g': 9} + + def handle_request(req): + # do the work + ... + req = ... + interp.call(handle_request, req) + +Return values are a way an interpreter can send information back:: + + from concurrent import interpreters + + interp = interpreters.create() + + data = {} + + def put(key, value): + data[key] = value + def get(key, default=None): + return data.get(key, default) + + res = interp.call(get, 'spam') + # res: None + res = interp.call(get, 'spam', -1) + # res: -1 + interp.call(put, 'spam', True) + res = interp.call(get, 'spam') + # res: True + interp.call(put, 'spam', 42) + res = interp.call(get, 'spam') + # res: 42 + +Don't forget that the underlying data of few objects is actually shared +between interpreters. That means that, nearly always, arguments are +copied on the way in and return values on the way out. Furthermore, +this will happen with each call, so it's a new copy every time +and no state persists between calls. + +For example:: + + from concurrent import interpreters + + interp = interpreters.create() + + data = { + 'a': 1, + 'b': 2, + 'c': 3, + } + + interp.call(data.clear) + assert data == dict(a=1, b=2, c=3) + + def update_and_copy(data, **updates): + data.update(updates) + return dict(data) + + res = interp.call(update_and_copy, data) + assert res == dict(a=1, b=2, c=3) + assert res is not data + assert data == dict(a=1, b=2, c=3) + + res = interp.call(update_and_copy, data, d=4, e=5) + assert res == dict(a=1, b=2, c=3, d=4, e=5) + assert data == dict(a=1, b=2, c=3) + + res = interp.call(update_and_copy, data) + assert res == dict(a=1, b=2, c=3) + assert res is not data + assert data == dict(a=1, b=2, c=3) + +Supported and Unsupported Objects +--------------------------------- + +There are other ways of communicating between interpreters, like +:meth:`Interpreter.prepare_main` and :class:`Queue`, which we'll +cover shortly. Before that, we should talk about which objects are +supported by :meth:`Interpreter.call` and the others. + +Most objects are supported. This includes anything that can be +:mod:`pickled `, though for some pickleable objects we copy the +object in a more efficient way. Aside from pickleable objects, there +is a small set of other objects that are supported, such as +non-closure inner functions. + +If you try to pass an unsupported object as an argument to +:meth:`Interpreter.call` then you will get a :exc:`NotShareableError` +with ``__cause__`` set appropriately. :meth:`Interpreter.call` will +also raise that way if the return value is not supported. The same +goes for :meth:`~Interpreter.prepare_main` and :class:`Queue.* `. + +Relatedly, there's an interesting side-effect to how we use a fake +module for objects with a class defined in :mod:`!__main__`. If you +pass the class to :meth:`Interpreter.call`, which will create an +instance in the other interpreter, then that instance will fail to +unpickle in the original interpreter, due to the fake module name:: + + from concurrent import interpreters + + class Spam: + def __init__(self, x): + self.x = x + + interp = interpreters.create() + try: + spam = interp.call(Spam, 10) + except interpreters.NotShareableError: + pass + else: + raise AssertionError('unexpected success') + +Sharing Data +------------ + +There are actually a small number of objects that aren't just copied and +for which the underlying data is actually shared between interpreters. +We'll talk about :class:`Queue` in the next section. + +Another example is :class:`memoryview`, where the underlying +:ref:`buffer ` can be read and written by multiple +interpreters at once. Users are responsible to manage thread-safety +for that data in such cases. + +Support for actually sharing data for other objects is possible through +extension modules, but we won't get into that here. + +Using Queues +------------ + +:class:`concurrent.interpreters.Queue` is an implementation of the +:class:`queue.Queue` interface that supports safely and efficiently +passing data between interpreters:: + + import time + from concurrent import interpreters + + def push(queue, value, *extra, delay=0.1): + queue.put(value) + for val in extra: + if delay > 0: + time.sleep(delay) + queue.put(val) + + def pop(queue, count=1, timeout=-1): + return tuple(queue.get(timeout=timeout) + for _ in range(count)) + + interp1 = interpreters.create() + interp2 = interpreters.create() + queue = interpreters.create_queue() + + t = interp1.call_in_thread(push, queue, + 'spam!', 42, 'eggs') + res = interp2.call(pop, queue) + # res: ('spam!', 42, 'eggs') + t.join() + .. _interp-script-args: -Passing Values to an Interpreter +Initializing Values for a Script -------------------------------- When you call a function in Python, sometimes that function requires @@ -442,16 +835,18 @@ script you want to run in another interpreter requires some values. Providing such values to the interpreter, for the script to use, is the simplest kind of communication between interpreters. -Perhaps the simplest approach: you can use an f-string or format string -for the script and interpolate the required values:: +There's a method that supports this: :meth:`Interpreter.prepare_main`. +It binds values to names in the interpreter's ``__main__`` module, +which makes them available to any scripts that run in the interpreter +after that:: - import interpreters + from concurrent import interpreters interp = interpreters.create() def run_interp(interp, name, value): + interp.prepare_main(**{name: value}) interp.exec(f""" - {name} = {value!r} ... """) run_interp(interp, 'spam', 42) @@ -462,29 +857,43 @@ for the script and interpolate the required values:: pass else: with infile: + interp.prepare_main(fd=infile.fileno()) interp.exec(f""" import os - for line in os.fdopen({infile.fileno())): + for line in os.fdopen(fd): print(line) """)) -This works especially well for intrinsic values. For complex values -you'll usually want to reach for other solutions. +This is particularly useful when you want to use a queue in a script:: + + from textwrap import dedent + from concurrent import interpreters + + interp = interpreters.create() + queue = interpreters.create_queue() -There's one convenient alternative that works for some objects: -:func:`Interpreter.prepare_main`. It binds values to names in the -interpreter's ``__main__`` module, which makes them available to any -scripts that run in the interpreter after that:: + interp.prepare_main(queue=queue) + queue.put('spam!') - import interpreters + interp.exec(dedent(""" + obj = queue.get() + print(msg) + """)) + # prints: spam! + +For basic, intrinsic data it can also make sense to use an f-string or +format string for the script and interpolate the required values:: + + from textwrap import dedent + from concurrent import interpreters interp = interpreters.create() def run_interp(interp, name, value): - interp.prepare_main(**{name: value}) - interp.exec(f""" + interp.exec(dedent(f""" + {name} = {value!r} ... - """) + """)) run_interp(interp, 'spam', 42) try: @@ -493,97 +902,103 @@ scripts that run in the interpreter after that:: pass else: with infile: - interp.prepare_main(fd=infile.fileno()) - interp.exec(f""" + interp.exec(dedent(f""" import os - for line in os.fdopen(fd): + for line in os.fdopen({infile.fileno()}): print(line) """)) -Note that :func:`Interpreter.prepare_main` only works with "shareable" -objects. (See :ref:`interp-shareable-objects`.) In a little while -also see how shareable objects make queues and channels especially -convenient and efficient. - -For anything more complex that isn't shareable, you'll probably need -something more than f-strings or :func:`Interpreter.prepare_main`. - -That brings us the next part: 2-way communication, passing data back -and forth between interpreters. - Pipes, Sockets, etc. -------------------- -We'll start off with an approach to duplex communication between -interpreters that you can implement using OS-provided utilities. -The examples will use :func:`os.pipe`, but the approach applies equally -well to regular files and sockets. +For the sake of contrast, let's take a look at a different approach +to duplex communication between interpreters, which you can implement +using OS-provided utilities. You could do something similar with +subprocesses. The examples will use :func:`os.pipe`, but the approach +applies equally well to regular files and sockets. Keep in mind that pipes, sockets, and files work with :class:`bytes`, not other objects, so this may involve at least some serialization. Aside from that inefficiency, sending data through the OS will also slow things down. -After this we'll look at using queues and channels for simpler and more -efficient 2-way communication. The contrast should be clear then. - First, let's use a pipe to pass a message from one interpreter to another:: - import interpreters + from concurrent import interpreters import os - interp1 = interpreters.create() - interp2 = interpreters.create() + READY = b'\0' - rpipe, spipe = os.pipe() + def send(fd_tokens, fd_data, msg): + # Wait until ready. + token = os.read(fd_tokens, 1) + assert token == READY, token + # Ready! + os.write(fd_data, msg) + + interp = interpreters.create() + + r_tokens, s_tokens = os.pipe() + r_data, s_data = os.pipe() try: - def task(): - interp.exec(""" - import os - msg = os.read({rpipe}, 20) - print(msg) - """) - t = threading.thread(target=task) - t.start() - - # Sending the message: - os.write(spipe, b'spam!') + t = interp.call_in_thread( + send, r_tokens, s_data, 'spam!') + os.write(s_tokens, READY) + msg = os.read(r_data, 20) + # msg: 'spam!' t.join() finally: - os.close(rpipe) - os.close(spipe) + os.close(r_tokens) + os.close(s_tokens) + os.close(r_data) + os.close(s_data) One interesting part of that is how the subthread blocked until -we wrote to the pipe. In addition to delivering the message, the -read end of the pipe acted like a lock. +we sent the "ready" token. In addition to delivering the message, +the read end of the pipes acted like locks. -We can actually make use of that to synchronize -execution between the interpreters:: +We can actually make use of that to synchronize execution between the +interpreters (and use :class:`Queue` the same way):: - import interpreters + from concurrent import interpreters + + STOP = b'\0' + READY = b'\1' + + def task(tokens): + r, s = tokens + while True: + # Do stuff. + ... + + # Synchronize! + token = os.read(r, 1) + os.write(s, token) + if token == STOP: + break + + # Do other stuff. + ... + + steps = [...] interp = interpreters.create() r1, s1 = os.pipe() r2, s2 = os.pipe() try: - def task(): - interp.exec(""" - # Do stuff... - - # Synchronize! - msg = os.read(r1, 1) - os.write(s2, msg) - - # Do other stuff... - """) - t = threading.thread(target=task) - t.start() + t = interp.call_in_thread(task, (r1, s2)) + for step in steps: + # Do the step. + ... - # Synchronize! - os.write(s1, '') + # Synchronize! + os.write(s1, READY) + os.read(r2, 1) + os.write(s1, STOP) os.read(r2, 1) + t.join() finally: os.close(r1) os.close(s1) @@ -592,180 +1007,90 @@ execution between the interpreters:: You can also close the pipe ends and join the thread to synchronize. -Again, using :func:`os.pipe`, etc. to communicate between interpreters -is a little awkward, as well as inefficient. We'll look at some of -the alternatives next. +Using :func:`os.pipe` (or similar) to communicate between interpreters +is a little awkward, as well as inefficient. -Using Queues ------------- - -:class:`interpreters.queues.Queue` is an implementation of the -:class:`queue.Queue` interface that supports safely and efficiently -passing data between interpreters. - -:: - - import interpreters.queues - - queue = interpreters.queues.create() - interp = interpreters.create() - interp.prepare_main(queue=queue) - - queue.put('spam!') - interp.exec(""" - msg = queue.get() - print(msg) - """) +Capturing an interpreter's stdout +--------------------------------- -:: +While interpreters share the same default file descriptors for stdout, +stderr, and stdin, each interpreter still has its own :mod:`sys` module, +with its own :data:`~sys.stdout` and so forth. That means it isn't +obvious how to capture stdout for multiple interpreters. - queue = interpreters.queues.create() - interp = interpreters.create() - interp.prepare_main(queue=queue) +The solution is a bit anticlimactic; you have to capture it for each +interpreter manually. Here's a basic example:: - queue.put('spam!') - interp.exec(""" - obj = queue.get() - print(msg) - """) + from concurrent import interpreters + import contextlib + import io -:: + def task(): + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + # Do stuff in worker! + ... + return stdout.getvalue() - queue1 = interpreters.queues.create() - queue2 = interpreters.queues.create(syncobj=True) interp = interpreters.create() - interp.prepare_main(queue1=queue1, queue2=queue2) - - queue1.put('spam!') - queue1.put('spam!', syncobj=True) - queue2.put('spam!') - queue2.put('spam!', syncobj=False) - interp.exec(""" - msg1 = queue1.get() - msg2 = queue1.get() - msg3 = queue2.get() - msg4 = queue2.get() - print(msg) - """) + output = interp.call(task) + ... +Here's a more elaborate example:: -Using Channels --------------- - -... - -:: - - import interpreters.channels - - rch, sch = interpreters.channels.create() - interp = interpreters.create() - interp.prepare_main(rch=rch) - - sch.send_nowait('spam!') - interp.exec(""" - msg = rch.recv() - print(msg) - """) + from concurrent import interpreters + import contextlib + import errno + import os + import sys + import threading -:: + def run_and_capture(fd, task, args, kwargs): + stdout = open(fd, 'w', closefd=False) + with contextlib.redirect_stdout(stdout): + return task(*args, **kwargs) + + @contextlib.contextmanager + def running_captured(interp, task, *args, **kwargs): + def background(fd, stdout=sys.stdout): + # Pass through captured output to local stdout. + # Normally we would not ignore the encoding. + infile = open(fd, 'r', closefd=False) + while True: + try: + line = infile.readline() + except OSError as exc: + if exc.errno == errno.EBADF: + break + raise # re-raise + stdout.write(line) + + bg = None + r, s = os.pipe() + try: + bg = threading.Thread(target=background, args=(r,)) + bg.start() + t = interp.call_in_thread(run_and_capture, s, args, kwargs) + try: + yield + finally: + t.join() + finally: + os.close(r) + os.close(s) + if bg is not None: + bg.join() + + def task(): + # Do stuff in worker! + ... - rch, sch = interpreters.channels.create() interp = interpreters.create() - interp.prepare_main(rch=rch) + with running_captured(interp, task): + # Do stuff in main! + ... - data = bytearray(100) - - sch.send_buffer_nowait(data) - interp.exec(""" - data = rch.recv() - for i in range(len(data)): - data[i] = i - """) - assert len(data) == 100 - for i in range(len(data)): - assert data[i] == i - -Capturing an interpreter's stdout ---------------------------------- - -:: - - interp = interpreters.create() - stdout = io.StringIO() - with contextlib.redirect_stdout(stdout): - interp.exec(tw.dedent(""" - print('spam!') - """)) - assert(stdout.getvalue() == 'spam!') - - # alternately: - interp.exec(tw.dedent(""" - import contextlib, io - stdout = io.StringIO() - with contextlib.redirect_stdout(stdout): - print('spam!') - captured = stdout.getvalue() - """)) - captured = interp.get_main_attr('captured') - assert(captured == 'spam!') - -:func:`os.pipe()` could be used similarly. - -Capturing Unhandled Exceptions ----------------------------- - -... - -Falling Back to Pickle ----------------------- - -For other objects you can use pickle:: - - import interpreters - import pickle - - SCRIPT = """ - import pickle - data = pickle.loads(_pickled) - setattr(data['name'], data['value']) - """ - data = dict(name='spam', value=42) - - interp = interpreters.create() - interp.prepare_main(_pickled=pickle.dumps(data)) - interp.exec(SCRIPT) - -Passing objects via pickle:: - - interp = interpreters.create() - r, s = os.pipe() - interp.exec(tw.dedent(f""" - import os - import pickle - reader = {r} - """)) - interp.exec(tw.dedent(""" - data = b'' - c = os.read(reader, 1) - while c != b'\x00': - while c != b'\x00': - data += c - c = os.read(reader, 1) - obj = pickle.loads(data) - do_something(obj) - c = os.read(reader, 1) - """)) - for obj in input: - data = pickle.dumps(obj) - os.write(s, data) - os.write(s, b'\x00') - os.write(s, b'\x00') - -Gotchas -------- - -... +Using a :mod:`logger ` can also help. .. _interp-recipes: @@ -773,24 +1098,7 @@ Gotchas Recipes (Practical Examples) ============================ -Concurrency-Related Python Workloads ------------------------------------- - -... - -Comparisons With Other Concurrency Models ------------------------------------------ - -... - -Concurrency ------------ - -(incl. minimal comparisons with multiprocessing, threads, and async) - -... - -Isolation ---------- +Example: ... +------------ ... From aea827888b723d05152fed56c09c06082176a7e9 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 26 Jun 2025 14:34:12 -0600 Subject: [PATCH 04/19] Add a TODO list. --- Doc/howto/multiple-interpreters.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 340565920f1f24..1df5546c1538a4 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -37,6 +37,16 @@ comparisons with Python's various existing concurrency models. For information about changing an extension module to work with multiple interpreters, see :ref:`isolating-extensions-howto`. +.. TODO: + + * tutorial: explain how to use interpreters C-API? + * recipes: add some! + * recipes: add short examples of how to solve specific small problems + * recipes: add a section specifically for workflow-oriented examples + * recipes: add comparisons with other concurrency models + * recipes: add examples focused just on taking advantage of isolation? + * recipes: add examples of using C-API in extensions? embedded? + Introduction ============ From 4bb1e2f6670207bf804a2f7b15a79465eaaafc55 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Mon, 30 Jun 2025 11:02:54 -0600 Subject: [PATCH 05/19] Drop the execcomponents ref. --- Doc/howto/multiple-interpreters.rst | 4 +--- Doc/library/concurrent.interpreters.rst | 2 ++ 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 1df5546c1538a4..f5c89878e1f256 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -25,9 +25,7 @@ comparisons with Python's various existing concurrency models. :mod:`concurrent.interpreters` - :ref:`execcomponents` - more details about how interpreters fit into - Python's execution model + .. XXX Add a ref to the upcoming lang ref runtime components section. .. XXX Add a reference to the upcoming concurrency HOWTO doc. diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index ae2364139f1494..e5b0f78be572e1 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -31,6 +31,8 @@ Actual concurrency is available separately through :class:`~concurrent.futures.InterpreterPoolExecutor` combines threads with interpreters in a familiar interface. + .. XXX Add a ref to the upcoming lang ref runtime components section. + :ref:`multiple-interpreters-howto` how to use multiple interpreters From d1ab4023eb98acf5a06b24105f21361e44ea9a4b Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Mon, 30 Jun 2025 16:56:22 -0600 Subject: [PATCH 06/19] Fix a ref. --- Doc/howto/multiple-interpreters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index f5c89878e1f256..674c54562ce076 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -50,7 +50,7 @@ Introduction ============ You can find a thorough explanation about what interpreters are and -what they're for in the :mod:`concurrent.interpreters ` +what they're for in the :ref:`concurrent.interpreters ` docs. In summary, think of an interpreter as the Python runtime's execution From 1c2d40f31708d07c3a88b45728235f8b80fa64f8 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Mon, 30 Jun 2025 16:59:35 -0600 Subject: [PATCH 07/19] Drop the examples section for now. --- Doc/howto/multiple-interpreters.rst | 18 ++---------------- 1 file changed, 2 insertions(+), 16 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 674c54562ce076..189a3d20e44a8e 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -9,15 +9,11 @@ multiple interpreters in a Python program. We will focus on doing so in Python code, through the stdlib :mod:`concurrent.interpreters` module. -This document has 3 parts: first a brief introduction, then a tutorial, -and then a set of recipes and practical examples. +This document has 2 parts: first a brief introduction and then a tutorial. The tutorial covers the basics of using multiple interpreters (:ref:`interp-tutorial-basics`), as well as how to communicate between interpreters (:ref:`interp-tutorial-communicating`). -The examples section (:ref:`interp-recipes`) is focused on providing -effective solutions for real concurrency use cases. This includes -comparisons with Python's various existing concurrency models. .. currentmodule:: concurrent.interpreters @@ -38,6 +34,7 @@ comparisons with Python's various existing concurrency models. .. TODO: * tutorial: explain how to use interpreters C-API? + * add a top-level recipes/examples section * recipes: add some! * recipes: add short examples of how to solve specific small problems * recipes: add a section specifically for workflow-oriented examples @@ -1099,14 +1096,3 @@ Here's a more elaborate example:: ... Using a :mod:`logger ` can also help. - - -.. _interp-recipes: - -Recipes (Practical Examples) -============================ - -Example: ... ------------- - -... From ade2ce347112187b5b42bb706347764d28efdc37 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 1 Jul 2025 10:55:35 -0600 Subject: [PATCH 08/19] Fix typos. Co-authored-by: neonene <53406459+neonene@users.noreply.github.com> --- Doc/howto/multiple-interpreters.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 189a3d20e44a8e..b314cba6f94dcc 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -61,10 +61,10 @@ completely isolated from the others. isolated from one another since there are few restrictions on memory access within the same process. The Python runtime makes a best effort at isolation but extension modules may easily violate that. - Therefore, do not use multiple interpreters in security-senstive + Therefore, do not use multiple interpreters in security-sensitive situations, where they shouldn't have access to each other's data. -That isolation facilitates a concurrency model based an independent +That isolation facilitates a concurrency model based on independent logical threads of execution, like CSP or the actor model. Each actual thread in Python, even if you're only running in the main @@ -86,7 +86,7 @@ There are some downsides and temporary limitations: discipline about how the isolated components in your program interact * not all PyPI extension modules support multiple interpreters yet * the existing tools for passing data between interpreters safely - is still relatively inefficient and limited + are still relatively inefficient and limited * actually *sharing* data safely is tricky (true for free-threading too) * all necessary modules must be imported separately in each interpreter * relatively slow startup time per interpreter @@ -103,7 +103,7 @@ Tutorial: Basics First of all, keep in mind that using multiple interpreters is like using multiple processes. They are isolated and independent from each other. The main difference is that multiple interpreters live in the -same process, which makes it all more efficient and use fewer +same process, which makes it all more efficient and uses fewer system resources. Each interpreter has its own :mod:`!__main__` module, its own @@ -375,7 +375,7 @@ the same rules as functions:: Mutable State is not Shared --------------------------- -Just be be clear, the underlying data of very few mutable objects is +Just to be clear, the underlying data of very few mutable objects is actually shared between interpreters. The notable exceptions are :class:`Queue` and :class:`memoryview`, which we will explore in a little while. In nearly every case, the raw data is copied in @@ -428,7 +428,7 @@ the builtin :func:`exec`, it doesn't reset the namespace it uses to run the code. In the same way, running code in an interpreter does not reset that -interpreter.. The next time you run code in that interpreter, the +interpreter. The next time you run code in that interpreter, the :mod:`!__main__` module will be in exactly the state in which you left it:: @@ -535,7 +535,7 @@ and there's an unhandled exception. In that case, Python will print the traceback and the process will exit with a failure code. The behavior is very similar when code is run in an interpreter. -The traceback get printed and, rather than a failure code, +The traceback gets printed and, rather than a failure code, an :class:`ExecutionFailed` exception is raised:: from concurrent import interpreters From 40f50e7bbf42d44808610053cce172d50926572f Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 1 Jul 2025 11:14:57 -0600 Subject: [PATCH 09/19] Add a misc section to the tutorial. --- Doc/howto/multiple-interpreters.rst | 49 ++++++++++++++++------------- 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index b314cba6f94dcc..f897a705abe2ce 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -11,9 +11,10 @@ module. This document has 2 parts: first a brief introduction and then a tutorial. -The tutorial covers the basics of using multiple interpreters -(:ref:`interp-tutorial-basics`), as well as how to communicate between -interpreters (:ref:`interp-tutorial-communicating`). +The tutorial covers :ref:`the basics ` of using +multiple interpreters, as well as how :ref:`to communicate +` between interpreters. It finishes with +some :ref:`extra information `. .. currentmodule:: concurrent.interpreters @@ -620,24 +621,6 @@ can explicitly clean it up sooner:: finally: interp.close() -concurrent.futures.InterpreterPoolExecutor ------------------------------------------- - -The :mod:`concurrent.futures` module is a simple, popular concurrency -framework that wraps both threading and multiprocessing. You can also -use it with multiple interpreters, via -:class:`~concurrent.futures.InterpreterPoolExecutor`:: - - from concurrent.futures import InterpreterPoolExecutor - - def script(): - return 'spam!' - - with InterpreterPoolExecutor(max_workers=5) as executor: - fut = executor.submit(script) - res = fut.result() - # res: 'spam!' - .. _interp-tutorial-communicating: @@ -1015,6 +998,30 @@ You can also close the pipe ends and join the thread to synchronize. Using :func:`os.pipe` (or similar) to communicate between interpreters is a little awkward, as well as inefficient. + +.. _interp-tutorial-misc: + +Tutorial: Miscellaneous +======================= + +concurrent.futures.InterpreterPoolExecutor +------------------------------------------ + +The :mod:`concurrent.futures` module is a simple, popular concurrency +framework that wraps both threading and multiprocessing. You can also +use it with multiple interpreters, via +:class:`~concurrent.futures.InterpreterPoolExecutor`:: + + from concurrent.futures import InterpreterPoolExecutor + + def script(): + return 'spam!' + + with InterpreterPoolExecutor(max_workers=5) as executor: + fut = executor.submit(script) + res = fut.result() + # res: 'spam!' + Capturing an interpreter's stdout --------------------------------- From 0c3105b47963c54ab895c56052d07455897c1a5b Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 1 Jul 2025 11:20:11 -0600 Subject: [PATCH 10/19] Clarify about prepare_main(). --- Doc/howto/multiple-interpreters.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index f897a705abe2ce..8a6e5a1743273c 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -814,17 +814,18 @@ passing data between interpreters:: .. _interp-script-args: -Initializing Values for a Script --------------------------------- +Initializing Globals for a Script +--------------------------------- -When you call a function in Python, sometimes that function requires -arguments and sometimes it doesn't. In the same way, sometimes a -script you want to run in another interpreter requires some values. -Providing such values to the interpreter, for the script to use, +When you call a function in Python, sometimes that function depends on +arguments, globals, and non-locals, and sometimes it doesn't. In the +same way, sometimes a script you want to run in another interpreter +depends on some global variables. Setting them ahead of time on the +interpreter's :mod:`!__main__` module, where the script will run, is the simplest kind of communication between interpreters. There's a method that supports this: :meth:`Interpreter.prepare_main`. -It binds values to names in the interpreter's ``__main__`` module, +It binds values to names in the interpreter's :mod:`!__main__` module, which makes them available to any scripts that run in the interpreter after that:: From 6028349b3910cde42f95b9acd77c4952f0650007 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 1 Jul 2025 11:23:08 -0600 Subject: [PATCH 11/19] Fix a pseudo-ref. --- Doc/howto/multiple-interpreters.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 8a6e5a1743273c..b6ece7a8d8e60b 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -146,9 +146,9 @@ Calling a simple function in an interpreter works the same way:: (See :meth:`Interpreter.call`.) -When it runs, the code is executed using the interpreter's ``__main__`` -module, just like a Python process normally does when invoked from -the command-line:: +When it runs, the code is executed using the interpreter's +:mod:`!__main__` module, just like a Python process normally does when +invoked from the command-line:: from concurrent import interpreters From d7a7cf0593de05250a910d36120c040fc60d4113 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 1 Jul 2025 16:57:26 -0600 Subject: [PATCH 12/19] Fix the examples. --- Doc/howto/multiple-interpreters.rst | 525 +++++++++++++++------------- 1 file changed, 290 insertions(+), 235 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index b6ece7a8d8e60b..f88d88a6b1e497 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -137,12 +137,13 @@ Calling a simple function in an interpreter works the same way:: from concurrent import interpreters - interp = interpreters.create() - def script(): print('spam!') - interp.call(script) - # prints: spam! + + if __name__ == '__main__': + interp = interpreters.create() + interp.call(script) + # prints: spam! (See :meth:`Interpreter.call`.) @@ -172,8 +173,8 @@ In fact, a comparison with ``python -c`` is quite direct:: It's also fairly easy to simulate the other forms of the Python CLI:: - from textwrap import dedent from concurrent import interpreters + from textwrap import dedent SCRIPT = """ print('spam!') @@ -196,6 +197,7 @@ It's also fairly easy to simulate the other forms of the Python CLI:: ################## interp = interpreters.create() + interp.exec('import os, sys; sys.path.insert(0, os.getcwd())') interp.exec(dedent(""" import runpy runpy.run_module('script') @@ -211,12 +213,13 @@ You can just as easily call a function in another interpreter:: from concurrent import interpreters - interp = interpreters.create() - def spam(): print('spam!') - interp.call(spam) - # prints: spam! + + if __name__ == '__main__': + interp = interpreters.create() + interp.call(spam) + # prints: spam! In fact, nearly all Python functions and callables are supported, with the notable exception of closures. Support includes arguments @@ -239,13 +242,14 @@ The same is true for Python functions that don't use any globals:: from concurrent import interpreters - interp = interpreters.create() - def spam(): # globals is a builtin. print(globals()['__name__']) - interp.call(spam) - # prints: __main__ + + if __name__ == '__main__': + interp = interpreters.create() + interp.call(spam) + # prints: __main__ There are very few cases where that matters, though. @@ -255,9 +259,10 @@ in that module:: from concurrent import interpreters from mymod import spam - interp = interpreters.create() - interp.call(spam) - # prints: mymod + if __name__ == '__main__': + interp = interpreters.create() + interp.call(spam) + # prints: mymod ########## # mymod.py @@ -272,12 +277,13 @@ the :mod:`!__main__` module:: from concurrent import interpreters - interp = interpreters.create() - def spam(): print(__name__) - interp.call(spam) - # prints: '' + + if __name__ == '__main__': + interp = interpreters.create() + interp.call(spam) + # prints: '' This means global state used in such a function won't be reflected in the interpreter's :mod:`!__main__` module:: @@ -285,43 +291,49 @@ in the interpreter's :mod:`!__main__` module:: from textwrap import dedent from concurrent import interpreters - interp = interpreters.create() - total = 0 def inc(): global total total += 1 + def show_total(): print(total) - interp.call(show_total) - # prints: 0 - interp.call(inc) - interp.call(show_total) - # prints: 1 - interp.exec(dedent("""' - try: - print(total) - except NameError: - pass - else: - raise AssertionError('expected NameError') - """)) + if __name__ == '__main__': + interp = interpreters.create() + + interp.call(show_total) + # prints: 0 + + interp.call(inc) + interp.call(show_total) + # prints: 1 + + interp.exec(dedent(""" + try: + print(total) + except NameError: + pass + else: + raise AssertionError('expected NameError') + """)) + interp.exec('total = -1') + interp.exec('print(total)') + # prints: -1 + + interp.call(show_total) + # prints: 1 + + interp.call(inc) + interp.call(show_total) + # prints: 2 - interp.exec('total = -1') - interp.exec('print(total)') - # prints: -1 - interp.call(show_total) - # prints: 1 - interp.call(inc) - interp.call(show_total) - # prints: 2 - interp.exec('print(total)') - # prints: -1 + interp.exec('print(total)') + # prints: -1 - print(total) - # prints: 0 + print(total) + # prints: 0 Calling Methods and Other Objects in an Interpreter @@ -333,16 +345,6 @@ the same rules as functions:: from concurrent import interpreters from mymod import Spam - interp = interpreters.create() - - spam = Spam() - interp.call(Spam.modname) - # prints: mymod - interp.call(spam) - # prints: mymod - interp.call(spam.spam) - # prints: mymod - class Eggs: @classmethod def modname(cls): @@ -352,13 +354,28 @@ the same rules as functions:: def eggs(self): print(__name__) - eggs = Eggs() - res = interp.call(Eggs.modname) - # prints: - res = interp.call(eggs) - # prints: - res = interp.call(eggs.eggs) - # prints: + if __name__ == '__main__': + interp = interpreters.create() + + spam = Spam() + interp.call(Spam.modname) + # prints: mymod + + interp.call(spam) + # prints: mymod + + interp.call(spam.spam) + # prints: mymod + + eggs = Eggs() + res = interp.call(Eggs.modname) + # prints: + + res = interp.call(eggs) + # prints: + + res = interp.call(eggs.eggs) + # prints: ########## # mymod.py @@ -384,8 +401,6 @@ the other interpreter and never automatically synchronized:: from concurrent import interpreters - interp = interpreters.create() - class Counter: def __init__(self, initial=0): self.value = initial @@ -395,30 +410,33 @@ the other interpreter and never automatically synchronized:: self.value -= 1 def show(self): print(self.value) - counter = Counter(17) - - interp.call(counter.show) - # prints: 17 - counter.show() - # prints: 17 - - interp.call(counter.inc) - interp.call(counter.show) - # prints: 18 - counter.show() - # prints: 17 - - interp.call(counter.inc) - interp.call(counter.show) - # prints: 18 - counter.show() - # prints: 17 - - interp.call(counter.inc) - interp.call(counter.show) - # prints: 18 - counter.show() - # prints: 17 + + if __name__ == '__main__': + counter = Counter(17) + interp = interpreters.create() + + interp.call(counter.show) + # prints: 17 + counter.show() + # prints: 17 + + interp.call(counter.inc) + interp.call(counter.show) + # prints: 18 + counter.show() + # prints: 17 + + interp.call(counter.inc) + interp.call(counter.show) + # prints: 18 + counter.show() + # prints: 17 + + interp.call(counter.inc) + interp.call(counter.show) + # prints: 18 + counter.show() + # prints: 17 Preparing and Reusing an Interpreter ------------------------------------ @@ -436,16 +454,31 @@ left it:: from concurrent import interpreters interp = interpreters.create() - interp.exec("""if True + + interp.exec("""if True: answer = 42 """) - interp.exec("""if True + interp.exec("""if True: assert answer == 42 """) - def script(): - assert answer == 42 - interp.call(script) +Similarly:: + + from concurrent import interpreters + + def set(value): + assert __name__ == '', __name__ + global answer + answer = value + + def check(expected): + assert answer == expected + + if __name__ == '__main__': + interp = interpreters.create() + interp.call(set, 100) + interp.call(check, 100) + You can take advantage of this to prepare an interpreter ahead of time or use it incrementally:: @@ -453,30 +486,31 @@ or use it incrementally:: from textwrap import dedent from concurrent import interpreters - interp = interpreters.create() + if __name__ == '__main__': + interp = interpreters.create() - # Prepare the interpreter. - interp.exec(dedent(""" - # We will need this later. - import math - - # Initialize the value. - value = 1 + # Prepare the interpreter. + interp.exec(dedent(""" + # We will need this later. + import math - def double(val): - return val + val - """)) + # Initialize the value. + value = 1 - # Do the work. - for _ in range(9): - interp.exec(dedent(""" - assert math.factorial(value + 1) >= double(value) - value = double(value) + def double(val): + return val + val """)) - # Show the result. - interp.exec('print(value)') - # prints: 1024 + # Do the work. + for _ in range(9): + interp.exec(dedent(""" + assert math.factorial(value + 1) >= double(value) + value = double(value) + """)) + + # Show the result. + interp.exec('print(value)') + # prints: 1024 In case you're curious, in a little while we'll look at how to pass data in and out of an interpreter (instead of just printing things). @@ -521,12 +555,13 @@ There's also a helper method for that:: from concurrent import interpreters - interp = interpreters.create() - def script(): print('spam!') - t = interp.call_in_thread(script) - t.join() + + if __name__ == '__main__': + interp = interpreters.create() + t = interp.call_in_thread(script) + t.join() Handling Uncaught Exceptions ---------------------------- @@ -588,22 +623,26 @@ exceptions that can be pickled:: interp = interpreters.create() try: - interp.exec(dedent(""" - try: - 1/0 - except Exception as exc: + try: + interp.exec(dedent(""" + try: + 1/0 + except Exception as exc: + import pickle + data = pickle.dumps(exc) + class PickledException(Exception): + pass + raise PickledException(data) + """)) + except interpreters.ExecutionFailed as exc: + if exc.excinfo.type.__name__ == 'PickledException': import pickle - data = pickle.dumps(exc) - class PickledException(Exception): - pass - raise PickledException(data) - """)) - except interpreters.ExecutionFailed as exc: - if exc.excinfo.type.__name__ == 'PickledException': - import pickle - raise pickle.loads(exc.excinfo.msg) - else: - raise # re-raise + raise pickle.loads(eval(exc.excinfo.msg)) + else: + raise # re-raise + except ZeroDivisionError: + # Handle it! + ... Managing Interpreter Lifetime ----------------------------- @@ -645,47 +684,56 @@ Arguments provide a way to send information to an interpreter:: from concurrent import interpreters - interp = interpreters.create() - def show(arg): print(arg) - interp.call(show, 'spam!') - # prints: spam! def ident_full(a, /, b, c=42, *args, d, e='eggs', **kwargs): print([a, b, c, d, e], args, kwargs) - interp.call(ident_full, 1, 2, 3, 4, 5, d=6, e=7, f=8, g=9) - # prints: [1, 2, 3, 6, 7] (4, 5) {'f': 8, 'g': 9} def handle_request(req): # do the work ... - req = ... - interp.call(handle_request, req) + + if __name__ == '__main__': + interp = interpreters.create() + + interp.call(show, 'spam!') + # prints: spam! + + interp.call(ident_full, 1, 2, 3, 4, 5, d=6, e=7, f=8, g=9) + # prints: [1, 2, 3, 6, 7] (4, 5) {'f': 8, 'g': 9} + + req = ... + interp.call(handle_request, req) Return values are a way an interpreter can send information back:: from concurrent import interpreters - interp = interpreters.create() - data = {} def put(key, value): data[key] = value + def get(key, default=None): return data.get(key, default) - res = interp.call(get, 'spam') - # res: None - res = interp.call(get, 'spam', -1) - # res: -1 - interp.call(put, 'spam', True) - res = interp.call(get, 'spam') - # res: True - interp.call(put, 'spam', 42) - res = interp.call(get, 'spam') - # res: 42 + if __name__ == '__main__': + interp = interpreters.create() + + res = interp.call(get, 'spam') + # res: None + + res = interp.call(get, 'spam', -1) + # res: -1 + + interp.call(put, 'spam', True) + res = interp.call(get, 'spam') + # res: True + + interp.call(put, 'spam', 42) + res = interp.call(get, 'spam') + # res: 42 Don't forget that the underlying data of few objects is actually shared between interpreters. That means that, nearly always, arguments are @@ -697,34 +745,35 @@ For example:: from concurrent import interpreters - interp = interpreters.create() - data = { 'a': 1, 'b': 2, 'c': 3, } - interp.call(data.clear) - assert data == dict(a=1, b=2, c=3) + if __name__ == '__main__': + interp = interpreters.create() + + interp.call(data.clear) + assert data == dict(a=1, b=2, c=3) - def update_and_copy(data, **updates): - data.update(updates) - return dict(data) + def update_and_copy(data, **updates): + data.update(updates) + return dict(data) - res = interp.call(update_and_copy, data) - assert res == dict(a=1, b=2, c=3) - assert res is not data - assert data == dict(a=1, b=2, c=3) + res = interp.call(update_and_copy, data) + assert res == dict(a=1, b=2, c=3) + assert res is not data + assert data == dict(a=1, b=2, c=3) - res = interp.call(update_and_copy, data, d=4, e=5) - assert res == dict(a=1, b=2, c=3, d=4, e=5) - assert data == dict(a=1, b=2, c=3) + res = interp.call(update_and_copy, data, d=4, e=5) + assert res == dict(a=1, b=2, c=3, d=4, e=5) + assert data == dict(a=1, b=2, c=3) - res = interp.call(update_and_copy, data) - assert res == dict(a=1, b=2, c=3) - assert res is not data - assert data == dict(a=1, b=2, c=3) + res = interp.call(update_and_copy, data) + assert res == dict(a=1, b=2, c=3) + assert res is not data + assert data == dict(a=1, b=2, c=3) Supported and Unsupported Objects --------------------------------- @@ -758,13 +807,14 @@ unpickle in the original interpreter, due to the fake module name:: def __init__(self, x): self.x = x - interp = interpreters.create() - try: - spam = interp.call(Spam, 10) - except interpreters.NotShareableError: - pass - else: - raise AssertionError('unexpected success') + if __name__ == '__main__': + interp = interpreters.create() + try: + spam = interp.call(Spam, 10) + except interpreters.NotShareableError: + pass + else: + raise AssertionError('unexpected success') Sharing Data ------------ @@ -798,19 +848,20 @@ passing data between interpreters:: time.sleep(delay) queue.put(val) - def pop(queue, count=1, timeout=-1): + def pop(queue, count=1, timeout=None): return tuple(queue.get(timeout=timeout) for _ in range(count)) - interp1 = interpreters.create() - interp2 = interpreters.create() - queue = interpreters.create_queue() + if __name__ == '__main__': + interp1 = interpreters.create() + interp2 = interpreters.create() + queue = interpreters.create_queue() - t = interp1.call_in_thread(push, queue, - 'spam!', 42, 'eggs') - res = interp2.call(pop, queue) - # res: ('spam!', 42, 'eggs') - t.join() + t = interp1.call_in_thread(push, queue, + 'spam!', 42, 'eggs') + res = interp2.call(pop, queue) + # res: ('spam!', 42, 'eggs') + t.join() .. _interp-script-args: @@ -830,14 +881,15 @@ which makes them available to any scripts that run in the interpreter after that:: from concurrent import interpreters + from textwrap import dedent interp = interpreters.create() def run_interp(interp, name, value): interp.prepare_main(**{name: value}) - interp.exec(f""" + interp.exec(dedent(f""" ... - """) + """)) run_interp(interp, 'spam', 42) try: @@ -847,7 +899,7 @@ after that:: else: with infile: interp.prepare_main(fd=infile.fileno()) - interp.exec(f""" + interp.exec(dedent(f""" import os for line in os.fdopen(fd): print(line) @@ -865,7 +917,7 @@ This is particularly useful when you want to use a queue in a script:: queue.put('spam!') interp.exec(dedent(""" - obj = queue.get() + msg = queue.get() print(msg) """)) # prints: spam! @@ -926,22 +978,22 @@ to another:: # Ready! os.write(fd_data, msg) - interp = interpreters.create() - - r_tokens, s_tokens = os.pipe() - r_data, s_data = os.pipe() - try: - t = interp.call_in_thread( - send, r_tokens, s_data, 'spam!') - os.write(s_tokens, READY) - msg = os.read(r_data, 20) - # msg: 'spam!' - t.join() - finally: - os.close(r_tokens) - os.close(s_tokens) - os.close(r_data) - os.close(s_data) + if __name__ == '__main__': + interp = interpreters.create() + r_tokens, s_tokens = os.pipe() + r_data, s_data = os.pipe() + try: + t = interp.call_in_thread( + send, r_tokens, s_data, b'spam!') + os.write(s_tokens, READY) + msg = os.read(r_data, 20) + # msg: b'spam!' + t.join() + finally: + os.close(r_tokens) + os.close(s_tokens) + os.close(r_data) + os.close(s_data) One interesting part of that is how the subthread blocked until we sent the "ready" token. In addition to delivering the message, @@ -951,6 +1003,7 @@ We can actually make use of that to synchronize execution between the interpreters (and use :class:`Queue` the same way):: from concurrent import interpreters + import os STOP = b'\0' READY = b'\1' @@ -972,27 +1025,27 @@ interpreters (and use :class:`Queue` the same way):: steps = [...] - interp = interpreters.create() - - r1, s1 = os.pipe() - r2, s2 = os.pipe() - try: - t = interp.call_in_thread(task, (r1, s2)) - for step in steps: - # Do the step. - ... - - # Synchronize! - os.write(s1, READY) + if __name__ == '__main__': + interp = interpreters.create() + r1, s1 = os.pipe() + r2, s2 = os.pipe() + try: + t = interp.call_in_thread(task, (r1, s2)) + for step in steps: + # Do the step. + ... + + # Synchronize! + os.write(s1, READY) + os.read(r2, 1) + os.write(s1, STOP) os.read(r2, 1) - os.write(s1, STOP) - os.read(r2, 1) - t.join() - finally: - os.close(r1) - os.close(s1) - os.close(r2) - os.close(s2) + t.join() + finally: + os.close(r1) + os.close(s1) + os.close(r2) + os.close(s2) You can also close the pipe ends and join the thread to synchronize. @@ -1045,9 +1098,10 @@ interpreter manually. Here's a basic example:: ... return stdout.getvalue() - interp = interpreters.create() - output = interp.call(task) - ... + if __name__ == '__main__': + interp = interpreters.create() + output = interp.call(task) + ... Here's a more elaborate example:: @@ -1083,7 +1137,7 @@ Here's a more elaborate example:: try: bg = threading.Thread(target=background, args=(r,)) bg.start() - t = interp.call_in_thread(run_and_capture, s, args, kwargs) + t = interp.call_in_thread(run_and_capture, s, task, args, kwargs) try: yield finally: @@ -1098,9 +1152,10 @@ Here's a more elaborate example:: # Do stuff in worker! ... - interp = interpreters.create() - with running_captured(interp, task): - # Do stuff in main! - ... + if __name__ == '__main__': + interp = interpreters.create() + with running_captured(interp, task): + # Do stuff in main! + ... Using a :mod:`logger ` can also help. From fb13944a6eb1256a0935191640068fe626d642c1 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 1 Jul 2025 17:35:18 -0600 Subject: [PATCH 13/19] Tweak one example. --- Doc/howto/multiple-interpreters.rst | 39 ++++++++++++++--------------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index f88d88a6b1e497..e3e3f845b92dc7 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -486,31 +486,30 @@ or use it incrementally:: from textwrap import dedent from concurrent import interpreters - if __name__ == '__main__': - interp = interpreters.create() + interp = interpreters.create() - # Prepare the interpreter. - interp.exec(dedent(""" - # We will need this later. - import math + # Prepare the interpreter. + interp.exec(dedent(""" + # We will need this later. + import math - # Initialize the value. - value = 1 + # Initialize the value. + value = 1 - def double(val): - return val + val - """)) + def double(val): + return val + val + """)) - # Do the work. - for _ in range(9): - interp.exec(dedent(""" - assert math.factorial(value + 1) >= double(value) - value = double(value) - """)) + # Do the work. + for _ in range(9): + interp.exec(dedent(""" + assert math.factorial(value + 1) >= double(value) + value = double(value) + """)) - # Show the result. - interp.exec('print(value)') - # prints: 1024 + # Show the result. + interp.exec('print(value)') + # prints: 1024 In case you're curious, in a little while we'll look at how to pass data in and out of an interpreter (instead of just printing things). From 72e46fb356d0d8d48b1f435d60fb2de0a2cab347 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 2 Jul 2025 13:04:19 -0600 Subject: [PATCH 14/19] Add a ref. --- Doc/howto/multiple-interpreters.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index e3e3f845b92dc7..afdea9a812aea7 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -223,7 +223,8 @@ You can just as easily call a function in another interpreter:: In fact, nearly all Python functions and callables are supported, with the notable exception of closures. Support includes arguments -and return values, which we'll explore soon. +and return values, which we'll +:ref:`explore soon `. Builtin functions always execute in the target interpreter's :mod:`!__main__` module:: @@ -673,6 +674,8 @@ between interpreters. In this half of the tutorial, we explore various ways you can do so. +.. _interp-tutorial-return-values: + Call Args and Return Values --------------------------- From 675246d2e75f3762fba50c6695907eaef368f536 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 2 Jul 2025 18:22:19 -0600 Subject: [PATCH 15/19] Clarify about Interpreter.exec(). --- Doc/howto/multiple-interpreters.rst | 65 ++++++++++++++++++++++------- 1 file changed, 49 insertions(+), 16 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index afdea9a812aea7..fcd4d1b684ef45 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -133,23 +133,12 @@ builtin :func:`exec` using that interpreter:: (See :meth:`Interpreter.exec`.) -Calling a simple function in an interpreter works the same way:: +Calls are also supported. +See :ref:`the next section `. - from concurrent import interpreters - - def script(): - print('spam!') - - if __name__ == '__main__': - interp = interpreters.create() - interp.call(script) - # prints: spam! - -(See :meth:`Interpreter.call`.) - -When it runs, the code is executed using the interpreter's -:mod:`!__main__` module, just like a Python process normally does when -invoked from the command-line:: +When :meth:`Interpreter.exec` runs, the code is executed using the +interpreter's :mod:`!__main__` module, just like a Python process +normally does when invoked from the command-line:: from concurrent import interpreters @@ -206,6 +195,50 @@ It's also fairly easy to simulate the other forms of the Python CLI:: That's more or less what the ``python`` executable is doing for each of those cases. +You can also exec any function that doesn't take any arguments, nor +returns anything. Closures are not allowed but other nested functions +are. It works the same as if you had passed the script corresponding +to the function's body:: + + from concurrent import interpreters + + def script(): + print('spam!') + + def get_script(): + def nested(): + print('eggs!') + return nested + + if __name__ == '__main__': + interp = interpreters.create() + + interp.exec(script) + # prints: spam! + + script2 = get_script() + interp.exec(script2) + # prints: eggs! + +Any referenced globals are resolved relative to the interpreter's +:mod:`!__main__` module, just like happens for scripts, rather than +the original function's module:: + + from concurrent import interpreters + + def script(): + print(__name__) + + if __name__ == '__main__': + interp = interpreters.create() + interp.exec(script) + # prints: __main__ + +One practical difference is that with a script function you get syntax +highlighting in your editor. With script text you probably don't. + +.. _interp-tutorial-calls: + Calling a Function in an Interpreter ------------------------------------ From 22e3ee0c4605a7068d60f422cb0e2bc6ce2be7cc Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 2 Jul 2025 19:37:33 -0600 Subject: [PATCH 16/19] Clarify about calling different kinds of function. --- Doc/howto/multiple-interpreters.rst | 100 +++++++++++++++++++++------- 1 file changed, 76 insertions(+), 24 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index fcd4d1b684ef45..bd5eb762d88d5f 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -242,7 +242,7 @@ highlighting in your editor. With script text you probably don't. Calling a Function in an Interpreter ------------------------------------ -You can just as easily call a function in another interpreter:: +You can just as easily *call* a function in another interpreter:: from concurrent import interpreters @@ -254,13 +254,30 @@ You can just as easily call a function in another interpreter:: interp.call(spam) # prints: spam! -In fact, nearly all Python functions and callables are supported, -with the notable exception of closures. Support includes arguments -and return values, which we'll -:ref:`explore soon `. +(See :meth:`Interpreter.call`.) -Builtin functions always execute in the target interpreter's -:mod:`!__main__` module:: +In fact, nearly all Python functions and callables are supported, with +the notable exclusion of closures. We'll focus on plain functions +for the moment. + +Relative to :meth:`Interpreter.call`, there are five kinds of functions: + +1. builtin functions +2. extension module functions +3. "stateless" user functions +4. stateful user functions, defined in :mod:`!__main__` +5. stateful user functions, not defined in :mod:`!__main__` + +A "stateless" function is one that doesn't use any globals. It also +can't be a closure. It can have parameters but not argument defaults. +It can have return values. We'll cover +:ref:`arguments and returning ` soon. + +Some builtin functions, like :func:`exec` and :func:`eval`, use the +current globals as a default (or :func:`implicit `) argument +value. When such functions are called directly with +:meth:`Interpreter.call`, they use the :mod:`!__main__` module's +:data:`!__dict__` as the default/implicit "globals":: from concurrent import interpreters @@ -272,23 +289,29 @@ Builtin functions always execute in the target interpreter's interp.call(eval, 'print(__name__)') # prints: __main__ -The same is true for Python functions that don't use any globals:: +Note that, for some of those builtin functions, like :func:`globals` +and even :func:`eval` (for some expressions), the return value may +be :ref:`"unshareable" ` and the call will +fail. - from concurrent import interpreters +In each of the above cases, the function is more-or-less copied, when +sent to the other interpreter. For most of the cases, the function's +module is imported in the target interpreter and the function is pulled +from there. The exceptions are cases (3) and (4). - def spam(): - # globals is a builtin. - print(globals()['__name__']) - - if __name__ == '__main__': - interp = interpreters.create() - interp.call(spam) - # prints: __main__ +In case (3), the function is also copied, but efficiently and only +temporarily (long enough to be called) and is not actually bound to +any module. This temporary function object's :data:`!__name__` will +be set to match the code object, but the :data:`!__module__`, +:data:`!__qualname__`, and other attributes are not guaranteed to +be set. This shouldn't be a problem in practice, though, since +introspecting the currently running function is fairly rare. -There are very few cases where that matters, though. +We'll cover case (4) in +:ref:`the next section `. -Otherwise, functions defined in modules other than :mod:`!__main__` run -in that module:: +In all the cases, the function will get any global variables from the +module (in the target interpreter), like normal:: from concurrent import interpreters from mymod import spam @@ -305,9 +328,25 @@ in that module:: def spam(): print(__name__) -For a function actually defined in the :mod:`!__main__` module, -it executes in a separate dummy module, in order to not pollute -the :mod:`!__main__` module:: +Note, however, that in case (3) functions rarely look up any globals. + +.. _interp-tutorial-funcs-in-main: + +Functions Defined in __main__ +----------------------------- + +Functions defined in :mod:`!__main__` are treated specially by +:meth:`Interpreter.call`. This is because the script or module that +ran in the main interpreter will have not run in other interpreters, +so any functions defined in the script would only be accessible by +running it in the target interpreter first. That is essentially +how :meth:`Interpreter.call` handles it. + +If the function isn't found in the target interpreter's :mod:`!__main__` +module then the script that ran in the main interpreter gets run in the +target interpreter, though under a different name than "__main__". +The function is then looked up on that resulting fake __main__ module. +For example:: from concurrent import interpreters @@ -319,7 +358,12 @@ the :mod:`!__main__` module:: interp.call(spam) # prints: '' -This means global state used in such a function won't be reflected +The dummy module is never added to :data:`sys.modules`, though it may +be cached away internally. + +This approach with a fake :mod:`!__main__` module means we can avoid +polluting the :mod:`!__main__` module of the target interpreter. It +also means global state used in such a function won't be reflected in the interpreter's :mod:`!__main__` module:: from textwrap import dedent @@ -369,6 +413,12 @@ in the interpreter's :mod:`!__main__` module:: print(total) # prints: 0 +Note that the recommended ``if __name__ == '__main__`:`` idiom is +especially important for such functions, since the script will be +executed with :data:`!__name__` set to something other than +:mod:`!__main__`. Thus, for any code you don't want run repeatedly, +put it in the if block. You will probably only want functions or +classes outside the if block. Calling Methods and Other Objects in an Interpreter --------------------------------------------------- @@ -810,6 +860,8 @@ For example:: assert res is not data assert data == dict(a=1, b=2, c=3) +.. _interp-tutorial-shareable: + Supported and Unsupported Objects --------------------------------- From f7661da73ba3f03deb5ead26a5efded13d9c248b Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 2 Jul 2025 19:40:19 -0600 Subject: [PATCH 17/19] Clarify a note. --- Doc/howto/multiple-interpreters.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index bd5eb762d88d5f..93f5aeb5c6b6ca 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -480,8 +480,8 @@ Mutable State is not Shared Just to be clear, the underlying data of very few mutable objects is actually shared between interpreters. The notable exceptions are :class:`Queue` and :class:`memoryview`, which we will explore in a -little while. In nearly every case, the raw data is copied in -the other interpreter and never automatically synchronized:: +little while. In nearly every case, the raw data is only copied in +the other interpreter and thus never automatically synchronized:: from concurrent import interpreters From 28f0218c39f46bd8563532523a0c1639fc10e7fd Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 2 Jul 2025 19:45:29 -0600 Subject: [PATCH 18/19] Add a caveat about -c and the REPL. --- Doc/howto/multiple-interpreters.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 93f5aeb5c6b6ca..384a01c9ecc33d 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -420,6 +420,12 @@ executed with :data:`!__name__` set to something other than put it in the if block. You will probably only want functions or classes outside the if block. +Another thing to keep in mind is that, when you run the REPL or run +``python -c ...``, the code that runs is essentially unrecoverable. The +contents of the :mod:`__main__` module cannot be reproduced by executing +a script (or module) file. Consequently, calling a function defined +in :mod:`__main__` in these cases will probably fail. + Calling Methods and Other Objects in an Interpreter --------------------------------------------------- From 1b79755161b322869dfd5a9a979b09bd00a75241 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 2 Jul 2025 19:55:29 -0600 Subject: [PATCH 19/19] Add a pro tip. --- Doc/howto/multiple-interpreters.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Doc/howto/multiple-interpreters.rst b/Doc/howto/multiple-interpreters.rst index 384a01c9ecc33d..c20c5339a00414 100644 --- a/Doc/howto/multiple-interpreters.rst +++ b/Doc/howto/multiple-interpreters.rst @@ -426,6 +426,11 @@ contents of the :mod:`__main__` module cannot be reproduced by executing a script (or module) file. Consequently, calling a function defined in :mod:`__main__` in these cases will probably fail. +Here's one last tip before we move on. You can avoid the extra +complexity involved with functions (and classes) defined in +:mod:`__main__` by simply not defining them in :mod:`__main__`. +Instead, put them in another module and import them in your script. + Calling Methods and Other Objects in an Interpreter ---------------------------------------------------