Delegate reactor implementation to user-provided Python classes

[speth]

Changes proposed in this pull request

• Introduce the Delegator class which can be used to delegate methods of a Cantera object to user-provided methods
• Use this capability to introduce the ExtensibleReactor class in Python, which allows users to write a Python class that augments or replaces behavior of the Reactor class. Similar lightweight classes are added for the other reactor types, namely ExtensibleIdealGasReactor, ExtensibleConstPressureReactor, and ExtensibleIdealGasConstPressureReactor.
• Modify function signatures for Reactor evaluation to meet a couple of needs
  • Rename evalEqs to eval and remove the y and params arrays that shouldn't be used within eval for simplicity and to prevent some potential for errors
  • Modify the evalSurfaces method to eliminate the return value and to add the bulk phase species production rates as an additional output
  • Modify both eval and evalSurfaces to split the ydot argument into "left hand" and "right hand" sides of the ODE, to make it easier for users to augment the existing implementations
• Fix "declared-species" option to work for surfaces (this is unrelated, but was a bug that came up during testing)
• Make attached surfaces available via Python Reactor object
• Make demangle available outside AnyMap

Remaining To-do items:

• Eliminate lambda used to determine array sizes
• Use map of std::function<...>* to remove need for specializing setDelegate in derived classes
• Rename the DelegatedReactor class to ExtensibleReactor
• Document requirements for adding a new delegated method (both for cases where the function signature is already handled, and for cases where it isn't)
• Adjust naming mnemonic to be consistent between Delegator and Cython callback functions
• Add more tests

An example of a user-provided class is given in the new example custom2.py:

cantera/interfaces/cython/cantera/examples/reactors/custom2.py

Lines 23 to 64 in eed11d7

	class InertialWallReactor(ct.ExtensibleIdealGasReactor):
	def __init__(self, *args, neighbor, **kwargs):
	super().__init__(*args, **kwargs)
	self.v_wall = 0 # initial wall velocity
	self.k_wall = 1e-2 # proportionality constant, a_wall = k_wall * delta P
	self.neighbor = neighbor
	def after_initialize(self, t0):
	# The initialize function for the base Reactor class will have set
	# n_vars to already include the volume, internal energy, mass, and mass
	# fractions of all the species. Increase this by one to account for
	# the added variable of the wall velocity.
	self.n_vars += 1
	# The index for the new variable / equation, which is at the end of the
	# state vector
	self.i_wall = self.n_vars - 1
	def after_get_state(self, y):
	# This method is used to set the initial condition used by the ODE solver
	y[self.i_wall] = self.v_wall
	def after_update_state(self, y):
	# This method is used to set the state of the Reactor and Wall objects
	# based on the new values for the state vector provided by the ODE solver
	self.v_wall = y[self.i_wall]
	self.walls[0].set_velocity(self.v_wall)
	def after_eval(self, t, LHS, RHS):
	# Calculate the time derivative for the additional equation
	a = self.k_wall * (self.thermo.P - self.neighbor.thermo.P)
	RHS[self.i_wall] = a
	def before_component_index(self, name):
	# Other components are handled by the method from the base Reactor class
	if name == 'v_wall':
	return self.i_wall
	def before_component_name(self, i):
	# Other components are handled by the method from the base Reactor class
	if i == self.i_wall:
	return 'v_wall'

If applicable, fill in the issue number this pull request is fixing

This is the first part of implementing the capabilities described in Cantera/enhancements#79. The intention is to exercise this capability first for the Reactor class before expanding its use to other areas like reactions and thermo models.

Checklist

• There is a clear use-case for this code change
• The commit message has a short title & references relevant issues
• Build passes (scons build & scons test) and unit tests address code coverage
• The pull request is ready for review

[codecov]

Codecov Report

Merging #1003 (3d813ed) into main (da12213) will increase coverage by 0.19%.
The diff coverage is 86.20%.

@@            Coverage Diff             @@
##             main    #1003      +/-   ##
==========================================
+ Coverage   66.10%   66.30%   +0.19%     
==========================================
  Files         313      315       +2     
  Lines       44955    45283     +328     
  Branches    19137    19237     +100     
==========================================
+ Hits        29719    30023     +304     
+ Misses      12652    12644       -8     
- Partials     2584     2616      +32     

Impacted Files	Coverage Δ
include/cantera/base/AnyMap.h	84.37% <ø> (ø)
include/cantera/base/AnyMap.inl.h	51.85% <ø> (ø)
include/cantera/base/global.h	84.21% <ø> (ø)
include/cantera/kinetics/Reaction.h	100.00% <ø> (ø)
include/cantera/zeroD/ConstPressureReactor.h	33.33% <ø> (-16.67%)	⬇️
include/cantera/zeroD/FlowReactor.h	66.66% <ø> (ø)
...clude/cantera/zeroD/IdealGasConstPressureReactor.h	50.00% <ø> (ø)
include/cantera/zeroD/IdealGasReactor.h	50.00% <ø> (ø)
include/cantera/zeroD/ReactorNet.h	76.66% <ø> (ø)
include/cantera/zeroD/ReactorSurface.h	71.42% <ø> (ø)
... and 29 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da12213...3d813ed. Read the comment docs.

[ischoegl]

While I know this is work-in-progress, I was wondering about names … in some of my recent work in the context of reactions/rates, I used Custom*, whereas here the prefix is Delegated*. Would it make sense to use a more descriptive UserDefined* prefix across the board?

[bryanwweber]

While I know this is work-in-progress, I was wondering about names

IMO, this is exactly the point of draft PRs, to address and resolve bigger questions about the implementation 😊

I agree we should try to be consistent. Delegated feels too technical to me, while Custom doesn't quite feel right either, and UserDefined seems too long. I don't have any better suggestions at the moment though 🤣

[ischoegl]

UserDefined seems too long ...

We recently gained 8 characters per line!! 😄 ... Joking aside, I was suggesting UserDefined as UDF's (user-defined-functions) appear elsewhere and are intuitively named. Of course, other suggestions are welcome.

[speth]

I went through several names while working up this implementation, and I would agree that it's tricky to come up with an ideal name. My rationale for using the term Delegator over something like Custom or UserDefined is that it hopefully provides some clarity that this class itself isn't really doing any work -- that's the responsibility of the derived class.

I guess one alternative would be Proxy, which is shorter, more frequently used in software engineering (a delegate sounds like someone who goes to political conventions), but still maintains the notion of this class as an intermediary.

(and no worries about comments on the draft PR - they're entirely welcome, from my end)

[ischoegl]

Thanks, @speth. I actually don’t have any concerns about the internal *Delegator classes, which are imho aptly named (Proxy is too generic). My comment was more about the user-facing Delegated* classes, which are not intuitively recognizable. The comment about UserDefined* for something that is not doing any work is legitimate, but at the same time it clearly signals the intent, which is what ultimately counts for the API.

[speth]

I'm concerned that a name like UserDefinedReactor would encourage users to try putting their equations into that class directly, rather than seeing it as a class from which they should inherit. I think the terminology "user defined x" is reasonable for the overall capability, and should be used in the documentation for this broader capability, but I'm not sure it's the best fit for the specific class names.

[ischoegl]

I'm concerned that a name like UserDefinedReactor would encourage users [...]

That remains a valid point. What about ReactorTemplate? This should clearly indicate that it's incomplete, where a suitable docstring would instruct that things need to be overloaded. My own concern about Delegated* remains that it is somewhat technical and not intuitive ...

[ischoegl]

@speth ... thanks for your comments. I am responding outside of the thread above as I believe I have some bigger questions about the proposed implementation.

I think the challenge is that a function with an unspecified signature is an incomplete type, so you'd have to define child classes of GenericFunc for each function signature, with a member variable holding a std::function of the specific type.

It would be relatively straight forward to create a generic FunctionFactory that takes care of this? That way, all functions would be well defined, and nothing is tied to a specific Cantera object. That way, the boilerplate of various function declarations (currently duplicated in Delegator and ReactorDelegator::setDelegate) is in one place where it can be easily tested.

Then, you'd have to check for each option in the single setDelegate function and do a dynamic_cast to be able to get at the underlying std::function object in order to be able set the appropriate member variable. So I think the current implementation is preferable.

I don't think that it would be a good idea to thread everything through a single setDelegate function. My main concern of the current implementation is that the association of delegated functions happens by if-else inside of various setDelegate variants in a fashion similar to magic numbers, which themselves are specific to a given Cantera object and thus are not reusable. Let's take this example from ReactorDelegator (I cut some of the details for clarity):

    // For functions with the signature void(double*)
    virtual void setDelegate(
        const std::string& name,
        const std::function<void(std::array<size_t, 1>, double*)>& func,
        const std::string& when) override
    {
        if (name == "getState") {
            // ... set delegator
        } else if (name == "updateState") {
            // ... set delegator
        } else if (name == "updateSurfaceState") {
            // ... set delegator
        } else if (name == "getSurfaceInitialConditions") {
            // ... set delegator
        } else {
            throw NotImplementedError("ReactorDelegator::setDelegate",
                "For function named '{}' with signature"
                " void(array<size_t, 1>, double*)", name);
        }
    }

My main fear here is that this will be hard to maintain long term as what is done is well hidden. An alternative would be to do this explicitly by declaring a dedicated DoublePtrFunc (or any other appropriate name) that is equivalent to std::function<void(std::array<size_t, 1>, double*)> and accomplish the same explicitly as

   delegateGetState(const DoublePtrFunc& func)
   {
        // ... set delegator
   }

   delegateUpdateState(const DoublePtrFunc& func)
   {
        // ... set delegator
   }

   delegateUpdateSurfaceState(const DoublePtrFunc& func)
   {
        // ... set delegator
   }

   delegateGetSurfaceInitialConditions(const DoublePtrFunc& func)
   {
        // ... set delegator
   }

Some of the plumbing would be different (a Delegator may look somewhat different) with various details to be worked out, especially as the conversion from std::function<void(std::array<size_t, 1>, double*)> to DoublePtrFunc would take place earlier (although technically a DoublePtrFunc definition may be avoidable?). However, I honestly believe that Explicit is better than implicit here as well.

I honestly believe that an alternate explicit structure would be easier to maintain and could also be directly applied to other Cantera objects without adding too much to the boilerplate. I also believe that the outside interface (e.g. Python API) can be the exact same as what is already proposed.

PS: I see how your implementation is motivated by delegator.pyx with the catch-all assign_delegates method that handles everything. But passing everything through a single interface does have associated cost in terms of code readability. The nice thing is that edits in _cantera.pxd are avoided, but there may be some solutions that preserve both readability and ease of use.

PPS: As yet another alternative, what about setting Delegator up in a way that ReactorDelegator can link the accessors with something that looks more or less like a factory? I.e. in the constructor, set up the linkage between name, member variable and type of function, i.e.

reg("getState", [](const GenericFunc& func) { m_getState = new DoublePtrFunc(func); });

which then allows to simply call setDelegator("getState", func), where the type of func can be checked as appropriate by a copy constructor for DoublePtrFunc? The machinery for reg would be defined in Delegator and thus would be portable.

[corykinney]

I don't have the C++ experience to contribute any code suggestions, but I'm excited for this feature and the capability for custom functionality in Python.

I did want to comment on some of the naming discussed in the thread though and offer my thoughts. I also think that Delegated sounds unituitive. I would've preferred Custom or UserDefined but I understand the concern with those names.

I'm not sure if I fully understand how the class would work. Is it a simple reactor for which basic behavior is defined with the option to overload or extend certain components, or is it an empty reactor where certain functions must be defined in order for it to be usable? If it's the former, ReactorTemplate sounds descriptive and more intuitive, as it makes me think of a sort of abstract base class with no inherent functionality. If it's the latter, I would suggest ExtensibleReactor since extensible is a somewhat commonly used programming term and I think it would accurately indicate that you're adding functionality as required but the base functionality is already there.

[speth]

It would be relatively straight forward to create a generic FunctionFactory that takes care of this? That way, all functions would be well defined, and nothing is tied to a specific Cantera object. That way, the boilerplate of various function declarations (currently duplicated in Delegator and ReactorDelegator::setDelegate) is in one place where it can be easily tested.

For the factory pattern to be useful, you need to be able to construct different derived-type objects from input that has the same type, e.g. how we construct a ThermoPhase from parameters stored in an AnyMap. But there is no equivalent when the input is just a std::function of unspecified type, and I think any attempt to get around this would result in a lot of ugly dynamic_casting. I appreciate the attempt to find ways of simplifying the implementation of this feature, but I don't think adding a factory here would help.

---

I don't think that it would be a good idea to thread everything through a single setDelegate function. My main concern of the current implementation is that the association of delegated functions happens by if-else inside of various setDelegate variants in a fashion similar to magic numbers, which themselves are specific to a given Cantera object and thus are not reusable.

Switching from the relatively small set of setDelegate function signatures, which will apply across every set of classes where this concept is applied, to unique methods for each delegated method like delegateGetState would introduce a lot of additional boilerplate to the Cython interface - each and every one of these functions would require its own Cython wrapper function. This would increase the complexity of adding additional delegated methods, further slow down compilation of the Cython module, and make the work required to bring this capability to other interfaces (e.g. Matlab or Julia) that much more tedious as well. I think there's a lot to be gained from the fact that right now, the Cython interface doesn't even need to know about the specific derived types of the Delegator class.

I disagree that the use of strings with names that correspond to function names in the C++ interface is in akin to magic numbers. You're going to have some sort of name mapping no matter what when crossing the language interface, and this is no worse than the pattern of using names like delegateUpdateSurfaceState that you've suggested.

---

PPS: As yet another alternative, what about setting Delegator up in a way that ReactorDelegator can link the accessors with something that looks more or less like a factory? I.e. in the constructor, set up the linkage between name, member variable and type of function, i.e.

reg("getState", [](const GenericFunc& func) { m_getState = new DoublePtrFunc(func); });

This is an interesting suggestion, and I think you could use this to make the setDelegate methods of the Delegator class non-virtual and get rid of the if/else if trees the derived classes. However, one detail that I think is maybe not obvious (and which is a source of a good chunk of the complexity here) is that the only place where we can create a wrapper for calling a function from the base (non-delegated) class is from a method in the derived class (e.g. ReactorDelegator<R>), for example:

if (name == "initialize) {
    m_initialize = makeDelegate(func, when,
        [this](double t0) { R::initialize(t0); }
    );
}

With the introduction of the reg function, this would have to become a nested lambda from the constructor, something like:

reg("initialize",
    [this](const std::function<void(double)>& func, const std::string& when) {
        m_initialize = makeDelegate(func, when,
            [this](double t0) { R::initialize(); });
    }
);

which really doesn't win any awards for readability, and would just be worse for the cases where makeDelegate needs an additional lambda argument to provide array sizes. And there's also the implementation of reg, which I think would be quite complex as well, just figuring out what the various types even are (it's a function that takes a function of a function as one of it's arguments... 🤯).

---

I'm not sure if I fully understand how the class would work. Is it a simple reactor for which basic behavior is defined with the option to overload or extend certain components, or is it an empty reactor where certain functions must be defined in order for it to be usable? If it's the former, ReactorTemplate sounds descriptive and more intuitive, as it makes me think of a sort of abstract base class with no inherent functionality. If it's the latter, I would suggest ExtensibleReactor since extensible is a somewhat commonly used programming term and I think it would accurately indicate that you're adding functionality as required but the base functionality is already there.

The latter behavior is what this feature accomplishes. I like the idea of using the prefix Extensible for the final types that are presented to the user, such as ExtensibleIdealGasReactor, while leaving the base class (which end users will not interact with) as ReactorDelegator. Thanks for the suggestion, @corykinney.

[ischoegl]

@speth … in light of #1130 (which I believe builds on this), could you rebase?

[speth]

@ischoegl - I've rebased this so it has the same root as #1130. This PR contains my initial implementation of the core "delegation" feature, while @chinahg's PR introduces an additional change to the way all of the reactor governing equations are written to make it easier to introduce certain modifications to those equations. For review purposes, I think the best thing to do at least initially would be to review my changes in this PR, and limit comments on #1130 to the changes that are unique to that PR.

[ischoegl]

@speth ... while this is technically still in Draft mode, I left a couple of comments. The external interface looks like a good start (there may be a way of making it a little more user-friendly, but it would probably have to build on something similar to what is proposed here).

My main question is whether the 'glue' between generic Python functions and C++ can be written in a more generic fashion. The cascading if-else are hard to read, and the new PyFuncInfo look like they would be terrific to have outside of delegated objects.

[speth]

While I am still partial to having a GenericFunction defined outside of delegated methods, some of my concerns here boil down to relying on if-else cascades to detect what is done where.

I believe much of this would be more readable (and likely more maintainable) if it were implemented as a series of class methods. I realize that functions need to be recognized by name, but that could be handled by a std::map?

For the sake of exploration, I tried modifying this to use individual member functions to set delegates, for functions of one particular signature (void(double*)). You can see the difference here: speth/cantera@11c6093...e4dce0c. I would say that it's not quite as bad as I thought it would be, but it does take almost twice as many lines of code, in twice as many locations to add a new delegated method, and I don't really see much benefit.

Using the setDelegate method to cover all methods with the same signature requires just the branch in the correct else-if tree:

    } else if (name == "updateState") {
        m_updateState = makeDelegate<1>(func,
            [this]() {
                return std::array<size_t, 1>{R::neq()};
            },
            when,
            [this](double* y) {
                R::updateState(y);
            }
        );
    }

and a single line added to the Python DelegatedReactor.delegatable_methods list:

    'update_state': ('updateState', 'void(double*)'),

---

In contrast, with individual delegate setters, the setter in ReactorDelegator<R> itself is really no simpler:

virtual void setUpdateState(const std::function<void(std::array<size_t, 1>, double*)>& func,
                            const std::string& when) override
{
    m_updateState = makeDelegate<1>(func,
        [this]() {
            return std::array<size_t, 1>{R::neq()};
        },
        when,
        [this](double* y) {
            R::updateState(y);
        }
    );
}

and also requires a corresponding declaration in the newly-required ReactorDelegatorBase class:

    virtual void setUpdateState(const std::function<void(std::array<size_t, 1>, double*)>& func,
                                const std::string& when) = 0;

On the Python side, this method needs to be declared to Cython as part of the ReactorDelegatorBase class:

void setUpdateState(function[void(size_array1, double*)], string&) except +translate_exception

and a new if/else tree is needed to check for delegates in DelegatedReactor.__init__:

for when in ['before', 'after', 'replace']:
    cxx_when = stringify(when)
    if hasattr(self, f"{when}_update_state"):
        method = getattr(self, f"{when}_update_state")
        delegator.setUpdateState(pyOverride(<PyObject*>method, callback_v_dp), cxx_when)

If there was something else you had in mind, I'd be interested to hear it, but I don't think this is solving any problems.

[ischoegl]

@speth ... thank you for looking into this. I believe I have an idea that I'd like to try (i.e. I'm planning to actually implement something, but I won't have time until the end of the week). That said, the main question is about ReactorDelegator.h; my remaining comments are mostly about docstrings.

[ischoegl]

@speth ... I looked at this in some more detail, and agree that while generic functions and delegators have some common traits, they really just apply for the case "replace" (at least at the Delegator.h level).

Beyond, I agree that your alternative implementation doesn't improve things, as the issue is that call-back functions need to be built somehow. One thing that I tried (briefly) was to use C++ function pointers. For example:

    typedef void(R::*voidFromDoubleMethod)(double);
    template <voidFromDoubleMethod method>
    std::function<void(double)> voidFromDoubleFunc(
        const std::function<void(double)>& func,
        const std::string& when)
    {
        return makeDelegate(func, when, [this](double value) { (this->*method)(value); });
    }

    virtual void setDelegate(const std::string& name,
                             const std::function<void(double)>& func,
                             const std::string& when) override
    {
        if (name == "initialize") {
            m_initialize = voidFromDoubleFunc<&R::initialize>(func, when);
    [...]
    } 

The main idea is to move the hard-to-parse stuff to a template, and use self-explanatory templated functions to do the work. In theory, this should cut down boilerplate significantly. In practice, this compiles fine, but segfaults, although some smaller toy problems worked without issues. So this may be somewhat fragile.

Another - somewhat different - idea I stumbled across uses std::bind, where I'm wondering whether instead of creating a custom binding using the Delegator templates, it may not be simpler to just implement a series of

    void noVoidFromValue(double value) {}

    std::map<std::string, int> timing { {"before", 0}, {"replace", 1}, {"after", 2}};

    virtual void initialize(double t0)  override{
        m_initialize[0](t0); // set initially to std::bind(noVoidFromValue, _1);`
        m_initialize[1](t0); // set initially to std::bind(R::initalize, _1);
        m_initialize[2](t0); // set initially to std::bind(noVoidFromValue, _1);
    }

where setDelegator (which would presumably use the current function signatures) would essentially just update the std::bind setup, i.e.

    virtual void setDelegate(const std::string& name,
                             const std::function<void(double)>& func,
                             const std::string& when) override
    {
        if (name == "initialize") {
            m_initialize[timing[when]] = std::bind(func, _1);
        } else if (name == "evalWalls") {
            m_evalWalls[timing[when]] = std::bind(func, _1);
        } else {
            throw NotImplementedError("ReactorDelegator::setDelegate",
                "For function named '{}' with signature void(double)", name);
        }
    }

I'm pretty sure this can be simplified further.

One nice thing about this implementation is that it should be easy to follow. I'm aware of bind having a modest (?) performance penalty, but I wonder whether this would be a bottleneck here. It's probably possible to implement the same with lambda functions ...

PS: I obviously picked a really simple example here (no return arguments, etc.). In this case, the resulting code simplifies quite a bit, but it will get more involved for other setDelegate versions, where templates may make sense ...

[ischoegl]

@speth ... I ended up running a more comprehensive trial where I replaced ReactorDelecator::initialize, ReactorDelegator::eval and ReactorDelegator::evalWalls with an alternative approach that loosely follows what I suggested in my earlier post (although I used lambda functions). I added eval, as it is at the core of the reactor, and is also covered by unit tests. The implementation is straightforward; everything compiles and unit tests are successful.

The implementation is the last commit on https://github.com/ischoegl/cantera/tree/delegating-reactor (4beffaa) ... if fully implemented, this would avoid the templates introduced by Delegator, which are replaced by vectors of lambda functions (currently, they have a fixed length, but that is easily refined). Overall, I believe this should result in a net reduction of the repetitive boilerplate, while being easier to follow. I won't go further with this as the idea is very obvious, so what is uploaded now can serve as a basis for discussion.

[speth]

Thanks, @ischoegl, this has been helpful for generating some interesting ideas. One interesting idea that you have in the implementation that you linked to, but didn't mention in your description is that you've eliminated the need for a separate function object to get array sizes by just calculating those in the corresponding overridden method of the Reactor class, e.g.

virtual void eval(double t, double* ydot) override {
    std::array<size_t, 1> neq = {R::neq()};
    ....
}

I had given some thought early on to just calling a fixed chain of before/replace/after methods. I'm glad to see you got this working at least for methods that have no return type. The case where I think this gets more complex is for methods that do have a return value and still supporting the before/after options, which is why I went with the structure that exists now.

I did have an idea based on your mention of function pointers, though, which led me to a way that does actually eliminate the if/else tree in setDelegate, and even manages to eliminate the overrides of setDelegate from the ReactorDelegator class. You can see a preliminary implementation in commit d90b52e (just pushed here) for functions of the type void(double*). This should become even simpler with the change to how the array-size functions work.

[ischoegl]

🎉 @speth ... I really like this idea which I honestly believe is a great concept and a huge simplification of the boilerplate! As a friendly amendment, I further believe what you pointed out about my trial can be combined here to further simplify things as:

        install("updateState", &m_updateState,
                [this](std::array<size_t, 1> neq, double* y) { R::updateState(y); }
        );

and

    virtual void updateState(double* y) override {
        std::array<size_t, 1> neq = {R::neq()};
        m_updateState(neq, y);
    }

I didn't test this, but you probably get the idea ... the neq can be buffered centrally as a class variable as it is presumably shared by a few methods.

PS: one thing that comes to mind is that at some point there may be a need to call one function before and another function after? I think that if all functions (original and delegated) share the same signature, the function calls can be compounded, so there may be only a need for a single map in Delegator, i.e. the m_base_v_dp may not be necessay as everything can be absorbed by m_funcs_v_dp?

PPS:

The case where I think this gets more complex is for methods that do have a return value and still supporting the before/after options, which is why I went with the structure that exists now.

I had this mostly coded, but stumbled across some undocumented string parameters and just pushed what was working as it already served the purpose. The idea would have been to move the decisions into the local ReactorDelegator methods. But dropping the setDelegate overloads from ReactorDelegator is ultimately a superior option.

[speth]

tada @speth ... I really like this idea which I honestly believe is a great concept and a huge simplification of the boilerplate! As a friendly amendment, I further believe what you pointed out about my trial can be combined here to further simplify things

Yes, that's exactly what I was implying, and would have the simplification of install that you suggest. It also deletes an argument out of two of the makeDelegate overloads, which doesn't hurt, either.

I didn't test this, but you probably get the idea ... the neq can be buffered centrally as a class variable as it is presumably shared by a few methods.

I'm not sure whether there's anything to be buffered here. For one, the value returned by R::neq() can change. Not frequently, but it does happen as part of object set-up, and is even affected by other delegated methods. Second, it should be possible for the compiler to fully inline the call so the cost should be minimal, compared to calling the user-provided Python function. But if there is a case where calculating the appropriate sizes is complicated, moving that calculation out of the hot path would make sense.

PS: one thing that comes to mind is that at some point there may be a need to call one function before and another function after? I think that if all functions (original and delegated) share the same signature, the function calls can be compounded, so there may be only a need for a single map in Delegator, i.e. the m_base_v_dp may not be necessay as everything can be absorbed by m_funcs_v_dp?

Yes, this idea of using the current function and wrapping it with the incoming "before" or "after" function had crossed my mind previously. I had kind of shied away from it because it felt strange to lose a way to directly call the original version of the function, but as long as no one wants to keep changing the delegated methods on an existing object, it shouldn't really matter. I couldn't come up with a case where wanting to do both "before" and "after" seemed like the best option, but there might be such cases somewhere, and I think doing this would also enable those without any additional hassle.

[ischoegl]

Thank you, @speth! This looks good to me.