expose_stan_functions won't work with external C++ functions

[dheimgartner]

Summary:

rstan::expose_stan_functions() yields an error during compilation when the stan function uses an external C++ function in its definition.

Description:

• The vignette vignette("external", "rstan") describes how to leverage user-defined C++ functions in a stan model.
• In particular, I would like to define a helper function in the stan functions block that uses my own C++ function (in its definition).
• Include directives are passed as arguments as described in the above mentioned vignette.
• The rstan::stan_model() compiles the model successfully and I can sample from it. However,
• rstan::expose_stan_functions() yields an error during compilation.

Reproducible Steps:

Here follows a reprex. The api() function is just a wrapper around expose_stan_functions(). Issues are commented via ##Issue: :

library(rstan)
library(Rcpp)

rm(list = ls())


test <- '
functions {
  int my_sum(int a, int b) {
    return a + b;
  }
}
model {
  ;
}
'

test_model <- stan_model(model_code = test, model_name = "test", verbose = TRUE)

api <- function(stanmodel, includes = NULL, verbose = TRUE, ...) {
  e <- new.env()
  rstan::expose_stan_functions(stanmodel, includes = includes, env = e, verbose = verbose, ...)
  return(as.list(e))
}

test_api <- api(test_model)
test_api$my_sum(1, 2)


## Now, let's try to outsource the my_sum function to my_sum.hpp

## first check the expected signature
cat(test_model@model_cpp$model_cppcode)
## int my_sum(const int&a, const int&b, std::ostream* pstream__)


my_sum.hpp <- '
int my_sum(const int&a, const int&b, std::ostream* pstream__) {
  return a + b;
}
'

## write to file in wd
sink(file = "my_sum.hpp")
cat(my_sum.hpp)
sink()

mc <- '
functions {
  int my_sum(int a, int b);
}
model {
  ;
}
'

includes <- paste0('\n#include "', 
                   file.path(getwd(), 'my_sum.hpp'), '"\n')

external_model <- stan_model(model_code = mc, model_name = "external",
                             allow_undefined = TRUE,
                             includes = includes,
                             verbose = TRUE)

external_api_1 <- api(external_model, includes = includes)
## Issue: The my_sum function is not exported...

## my_sum is only declared but not defined...

mc <- '
functions {
  int my_sum(int a, int b);
  int my_sum_wrapper(int a, int b) {
    return my_sum(a, b);
  }
}
model {
  ;
}
'
external_model <- stan_model(model_code = mc, model_name = "external",
                             allow_undefined = TRUE,
                             includes = includes,
                             verbose = TRUE)

external_api_2 <- api(external_model, includes = includes, verbose = TRUE)
## Issue: won't compile

## what if we don't call my_sum in the functions block??
mc <- '
functions {
  int my_sum(int a, int b);
  void my_api() {
    print("hello from my_api");
  }
}
model {
  ;
}
'
external_model <- stan_model(model_code = mc, model_name = "external",
                             allow_undefined = TRUE,
                             includes = includes,
                             verbose = TRUE)

external_api_3 <- api(external_model, includes = includes, verbose = TRUE)
# Issue: will compile but (obviously) as before, not export my_sum

external_api$my_api()

Current Output:

The api() call (wrapping expose_stan_functions()) which should generate external_api_2 yields the following output before throwing the compilation error:

using C++ compiler: ‘Apple clang version 12.0.0 (clang-1200.0.32.29)’
using C++17
using SDK: ‘’
clang++ -std=gnu++17 -I"/usr/local/Cellar/r/4.3.1/lib/R/include" -DNDEBUG   -I"/usr/local/lib/R/4.3/site-library/Rcpp/include/"  -I"/usr/local/lib/R/4.3/site-library/RcppEigen/include/"  -I"/usr/local/lib/R/4.3/site-library/RcppEigen/include/unsupported"  -I"/usr/local/lib/R/4.3/site-library/BH/include" -I"/usr/local/lib/R/4.3/site-library/StanHeaders/include/src/"  -I"/usr/local/lib/R/4.3/site-library/StanHeaders/include/"  -I"/usr/local/lib/R/4.3/site-library/RcppParallel/include/"  -I"/usr/local/lib/R/4.3/site-library/rstan/include" -DEIGEN_NO_DEBUG  -DBOOST_DISABLE_ASSERTS  -DBOOST_PENDING_INTEGER_LOG2_HPP  -DSTAN_THREADS  -DUSE_STANC3 -DSTRICT_R_HEADERS  -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION  -D_HAS_AUTO_PTR_ETC=0  -include '/usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp'  -D_REENTRANT -DRCPP_PARALLEL_USE_TBB=1   -I"/usr/local/lib/R/4.3/site-library/Rcpp/include" -I"/usr/local/lib/R/4.3/site-library/StanHeaders/include" -I"/usr/local/lib/R/4.3/site-library/rstan/include" -I"/usr/local/lib/R/4.3/site-library/RcppEigen/include" -I"/usr/local/lib/R/4.3/site-library/BH/include" -I"/private/var/folders/w5/j_vympl57nq2plg57t3yvy1r0000gn/T/RtmpDPYWid/sourceCpp-x86_64-apple-darwin19.6.0-1.0.11" -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/xz/include -I/usr/local/include    -fPIC  -g -O2  -c filea1c8ba937c.cpp -o filea1c8ba937c.o
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:1:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Core:540:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:2:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/LU:47:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:3:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Cholesky:12:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Jacobi:29:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:3:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Cholesky:43:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:4:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/QR:15:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Householder:27:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:4:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/QR:48:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:5:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/SVD:48:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:6:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Geometry:58:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Dense:7:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Eigenvalues:58:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:23:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Sparse:26:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/SparseCore:66:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:23:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Sparse:27:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/OrderingMethods:71:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:23:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Sparse:29:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/SparseCholesky:43:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:23:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Sparse:32:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/SparseQR:34:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from <built-in>:1:
In file included from /usr/local/lib/R/4.3/site-library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:23:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/Sparse:33:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/IterativeLinearSolvers:46:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from filea1c8ba937c.cpp:10:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/RcppEigen.h:25:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/RcppEigenForward.h:32:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/unsupported/Eigen/KroneckerProduct:34:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from filea1c8ba937c.cpp:10:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/RcppEigen.h:25:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/RcppEigenForward.h:36:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/unsupported/Eigen/Polynomials:135:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from filea1c8ba937c.cpp:10:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/RcppEigen.h:25:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/RcppEigenForward.h:37:
In file included from /usr/local/lib/R/4.3/site-library/RcppEigen/include/unsupported/Eigen/SparseExtra:51:
/usr/local/lib/R/4.3/site-library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
17 warnings generated.
clang++ -std=gnu++17 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/Cellar/r/4.3.1/lib/R/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/xz/lib -L/usr/local/lib -o sourceCpp_37.so filea1c8ba937c.o /usr/local/lib/R/4.3/site-library/rstan/lib//libStanServices.a -L/usr/local/lib/R/4.3/site-library/StanHeaders/lib/ -lStanHeaders -L/usr/local/lib/R/4.3/site-library/RcppParallel/lib/ -ltbb -L/usr/local/Cellar/r/4.3.1/lib/R/lib -lR -lintl -Wl,-framework -Wl,CoreFoundation
Error in rstan::expose_stan_functions(stanmodel, includes = includes, :
Compilation failed!

Expected Output:

• I would expect the expose_stan_functions() to export both the my_sum and my_sum_wrapper functions...
• From its function documentation ?expose_stan_functions (in particular the description of the includes argument), I would expect that the function was intended to be used together with external C++ source...

Here is the excerpt of the documentation on includes:

"If not NULL (the default), then a character vector of length one (possibly containing one or more "\n") of the form '#include "/full/path/to/my_header.hpp"', which will be inserted into the C++ code in the model's namespace and can be used to provide definitions"

RStan Version:

2.32.3

R Version:

R version 4.3.1 (2023-06-16)

Operating System:

OS X 10.15.7

Your help would be much appreciated! Thank you!