Build model methods with -fno-gnu-unique on Linux with GCC, so that they can be loaded together with rstan

[avehtari]

Fixes #1177.

Fixes the "mixed mode" crash in #1177: on Linux with GCC, R segfaults when the model methods and rstan are loaded in the same R process, in either order.

This PR was assisted by Claude

Summary

Cause. GCC emits the static data of templates such as stan::math::AutodiffStackSingleton<vari_base, chainable_alloc> as STB_GNU_UNIQUE symbols. The dynamic loader binds a unique symbol once for the whole process, also for libraries loaded with RTLD_LOCAL. The model methods library and rstan.so both define AutodiffStackSingleton<...>::instance_ and init()::is_initialized. So the library that loads second uses the autodiff stack of the first one, which comes from another Stan Math build, and it segfaults in its static initialization. The TBB is not the cause: the crash also happens when only one TBB is in the process (RcppParallel 6.2.1 with rcppparallel_tbb() active), and it does not happen with -fno-gnu-unique while two TBB builds are in the process (RcppParallel 5.1.11.2 and TBB 2020.3).

Change.

• rcpp_source_stan() adds -fno-gnu-unique to PKG_CXXFLAGS on Linux when R's C++ compiler is GCC. The symbols then become ordinary weak symbols, and each library keeps its own autodiff stack. This covers $init_model_methods() and $expose_functions().
• New internal r_cxx_is_gcc(): runs R CMD config CXX and <CXX> --version and looks for "Free Software Foundation", which GCC prints and clang does not. A leading ccache or sccache is skipped. The result is cached in .cmdstanr. clang does not emit STB_GNU_UNIQUE, so it does not need the flag.
• Two tests in test-model-methods.R:
  • a unit test (mocks, no compiler) that the flag is added only for GCC on Linux;
  • a regression test that runs a small model's $init_model_methods() and loadNamespace("rstan") in a fresh R process, in both orders, with the cmdstanr under test. It runs only on Linux with rstan installed.

Results.

	v1.0 (3cfa89a)	this PR
model methods first, then rstan	segfault	OK
rstan first, then model methods	segfault	OK
test-model-methods.R	61 passed	68 passed (7 new)
test-model-expose-functions.R	86 passed	86 passed

Tested on Ubuntu 22.04 (R 4.6.1, g++ 11.4.0, CmdStan 2.40.0) and on a RHEL 9 cluster (R 4.5.3, GCC 13.3.0, CmdStan 2.40.0-rc0), with rstan 2.36.0.9000, StanHeaders 2.36.0.9000 and RcppParallel 5.1.11.2. The two orders also work with RcppParallel 6.2.1, StanHeaders 2.39.1 and rstan 2.39.0.9000, where rcppparallel_tbb() is active.

Not tested: clang on Linux. The PR does not pass the flag to clang.

rstan has the same unique symbols, but the change on the cmdstanr side is enough: when one of the two libraries does not take part in the unique binding, each keeps its own autodiff stack.

Copyright and Licensing

Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):
Aalto University

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

• Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[jgabry]

Thanks! I'll test this now.

[jgabry]

A few changes I pushed to test them on CI:

• Instead of -fno-gnu-unique, rcpp_source_stan() now adds R's own $(CXX_VISIBILITY), which R's Makeconf sets to -fvisibility=hidden for compilers that support it. The Stan Math symbols then aren't exported at all, so there's nothing for the loader to bind (Rcpp's wrappers stay exported because RcppExport marks them visible). It fixes both orders in the same container and works with clang too, so r_cxx_is_gcc() could go.
• The unit test that mocked os_is_linux() turned out to call rcpp_source_stan() for real, so on macOS check_sundials_fpic() appended -fPIC to make/local and rebuilt CmdStan. The rstan test always skips on CI since no job installs rstan. I replaced both with a test that checks the model-methods library doesn't export the autodiff stack symbol. It fails on v1.0 on macOS and Linux and passes now.

[avehtari]

$(CXX_VISIBILITY) works on my laptop and on Triton cluster

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (v1.0@6002ebe). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             v1.0    #1327   +/-   ##
=======================================
  Coverage        ?   93.95%           
=======================================
  Files           ?       19           
  Lines           ?     9158           
  Branches        ?        0           
=======================================
  Hits            ?     8604           
  Misses          ?      554           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.