Weak linking CMake module #47

opadron · 2016-07-06T01:35:21Z

Adapted from vtk#1511.

Adds a new experimental CMake module: targetLinkLibrariesWithDynamicLookup that allows one to portably indicate that a target uses symbols from another library, but that the resolution of symbols should be delayed until runtime, i.e.: the linker should tolerate undefined symbols when linking. "Weak linking" has been used to refer to this code sharing scheme as well as its many subtle variants.

Reviewers, please note the major differences in this PR when compared to vtk#1511.

This version uses two instances of a sample project with try_compile to infer the level of support for dynamic symbol resolution. The first determines if the linker responds to the dynamic lookup flag currently present in linker on OS X. The second determines if the linker and run-time loader can properly coalesce symbols that have been duplicated across link boundaries, such as ld/Linux. The original version only included the first check.
In addition, each sample project is tested after compiling to ensure that the test program produces the expected output as well as compile successfully. The original used a much simpler test program that would only compile.
In the original version, no link is performed when the linker is found to not support the dynamic lookup flag. In general, a proper linking is still necessary on platforms like Linux, whose loader can coalesce duplicate symbols at run-time. In this version, the link is performed when such a loader has been detected.
The original version's system checks were performed immediately upon module load and then cached. This version runs the system checks and caches on demand.

Also, includes a fix for when testing the tower-of-babel sample project under anaconda.

…make

AppVeyorBot · 2016-07-06T02:11:48Z

❌ Build scikit-build 0.0.1.255 failed (commit 53b3797760 by @opadron)

jcfr · 2016-07-06T04:02:45Z

tests/samples/cmake-files/targetLinkLibrariesWithDynamicLookup.cmake

+  set(run_output)
+
+  if(project_compiles)
+    execute_process(COMMAND "${test_project_bin_dir}/main"


execute_process(COMMAND ${CMAKE_CROSSCOMPILING_EMULATOR} "${test_project_bin_dir}/main"

blowekamp · 2016-07-06T17:43:37Z

In the original version, no link is performed when the linker is found to not
support the dynamic lookup flag. In general, a proper linking is still necessary
 on platforms like Linux, whose loader can coalesce duplicate symbols at 
run-time. In this version, the link is performed when such a loader has been 
detected.

The goal of the original cmake file (in SimpleITK) was NOT to link again libpython on Linux and OSX as per distutils and Pypa recommendations. This sounds like you are trying to still link against the library?

I am not sure what the goal of this is now...

blowekamp · 2016-07-06T17:50:05Z

tests/samples/cmake-files/targetLinkLibrariesWithDynamicLookup.cmake

+endfunction()
+
+
+function(target_link_libraries_with_dynamic_lookup target)


This is the original logic:

if ( ${CMAKE_SYSTEM_NAME} MATCHES "Darwin" ) set_target_properties( ${target} PROPERTIES LINK_FLAGS "-undefined dynamic_lookup" ) elseif(VTK_UNDEFINED_SYMBOLS_ALLOWED) # linker allows undefined symbols, let's just not link else() target_link_libraries ( ${target} ${ARGN} ) endif()

This is doing something different now.

blowekamp · 2016-07-06T17:55:39Z

I am also going to point out that having a single executable linked to two shared libraries has quite a different scope for symbol resolution than an executable dynamically loading at runtime two separate libraries.

jcfr · 2016-07-06T18:10:55Z

Thanks for the review brad.

For references:

blowekamp · 2016-07-06T18:19:10Z

This I think is how I began down this issue by naively linking against the shared library:

Perhaps it would be work testing on the manylinux1 docker image too?

opadron · 2016-07-06T20:30:23Z

The logic change was on purpose, but clearly, it was a mistake if everything I read is true.

# linker allows undefined symbols, let's just not link

So, my main confusion stems from the fact that I could not replicate this behavior on Linux -- the linker always refuses to leave the symbols undefined. Is there a linker flag I should be using?

I am also going to point out that having a single executable linked to two shared libraries has quite a different scope for symbol resolution than an executable dynamically loading at runtime two separate libraries.

This sounds right to me, but I'm not sure what the point of bringing this up is? Neither of those scenarios are being tested in the test project. Should they be? Besides this tolerance for undefined symbols that I have yet to observe, is there anything we should be checking for that we're not?

blowekamp · 2016-07-06T21:33:13Z

What is your platform and what exactly are you trying to do to test if "undefined" symbols are allowed? The original CMake file seems to pass on all my linux (elf binaries) systems I have tests [1].

I may have been mistaken about what this CMake code and the test project was trying to do. And I don't know why the test is constructed this way as opposed to other permutations of executable, linked library and runtime loaded library.

I also don't know what problem or issue the HAS_SYMBOL_DEDUPE is trying to address.

I thought I was understanding the linker process pretty well now, but I am guess I have more to read.

[1] https://github.com/SimpleITK/SimpleITK/blob/master/CMake/sitkTargetLinkLibrariesWithDynamicLookup.cmake#L44-L64

opadron · 2016-07-06T22:17:42Z

I haven't actually tried that particular test code, but I was more referring to building actual Python extension modules. When trying to build a simple extension module, the link fails with undefined symbol errors on Linux. That is- when I don't link to any libpython.

The test project in this PR is based on my experiments here. There is a shared lib, L; a module lib, M; and the executable, E. E is linked against L. E also dynamically loads M. And M, itself uses code from L, but the main question is whether M should be linked against L. This test was supposed to emulate a typical example of building a python extension module. As an aside, I haven't been able to build M without linking it against L on Linux, I run into the exact same problems as I did with actual Python extension modules.

I figure besides library version mismatches, the only other potential source of problems with linking for both the module and the executable is that it may result in code duplication. If the library had some shared state, we should be able to observe the effects of such duplication by inspecting the shared state from the perspectives of E and M, and then comparing. If there is code duplication, the counting routines from each would have their own counters instead of sharing. If the loader coalesces the duplicate symbols, then they would share, which is the desired behavior. This is what I check for under the "SYMBOL_DEDUPING" name.

I'm going to check to see if I can get your simpler test code to compile and link. If so, then there's clearly something about how linking on Linux works that I'm not privy to.

blowekamp · 2016-07-06T22:33:41Z

Another specific case where not linking solved a problem was on OS X. With binaries of SimpleITK compiled against the system Python 2.7 with explicit linking, when these would create a load error when run on Anaconda or home brew. However, if the "-undefined dynamic_lookup" was used and not linked against libpython, then the binaries would work!

SimpleITK is not linking against libpython, on every linux system I have tested RH5,RH6,RH7, Cent OS5, Some Debian, Ubuntu 12, Ubuntu 14....

I'd be interested in a minimal example and the link line used. I'm thinking of starting tests with gcc command lines here.

opadron · 2016-07-06T22:40:29Z

Well, I've finally gotten extension modules to build and link on Linux as you and others describe -- without linking.

I have no idea what I'm doing differently, but I'm now a believer. 😅

I want to try to incorporate this into my weak-linking-demo and see how that changes things. Then, I'll come back to this and make the necessary corrections.

Thanks for keeping me honest. 👍

opadron · 2016-07-08T18:05:33Z

Check out opadron/weak-linking-demo#3 for an updated version of the module and an updated discussion in the README.

AppVeyorBot · 2016-07-16T16:38:49Z

❌ Build scikit-build 0.0.1.256 failed (commit fc1b6b4774 by @opadron)

AppVeyorBot · 2016-07-16T17:26:39Z

✅ Build scikit-build 0.0.1.259 completed (commit 64868c1743 by @opadron)

opadron · 2016-07-16T20:45:02Z

This should be ready to go.

AppVeyorBot · 2016-07-16T20:50:36Z

✅ Build scikit-build 0.0.1.260 completed (commit 3e3fc5e67b by @opadron)

jcfr · 2016-07-16T21:16:34Z

tests/samples/cmake-files/targetLinkLibrariesWithDynamicLookup.cmake

+    endif()
+
+    file(APPEND "${test_project_src_dir}/main.c" "
+        result = count()    != 0 ? 1 :


1 -> EXIT_FAILURE

opadron · 2016-07-16T21:38:42Z

continued in #51

This commit integrates changes associated with the module from scikit-build/scikit-build@a415df9 The corresponding changes were originally added in scikit-build/scikit-build#47 and scikit-build/scikit-build@a606dc7 (handle missing target case) IMPORTANT: This backported commit causes all tests to *FAIL*. It has been kept like this so that the next commit can clearly illustrate what is required to fix them. It also mean that the implementation in "scikit-build" is broken. For example, on Linux, output is: $ rm -rf _build/ && dockcross-manylinux-x64 ./testAllBuilds.bash CASE CONFIG BUILD TEST s00 fail fail RTE s01 fail fail RTE s10 fail fail RTE s11 pass pass pass d00 fail fail RTE d01 fail fail RTE d10 fail fail RTE d11 pass pass pass whereas the following is expected: CASE CONFIG BUILD TEST s00 pass pass RTE s01 pass pass pass s10 pass pass RTE s11 pass pass pass d00 pass pass RTE d01 pass pass pass d10 pass pass RTE d11 pass pass pass

This commit has two purposes: (1) it fixes logic checking for cross-compilation (the regression was introduced by #51 and #47 (2) It configure the text project setting CMAKE_ENABLE_EXPORTS to ON. Doing so ensure the executable compiled in the test exports symbols (if supported by the underlying platform). See https://cmake.org/cmake/help/v3.6/prop_tgt/ENABLE_EXPORTS.html?highlight=enable_export

opadron added 2 commits July 5, 2016 20:30

add experimental CMake Module: targetLinkLibrariesWithDynamicLookup.c…

ecc88b7

…make

fix issue with tower-of-babel tests failing when ran under anaconda

c7fa944

opadron mentioned this pull request Jul 6, 2016

Integrate (itk|vtk)TargetLinkLibrariesWithDynamicLookup module in upstream CMake #37

Closed

jcfr reviewed Jul 6, 2016
View reviewed changes

blowekamp reviewed Jul 6, 2016
View reviewed changes

opadron mentioned this pull request Jul 8, 2016

improve weak link handling opadron/weak-linking-demo#3

Merged

improve dynamic lookup module

f92eb41

opadron added 3 commits July 16, 2016 11:42

handle missing target case

a8e3298

debug tbabel

8c8aaad

debug tbabel

5eeb0a1

debug tbabel

eab8a30

jcfr reviewed Jul 16, 2016
View reviewed changes

x

65f8367

opadron mentioned this pull request Jul 16, 2016

[REBASED] weak linking module #51

Merged

opadron closed this Jul 16, 2016

opadron deleted the weak-linking-module branch July 16, 2016 21:38

jcfr mentioned this pull request Sep 25, 2016

Fix targetLinkLibrariesWithDynamicLookup #190

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Weak linking CMake module #47

Weak linking CMake module #47

opadron commented Jul 6, 2016

AppVeyorBot commented Jul 6, 2016

jcfr Jul 6, 2016

blowekamp commented Jul 6, 2016 •

edited by jcfr

Loading

blowekamp Jul 6, 2016

blowekamp commented Jul 6, 2016 •

edited

Loading

jcfr commented Jul 6, 2016

blowekamp commented Jul 6, 2016 •

edited

Loading

opadron commented Jul 6, 2016

blowekamp commented Jul 6, 2016 •

edited

Loading

opadron commented Jul 6, 2016

blowekamp commented Jul 6, 2016

opadron commented Jul 6, 2016

opadron commented Jul 8, 2016

AppVeyorBot commented Jul 16, 2016

AppVeyorBot commented Jul 16, 2016

opadron commented Jul 16, 2016

AppVeyorBot commented Jul 16, 2016

jcfr Jul 16, 2016

opadron commented Jul 16, 2016

		endfunction()


		function(target_link_libraries_with_dynamic_lookup target)

Weak linking CMake module #47

Weak linking CMake module #47

Conversation

opadron commented Jul 6, 2016

AppVeyorBot commented Jul 6, 2016

jcfr Jul 6, 2016

Choose a reason for hiding this comment

blowekamp commented Jul 6, 2016 • edited by jcfr Loading

blowekamp Jul 6, 2016

Choose a reason for hiding this comment

blowekamp commented Jul 6, 2016 • edited Loading

jcfr commented Jul 6, 2016

blowekamp commented Jul 6, 2016 • edited Loading

opadron commented Jul 6, 2016

blowekamp commented Jul 6, 2016 • edited Loading

opadron commented Jul 6, 2016

blowekamp commented Jul 6, 2016

opadron commented Jul 6, 2016

opadron commented Jul 8, 2016

AppVeyorBot commented Jul 16, 2016

AppVeyorBot commented Jul 16, 2016

opadron commented Jul 16, 2016

AppVeyorBot commented Jul 16, 2016

jcfr Jul 16, 2016

Choose a reason for hiding this comment

opadron commented Jul 16, 2016

blowekamp commented Jul 6, 2016 •

edited by jcfr

Loading

blowekamp commented Jul 6, 2016 •

edited

Loading

blowekamp commented Jul 6, 2016 •

edited

Loading

blowekamp commented Jul 6, 2016 •

edited

Loading