Determine how to package native shims

[nguerrera]

For issue #14752, we'll be creating native libraries that need to be compiled for each unix flavor we support. We need to figure out how these are distributed (e.g. alongside corefx nuget packages? with the runtime? etc.)

[ghost]

Looking at this project: https://www.nuget.org/profiles/MonoGame, IMO it will be better to package System.IO.Native packaged for each platform separately as:

• System.IO.Native.FreeBSD
• System.IO.Native.Linux
• System.IO.Native.OSX
• System.IO.Native.Windows

This will keep the package size minimal with no extra/unrelated platform binaries bundled together (like some NuGet packages are bundling Mono targets when installing on Windows).

If the conditional (platform-based) dependency is not possible in NuGet/MyGet infrastructure, then bundling them as one System.IO.Native is the only option I imagine.

In either case, CI build artifacts can be leveraged to build all the combinations of binary without hassling the release team to arrange VMs for all the (growing) OS set. On this note, it would be really cool to use GitHub tags with semver and set the CI to trigger the release assets build binaries with current master, only when new tags are pushed right after the "last commit which should go in the vNext release".

[weshaggard]

Temporarily I've added these shim assemblies to our runtime package (Microsoft.NETCore.Runtime.CoreCLR) until we figure out the exact packaging strategy.

[ericstj]

We were actually planning to package as follows,

• Each native shim will have its own "identity" package.
• Each native shim implementation will be a runtime specific implementation of the identity package.

System.Native
runtime.centos.7.1-x64.System.Native
runtime.osx.10.10-x64.System.Native
runtime.ubuntu.14.04-x64.System.Native

• Each package that depends on the shim will reference the identity package from its implementation package which depends on it. Alone this will hit the nuget bug where runtime.jsons are ignored if not in the compile graph. To avoid that we must make sure that all shim runtime.jsons are merged into the lineup.
• Note placing this information in the lineup is required universally since we can never garuntee that the package which requires platform-specific implementation is in the compile graph. Eventually this should get better with: Nuget needs to support dependencies specific to target runtime NuGet/Home#1660

[davidfowl]

We have to move away from individual packages and towards the lineup file for these types of packages.

[ericstj]

Thank you for agreeing with the proposal @davidfowl

[davidfowl]

@ericstj Yep. I don't want rely on this NuGet/Home#1660 for these specific packages even when it becomes available 😄

[ericstj]

Yeah, we won't rely on it that feature. Just wanted to point out why it's required. For the .NET framework we have the nice benefit of always being able to put that lineup in the compile graph. For regular 3rd party devs they cannot so they need NuGet/Home#1660.

[mellinoe]

What is "the lineup"? I don't think a lot of people are familiar with some of this new jargon, so it's a bit hard to understand the discussion 😄 .

And just to clarify/discuss: since a lot of different assemblies are dependent on System.Native, does this mean that once we release a stable version, we have to maintain that ABI forever? Otherwise you will be required to choose various versions of libraries that are all compatible with the version of the shim you use (or rather, all libraries must depend on the same version of the shim). Lots of libraries depend on System.Native, and I'm pretty sure we've made at least a few binary breaking changes so far. Just trying to figure out our thoughts on this, I didn't see it written down anywhere yet.

[nguerrera]

I do think we'll have to keep the ABI stable post-RTM, but I think this is less burdensome than with public managed API. We don't care about the aesthetics of the API as they are implementation details and we can always define Foo2(), etc. If absolutely necessary, we can also rename the .so and package and allow different, incompatible versions to live SxS.

[ericstj]

Lineup is a central representation of all the RID-specific associations of our packages. Each package represents its RID-specific implementation locally in our build system, but that all gets merged into a central runtime.json: the lineup.

[ghost]

Why would we want to have CentOS, Ubuntu, Fedora, segregation? I mean if you build something on CentOS5 (5.11 is currently officially supported in 5x branch for example), it will definitely run on 95+% of linux running on servers and probably 99% of clients (only if someone is still running very old linux with libc version older than the one packed with CentOS, then it wont work).

What is the deal with CentOS "5"?

Large amount of web hosting server till date are running CentOS 5, but that is not the real answer. I have experienced it and it is a proven fact from many C library folks including node.js guys, that binary built on CentOS5 is qualified as "ultimate cross linux distribution compatible".

To simplify your build system and save time, I would highly recommend a Docker recipe setup (which i can provide if you are interested as I have done it more than once). Only <package-manager> install docker once and then docker run <machine-name> whenever you want and that will magically produce both x86 and x64 compatible binaries in a zip file without any further hassle. With some extra work in future, we can incorporate ARM (and maybe good old MIPS) cross-compilation too. :)

In case of BSD though, we will have to target each kind of BSD separately: Free, Net, DragonFly yada yada (but BSD community can always wait and they are very cool about it 😄).

[eerhardt]

Why would we want to have CentOS, Ubuntu, Fedora, segregation?

Because this issue isn't about just 'libc', it is about the set of native libraries that we are using: libc, libcurl, openSSL, zlib.

For example, the version of libcurl used on our CentOS CI servers is v7.29, but on Ubuntu is v7.35, and we can have differing code based on which version we are compiling against.

[ghost]

One approach could be to bundle the dependent bins or compile them statically, so ldd native.so will only report libc and libc++ at the cost of gain in output binary size.
The other approach could be to branch and adapt to the change at runtime. I believe there are not many varied code paths which violate reasonable divergence bar.

To recap, it boils down to three choices (maybe there are more?):

1. runtime branching.
2. fat binary due to static compilation.
3. compile on every linux distribution (and possibly every version of those distros where the OOTB libs are different).

The third one is most cumbersome, <number of supported distros> x <number of their respective supported versions>-times the effort of building + nuget packaging and limits the ability to cover large scale with less effort.

One strong point about the third one is that for distributions which are very different (like Apline Linux, which doesn't even uses libc but microlibC OR other kinds of libc's which target RISC and their consumer distros), the drastic code variation will not impact the general, more commonly used, distributions.

[chcosta]

This feature was checked in. We now have native packages for these native libraries which are referenced by the "identity" package.