Build libmagic if not installed

[robo9k]

In addition to pkg-config (#1) another fallback option would be building libmagic if it is not installed for the target system.

This could be done using a Git submodule of file/file.
libmagic (and file) uses the GNU autotools to compile and install.

Building libmagic on Windows (#2) and OS X (#3) needs to be considered as well.

[koutheir]

Would a pull request be accepted if it starts solving this issue?

[robo9k]

Hej @koutheir !
It's difficult to give a blanket approval before having seen the pull request :)
I'm certainly open to merging improvements to the current state of things. What do you have in mind, what would be left unresolved?

[koutheir]

It's difficult to give a blanket approval before having seen the pull request :)

Sure, I just wanted to know there's a possibility before I start changing stuff.

What do you have in mind

Building libmagic from source and linking it statically, in addition to some dependencies like libz, libbz2, liblzma. All while making sure it supports cross compilation for Linux targets outside amd64.

what would be left unresolved?

Support for other operating systems, e.g., FreeBSD, OS X, Windows, etc.

[robo9k]

What do you have in mind?

Building libmagic from source and linking it statically, in addition to some dependencies like libz, libbz2, liblzma. All while making sure it supports cross compilation for Linux targets outside amd64.

That would be a huge step forward from what's in place right now!

what would be left unresolved?

Support for other operating systems, e.g., FreeBSD, OS X, Windows, etc.

As long as linking to an existing libmagic continues to work (see #11, doesn't have to be 1:1) I don't see that as a problem. Support for other platforms can always be added lateron. Currently they are neither documented nor tested in CI anyways.

I'd say go for it and thanks for your help and interest 👍

[koutheir]

I made some progress on the issue, but it's still on my local machine. I wanted to discuss with you some breaking changes that I intend to do in a future pull request, so that we agree on the ideas before I push some code.

[koutheir]

In order to align this crate to the pseudo-conventions for environment variables, established by crates such as cc and libz-sys, the environment variables controlling how libmagic is built, will become the following:

• MAGIC_SYS_DIR_<target-triple-with-underscores>, otherwise MAGIC_SYS_DIR.
• MAGIC_SYS_STATIC_<target-triple-with-underscores>, otherwise MAGIC_SYS_STATIC.
• CC_<target-triple-with-underscores>, otherwise CC.
• AR_<target-triple-with-underscores>, otherwise AR.
• RANLIB_<target-triple-with-underscores>, otherwise RANLIB.
• SYSROOT_<target-triple-with-underscores>, otherwise SYSROOT.

[koutheir]

libmagic can be provided externally, either in dynamic or static form. If provided in both forms, then the choice of which form to use will depend on the platform:

• On Linux, link dynamically unless the ABI is musl*.
• On Windows, link dynamically unless the ABI is mingw32*.

[koutheir]

If libmagic is not provided externally, then it is build from source and linked statically. The sources built are the following:

• libz.a built by the (now dependency) libz-sys crate.
• libbz2.a built by the (now dependency) bzip2-sys crate.
• liblzma.a built by the (now dependency) lzma-sys crate.
• If building for Windows based on mingw32, the library libgnurx is built from sources, producing libgnurx.dll.a and libregex.a.
• libmagic.a built from sources of the file tool, where support for Zip, BZip2 and XZ compressed files is enabled.

[koutheir]

The static building of libraries from sources is based on the external official build system, namely autoconf, instead of reimplementing the building steps manually based on the cc crate. This is chosen to avoid importing the complexity handled by those build systems in to build.rs (which is now already more than 600 lines).

[koutheir]

I built the following build configurations successfully:

• Targeting x86_64-unknown-linux-gnu, statically linking libmagic built from sources:

  $ cargo build
  

• Targeting x86_64-unknown-linux-gnu, dynamically linking to an external libmagic:

  $ MAGIC_SYS_DIR_x86_64_unknown_linux_gnu=/usr/lib/x86_64-linux-gnu cargo build
  

• Targeting arm-unknown-linux-gnueabihf, statically linking libmagic built from sources:

  $ CC_arm_unknown_linux_gnueabihf=.../gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf/bin/arm-none-linux-gnueabihf-gcc cargo build --target=arm-unknown-linux-gnueabihf
  

• Targeting armv5te-unknown-linux-gnueabi, statically linking libmagic built from sources:

  $ CC_armv5te_unknown_linux_gnueabi=.../armv5-unknown-linux-gnueabi/bin/armv5-unknown-linux-gnueabi-gcc cargo build --target=armv5te-unknown-linux-gnueabi
  

• Targeting aarch64-unknown-linux-musl, statically linking libmagic built from sources:

  $ CC_aarch64_unknown_linux_musl=.../aarch64-unknown-linux-musl/bin/aarch64-linux-musl-gcc cargo build --target=aarch64-unknown-linux-musl
  

• Targeting x86_64-pc-windows-gnu, statically linking libmagic built from sources:

  $ CC_x86_64_pc_windows_gnu=/usr/bin/x86_64-w64-mingw32-gcc cargo build --target=x86_64-pc-windows-gnu
  

[koutheir]

I pushed the code on my own fork of the repository, so that you can take a look if you need more details. I plan to make a pull request once we agree on these changes.

[koutheir]

Isn't there any interest in this discussion and future pull request?

[robo9k]

Couple things:

• is it neccessary for the added -sys crate dependencies to be unconditional or would it be possible to activate those only when needed?
• what are the consequences of including file and libgnurx sources in the repo?
  Currently magic-sys is licensed as MIT/Apache-2.0 and I'd like to keep it that way if possible (including your contributions, as stated in the README)
• are there any changes to the included sources? it's hard to see since it's not a git submodule or such
• would it be possible to trim down the included sources? I imagine they'll bloat the size of the magic-sys crate quite a bit
• could probably use a macro or such for the panic!(format!("rust-magic-sys: …")) repetitions
• I'm seeing explicit code for "linux" and "windows", does that mean building on MacOS will no longer work or only with certain (non-default?) build options?
• could you update the README for the new build options?
  In some cases it seems autotools is an external dependency that needs to be installed
• could you add some docs to/about magic-sys-helpers.c?
  Why is it needed, what does it do? Possibly link to other external documentation.

[koutheir]

is it neccessary for the added -sys crate dependencies to be unconditional or would it be possible to activate those only when needed?

I couldn't find a way to make a cargo dependency conditional. If you have an idea how to do so, then please tell me.

what are the consequences of including file and libgnurx sources in the repo?
Currently magic-sys is licensed as MIT/Apache-2.0 and I'd like to keep it that way if possible (including your contributions, as stated in the README)

I suppose you mean licensing consequences... I don't know, I'm not a lawyer. My contributions are licensed under the public domain, so that's not an issue.

are there any changes to the included sources? it's hard to see since it's not a git submodule or such

I made no changes to the sources in file/ and in libgnurx/. These are verbatim extracted sources from file-5.39.tar.gz and mingw-libgnurx-2.5.1-src.tar.gz.

would it be possible to trim down the included sources? I imagine they'll bloat the size of the magic-sys crate quite a bit

I suppose it's possible, but it requires understanding their build system and portability measures. If this is done, then we can also avoid relying on the build systems to build the C sources and generated data. However, this will further complicate build.rs.

could probably use a macro or such for the panic!(format!("rust-magic-sys: …")) repetitions

Yes, it's annoying me too. I'm just avoiding macros for the moment, but we might consider factorizing these calls as a cleanup measure.

I'm seeing explicit code for "linux" and "windows", does that mean building on MacOS will no longer work or only with certain (non-default?) build options?

I've no idea what happens on MacOS.

could you update the README for the new build options?

This definitely needs to be done, but I prefer doing it after the sources seem okay for all reviewers.

In some cases it seems autotools is an external dependency that needs to be installed

True, and that's the price of depending on these official external build systems.

could you add some docs to/about magic-sys-helpers.c?
Why is it needed, what does it do? Possibly link to other external documentation.

Sure.

[koutheir]

For simplification reasons, we might consider making a new -sys crate to wrap the libgnurx library. That would allow us to simply depend on libgnurx-sys crate instead of handling it internally.

[koutheir]

Here is my gnurx-sys: Unsafe Rust bindings for libgnurx.

[koutheir]

could you add some docs to/about magic-sys-helpers.c?
Why is it needed, what does it do? Possibly link to other external documentation.

See Added documentation for bz_internal_error() and ioctl().

[koutheir]

Well, it seems no real interest is there for any of this stuff I worked on. I'll keep my changes private.