Help building rusqlite for wasm32-unknown-unknown

[paulyoung]

I've described my issues with building rusqlite/rusqlite#1010 using crane in detail at https://discourse.nixos.org/t/help-building-rusqlite-for-wasm32-unknown-unknown-with-crane/21724 and created a standalone project at https://github.com/paulyoung/rusqlite-wasm32-unknown-unknown-nix . I'd appreciate it if someone could take a look.

My main motivation for filing this issue is that I (perhaps incorrectly) suspect that crane might be doing something to the build that I don't understand (perhaps via automatic handling of git dependencies?) such as trying to set certain environment variables.

Apologies if this is not an appropriate use of issues for this project. I suspect it might be an interesting use case to discuss and helpful for other crane users in the future.

Thanks for crane; this is my first time using it and I like it a lot so far.

[dpc]

I would be interested in this as well. AFAIU, it will not work because C compilation is involved and that is currently not supported/broken for this target.

fedimint/fedimint#441 (comment)

[ipetkov]

@paulyoung random question, but is there a reason you are targeting wasm32-unknown-unknown instead of wasm32-wasi?

My understanding is that sqlite expects to do I/O internally and the wasm32-unknown-unknown target might not have the appropriate bindings it needs. I think the wasm32-wasi target should be a better fit (depending on what you are trying to do), at least changing your .cargo/config.toml and flake.nix files to specify wasm32-wasi appears to build fine for me (on x86 linux)

Hope this helps!

[paulyoung]

@ipetkov yes, the platform I’m using has a custom Wasm environment that doesn’t support Wasi and requires wasm32-unknown-unknown.

The branch of rusqlite I’m trying to use added support for wasm32-unknown-unknown, and I’m registering a custom VFS which should work for that target (to answer your question about I/O).

[ipetkov]

@paulyoung ah I see! Unfortunately I'm not very familiar with libsqlite3-sys's internals, specifically around how it compiles the sqlite source to wasm when the bundled feature is enabled.

Are you able to successfully build the project outside of Nix? If yes, then clearly there is some part of the toolchain which is missing in the Nix version and needs to be included (or have the right environment variables set) to make the compilation work

[ipetkov]

@paulyoung I just saw this comment and taking it's advice to set AR in the build expression appears to build the project for me! Could you give that a try and see if it works?

diff --git a/flake.nix b/flake.nix
index 6b73a79..85751b4 100644
--- a/flake.nix
+++ b/flake.nix
@@ -43,17 +43,16 @@
           # our specific toolchain there.
           craneLib = (crane.mkLib pkgs).overrideToolchain rustWithWasmTarget;
 
+          stdenv = pkgs.llvmPackages_14.stdenv;
+
           rusqlite-wasm32-unknown-unknown-nix = craneLib.buildPackage ({
+            inherit stdenv;
             src = ./.;
             cargoExtraArgs = "--package rusqlite-wasm32-unknown-unknown-nix";
             # crane tries to run the Wasm file as if it were a binary
             doCheck = false;
-            # Without setting TARGET_CC we run into:
-            #
-            #   "valid target CPU values are: mvp, bleeding-edge, generic"
-            #
-            # https://github.com/rusqlite/rusqlite/pull/1010#issuecomment-1247333415
-            TARGET_CC = "${pkgs.stdenv.cc.nativePrefix}cc";
+            CC = "${stdenv.cc.nativePrefix}cc";
+            AR = "${stdenv.cc.nativePrefix}ar";
           });
         in
           {

[dpc]

Did it work?

[paulyoung]

I'm about to try this. It looks very promising!

[paulyoung]

I applied the changes above in paulyoung/rusqlite-wasm32-unknown-unknown-nix@8698a65 and ran into the same error as when I didn't set TARGET_CC.

Full log

cargoArtifacts not set, will not reuse any cargo artifacts
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/zym8hl55jd14f7ymznkkrm7bs3y9v1qg-dummy-src
source root is dummy-src
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
Executing configureCargoCommonVars
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
will append /private/tmp/nix-build-cargo-package-deps-0.0.1.drv-0/dummy-src/.cargo-home/config.toml with contents of /nix/store/awr9r66x6dldaiqzv4l20c3dal2j2h31-vendor-cargo-deps/config.toml
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
++ command cargo --version
cargo 1.63.0-nightly (39ad1039d 2022-05-25)
++ command cargo check --profile release --all-targets --package rusqlite-wasm32-unknown-unknown-nix
�[0m�[0m�[1m�[32m   Compiling�[0m proc-macro2 v1.0.43
�[0m�[0m�[1m�[32m   Compiling�[0m quote v1.0.21
�[0m�[0m�[1m�[32m   Compiling�[0m unicode-ident v1.0.4
�[0m�[0m�[1m�[32m   Compiling�[0m log v0.4.17
�[0m�[0m�[1m�[32m   Compiling�[0m wasm-bindgen-shared v0.2.83
�[0m�[0m�[1m�[32m   Compiling�[0m syn v1.0.99
�[0m�[0m�[1m�[32m   Compiling�[0m cfg-if v1.0.0
�[0m�[0m�[1m�[32m   Compiling�[0m bumpalo v3.11.0
�[0m�[0m�[1m�[32m   Compiling�[0m once_cell v1.14.0
�[0m�[0m�[1m�[32m   Compiling�[0m wasm-bindgen v0.2.83
�[0m�[0m�[1m�[32m   Compiling�[0m version_check v0.9.4
�[0m�[0m�[1m�[32m   Compiling�[0m vcpkg v0.2.15
�[0m�[0m�[1m�[32m   Compiling�[0m cc v1.0.73
�[0m�[0m�[1m�[32m   Compiling�[0m pkg-config v0.3.25
�[0m�[0m�[1m�[32m   Compiling�[0m memchr v2.5.0
�[0m�[0m�[1m�[32m   Compiling�[0m rusqlite-wasm32-unknown-unknown-nix v0.1.0 (/private/tmp/nix-build-cargo-package-deps-0.0.1.drv-0/dummy-src/crates/rusqlite-wasm32-unknown-unknown-nix)
�[0m�[0m�[1m�[32m    Checking�[0m fallible-streaming-iterator v0.1.9
�[0m�[0m�[1m�[32m    Checking�[0m bitflags v1.3.2
�[0m�[0m�[1m�[32m    Checking�[0m fallible-iterator v0.2.0
�[0m�[0m�[1m�[32m    Checking�[0m smallvec v1.9.0
�[0m�[0m�[1m�[32m   Compiling�[0m ahash v0.7.6
�[0m�[0m�[1m�[32m   Compiling�[0m libsqlite3-sys v0.23.2 (https://github.com/trevyn/rusqlite.git?branch=wasm32-unknown-unknown#004cf263)
�[0m�[0m�[1m�[32m    Checking�[0m hashbrown v0.11.2
�[0m�[0m�[1m�[32m    Checking�[0m hashlink v0.7.0
The following warnings were emitted during compilation:
�[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m clang-11: warning: argument unused during compilation: '-mmacos-version-min=11.0' [-Wunused-command-line-argument]

�[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m clang-11: warning: argument unused during compilation: '-arch arm64' [-Wunused-command-line-argument]

�[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m error: unknown target CPU 'apple-a13'

�[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m note: valid target CPU values are: mvp, bleeding-edge, generic
�[0m�[0m�[1m�[31merror�[0m�[1m:�[0m failed to run custom build command for libsqlite3-sys v0.23.2 (https://github.com/trevyn/rusqlite.git?branch=wasm32-unknown-unknown#004cf263)
Caused by:

process didn't exit successfully: /private/tmp/nix-build-cargo-package-deps-0.0.1.drv-0/dummy-src/target/release/build/libsqlite3-sys-e67e3d8f4797af26/build-script-build (exit status: 1)

--- stdout

cargo:rerun-if-changed=sqlite3/sqlite3.c

cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c

cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER

cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH

cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS

TARGET = Some("wasm32-unknown-unknown")

OPT_LEVEL = Some("3")

HOST = Some("aarch64-apple-darwin")

CC_wasm32-unknown-unknown = None

CC_wasm32_unknown_unknown = None

TARGET_CC = None

CC = Some("clang")

CFLAGS_wasm32-unknown-unknown = None

CFLAGS_wasm32_unknown_unknown = None

TARGET_CFLAGS = None

CFLAGS = None

CRATE_CC_NO_DEFAULTS = None

DEBUG = Some("false")

running: "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "sqlite3/wasm32-unknown-unknown/include" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_LOCALTIME_R" "-DSQLITE_OS_OTHER" "-DSQLITE_TEMP_STORE=3" "-DLONGDOUBLE_TYPE=double" "-DSQLITE_OMIT_LOCALTIME" "-o" "/private/tmp/nix-build-cargo-package-deps-0.0.1.drv-0/dummy-src/target/wasm32-unknown-unknown/release/build/libsqlite3-sys-26a45dca08af5b61/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c"

cargo:warning=clang-11: warning: argument unused during compilation: '-mmacos-version-min=11.0' [-Wunused-command-line-argument]

cargo:warning=clang-11: warning: argument unused during compilation: '-arch arm64' [-Wunused-command-line-argument]

cargo:warning=error: unknown target CPU 'apple-a13'

cargo:warning=note: valid target CPU values are: mvp, bleeding-edge, generic

exit status: 1
--- stderr
error occurred: Command "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "sqlite3/wasm32-unknown-unknown/include" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_LOCALTIME_R" "-DSQLITE_OS_OTHER" "-DSQLITE_TEMP_STORE=3" "-DLONGDOUBLE_TYPE=double" "-DSQLITE_OMIT_LOCALTIME" "-o" "/private/tmp/nix-build-cargo-package-deps-0.0.1.drv-0/dummy-src/target/wasm32-unknown-unknown/release/build/libsqlite3-sys-26a45dca08af5b61/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c" with args "clang" did not execute successfully (status code exit status: 1).
�[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m build failed, waiting for other jobs to finish...

[dpc]

I'm confused how things are supposed to work, and I'm struggling with my own project that I need to compile to wasm32.

The binaries produced by C compilers invoked by cc crate from build.rs scripts seem to target native system instead of the target, and I don't understand what is exactly responsible for passing something to them that would make them produce wasm32 code. When I tried to force it manually, they couldn't find C headers they needed, because Nix wrapper has to take care of that (AFAIU).

When I google how to do cross compilation in Nix, it always points me to https://nixos.org/guides/cross-compilation.html . But pkgsCross.<TAB> in nix repl does not show wasm32-unknown-unknown.

[ipetkov]

@paulyoung one thing I noticed is your flake is trying to build on darwin. In my experience the Apple/system toolchains present in nixpkgs tend to be a bit behind the latest and greatest versions released by Apple. Looking at the logs it seems like clang-11 is still being used (despite the updated stdenv) so I'm not sure if somehow clang is being picked up outside of the sandbox (does darwin even sandbox?) :(

If I update your flake to include "x86_64-linux" it builds just fine for me. Unfortunately I don't know enough about the state of nixpkgs and its interaction with Apple SDKs to figure out where the misconfiguration is happening

[ipetkov]

@dpc Nix's cross compiling utilities (via pkgsCross) set up a bunch of stuff which is normally super painful to configure in the C/C++ world (things like putting cross compilers in the right paths and having environment variables point to them and such). rustc is a cross-compiler by default so it makes it easier to do cross-compilation by just setting the target and cargo/rustc can usually figure stuff out on their own. Sometimes we have to explicitly set things like HOST_CC or TARGET_CC to help point things in the right direction when various crates' build scripts invoke those compilers directly

Fwiw nixpkgs does have pkgsCross.wasi32 as a proper target, though there isn't one for wasm32-unknown-unknown because very little would build for it

[dpc]

rustc is a cross-compiler by default so it makes it easier to do cross-compilation by just setting the target and cargo/rustc can usually figure stuff out on their own. Sometimes we have to explicitly set things like HOST_CC or TARGET_CC to help point things in the right direction when various crates' build scripts invoke those compilers directly

rustc is, but cc called from build.rs script's of some C-code wrapped in <crate>-sys package does not seem to be, AFAICT.

[trevyn]

Hmmmm, based on this comment: https://stackoverflow.com/questions/69039082/nixs-clang-wont-build-wasm/69124536#69124536

What if you change the CC/TARGET_CC to clang-11 instead of clang?

[dpc]

@trevyn

  TARGET = Some("wasm32-unknown-unknown")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_wasm32-unknown-unknown = None
  CC_wasm32_unknown_unknown = None
  TARGET_CC = None
  CC = Some("/nix/store/jblq3fym5skvrhj2zdn110dda20k9m8z-clang-14.0.1/bin/clang-14")
  CFLAGS_wasm32-unknown-unknown = None
  CFLAGS_wasm32_unknown_unknown = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CC_wasm32-unknown-unknown = None
  CC_wasm32_unknown_unknown = None
  TARGET_CC = None
  CC = Some("/nix/store/jblq3fym5skvrhj2zdn110dda20k9m8z-clang-14.0.1/bin/clang-14")
  CFLAGS_wasm32-unknown-unknown = None
  CFLAGS_wasm32_unknown_unknown = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  running: "/nix/store/jblq3fym5skvrhj2zdn110dda20k9m8z-clang-14.0.1/bin/clang-14" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DUSE_NUM_NONE=1" "-DUSE_FIELD_INV_BUILTIN=1" "-DUSE_SCALAR_INV_BUILTIN=1" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-DENABLE_MODULE_RECOVERY=1" "-o" "/home/dpc/lab/webimint/webimint/target/wasm32-unknown-unknown/debug/build/secp256k1-sys-21a9f49c57d0d7b0/out/wasm/wasm.o" "-c" "wasm/wasm.c"
  cargo:warning=wasm/wasm.c:1:10: fatal error: 'stddef.h' file not found
  cargo:warning=#include <stddef.h>
  cargo:warning=         ^~~~~~~~~~
  cargo:warning=1 error generated.
  exit status: 1

If you use Nix-unwrapped (export "CC"="${pkgs.llvmPackages_14.clang-unwrapped}/bin/clang-14"), then the compiler can't find header files scattered all around the /nix/store. All clang versions fail the same way.

[dpc]

I have figured out compilation for our project for Android and wasm32: https://github.com/fedimint/fedimint/blob/9fb4a608d7f3a5cd1d9775a3178fb0098c10d449/flake.nix#L50

[paulyoung]

@dpc can you share anything about how you figured this out?

I'm wondering if I can do anything similar to determine exactly what I need.

[dpc]

The general idea is to just use unwrapped compilers and set all the C flags that cc crate uses and .cargo/config.toml for given platforms so that C compilation and linking are successful. Quite a bit of time and effort to figure it out, but it works.

[paulyoung]

@dpc how did you figure that out exactly?

I tried using an unwrapped compiler and ran into the exact issue you described above with stddef.h.

[paulyoung]

I wonder how relevant this is these days (from rustwasm/team#291)

wasm32-unknown-unknown works in combination with clang-8 only(probably a bug), not before and not later(doesn't work on clang-9 and clang-10) rust-lang/cc-rs#378

[dpc]

@dpc how did you figure that out exactly?

Lots of googling, reading and trying things out.

I wonder how relevant this is these days (from rustwasm/team#291)

We use clang_14 or just clang for everything and wasm32-unknown-unknown works so it doesn't seem the case (anymore?).

https://github.com/fedimint/fedimint/actions/runs/3352395658