.%*. .-.
.%@@@+. .--=%@@@-
=@@@@@@- :--+@@@@@@@@@*
*@%@@@@@%: :--=#@@@@@@@@@@@@#@@+
@@::%@@@@@@-: :::-=%@@@@@@@@@@@@@@%#*- :@*
.@@ =@@@@@@@@@+-:::::::-=*@@@@@@@@@@@@@@@@@@@%##- @@:
#@# +%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%%#*: -@@
#@- *%@@@@@@@@@@@@@@@@@%%%#= %@=
@@: -=+++== .. @@:
@@ . -@@@ +@@
#@% =@@@@- +@@@@# *+ %@#
#@- -@@@@@@@. +@@@@@% .%@@@@= @@:
@@: +@@@@@@@@@ +@+ +@@@ %@@@@@@@@@ -@@
@@ *@%@* :@@@ *@+ -@@@ %@@@@@@@@@@@# @@@
#@% %@ @% %@@ *@* .@@@ =@@@*@@ :@@@@ @@:
%@:## *@+ :@@ :@@ @@@ %@@: @# .@@@ .@@
@@:. @@: .@@ @@ @@@ %@: @@ @@% @@%
@@ -@@. =@@ @@. @@@ %@. #@- @@# @@.
%@% +@@. @@. %@- @@@ #@ @@. :@@ -@@
%@: +@@+ .@% #@= @@@.@ *@@ +@+ @@*
@@: +@@%. .%% *@+ @@@ %@@ -@# @@
@@ :@@@@@@% :@@ @@% %@@ @# *@@
%@@ #@@@@* @@ @@% %@@. %@. @@:
@@: ===: @@. @@% %@@: =@%. .@@
@@. #@= @@% %@@%+%@#. @@%
@@ +@* @@# -@@@@@* .@@
%@@ :@@ @@# %@@#: =@@
@@: .@@ @@# : @@=
@@. @@. @@# :@@
@@ #@= @@% %@%
#@@ +@+ #@% .@@
@@: =@# +@% =@%
@@ :@% -@@ @@+
@@ .@@ -@: -@@
-@@ @% .: *@#
@@* # @@.
@@ =@@
.@@ @@*
.@@ .@@
*@@ =@#
@@* @@*
@@ -@@
.@@ +@*
.@@ @@-
-@@ =@@
%@% #@*
@@ @@:
@@ +@@
.@@ %@+
:@% @@.
:@% *@%
-@. %@=
%@ @@.
@@ *@%
@@ %@=
@% @@:
:@% +@@
:@* =+- #@+
:@: .%@@@. @@-
-@ =@@@@@# :@@
=@ *@#.=@@% #@%
*@ #@- -@@% %@-
@@ #@: :@@% @@:
@@ #% .@@% .#@@* -@@
@% #% .@@% +@@@@@- %@%
@# %@ :@@% .@@@@@@@@ %@:
@# %@ :@@% +@@% .@@@@ @@:
@# %@ :@@% #@@: :@@@ -@@
@# @@. .@@@ .@@% .@@@= %@%
@# @@. .@@@: %@@- @@@% %@:
@# @@ @@@@ %@@@ *@@@ @@:
@# .@@ @@@@@@@@@@= :@@@ @@
* @@@ :@@@@@@@@. :@@@ *@@
.@@@@@% -@@@@@. .@@@ @@=
@@@@@: .. @@@ @@.
-@%: @@@ @@.
@@@ @@
-@@ =@@
.@@ @@+
.@@:@@.
@@#@-
@@@.
+@-
-
mediocregopher's lil web corner
I spent some time recently figuring out how to cross compile a non-trivial rust project as a static binary to various operating systems and architectures. As is often the case with nix it was a frustrating experience of trying to find the exact perfect combination of inputs to make things come together.
This nix flake is the ultimate result of my efforts.
This post is going to be a walkthrough of that nix flake, both so that others might avoid my pain and also as a victory lap. Keep in mind that life in nix-land changes rapidly, so while the flake I've just linked will always work (if paired with the flake.lock file), the tooling involved will evolve over time, leaving this guide obsolete at some point in the (hopefully) near future.
inputs = {
fenix.url = "github:nix-community/fenix";
naersk.url = "github:nix-community/naersk/master";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11";
};
Obviously nixpkgs will be needed, because it always is.
fenix is a flake which provides a rust toolchain of any arbitrary version. There are other nix projects which perform a similar function, such as the mozilla rust-overlay and oxalica's fork of it. I chose fenix because it was used in a few different example projects which I was referencing, and I thought the difference might matter. In the end I think the other rust toolchain providers could probably be made to work just as well.
naersk (which probably stands for something, but I can't figure out what) takes a rust toolchain and a rust cargo project and does the hard work of actually building the damn thing. This includes pulling in and building the project's dependencies using the Cargo.lock file.
Cross-compilation deals with two different systems: the "build system" describes the system (CPU architecture+OS+stdlib) which is compiling the code, while "target system" describes the system which will run the binary. Juggling these is important, which leads to the next part of the flake:
buildTargets = {
"x86_64-linux" = {
crossSystemConfig = "x86_64-unknown-linux-musl";
rustTarget = "x86_64-unknown-linux-musl";
};
"i686-linux" = {
crossSystemConfig = "i686-unknown-linux-musl";
rustTarget = "i686-unknown-linux-musl";
};
"aarch64-linux" = {
crossSystemConfig = "aarch64-unknown-linux-musl";
rustTarget = "aarch64-unknown-linux-musl";
};
# Old Raspberry Pi's
"armv6l-linux" = {
crossSystemConfig = "armv6l-unknown-linux-musleabihf";
rustTarget = "arm-unknown-linux-musleabihf";
};
"x86_64-windows" = {
crossSystemConfig = "x86_64-w64-mingw32";
rustTarget = "x86_64-pc-windows-gnu";
makeBuildPackageAttrs = pkgsCross: {
depsBuildBuild = [
pkgsCross.stdenv.cc
pkgsCross.windows.pthreads
];
};
};
};
nix flakes don't allow for accepting any kind of input values, so it's not possible to pass in a "target system" variable from the outside to have it build. Instead you must enumerate all supported target systems in the flake itself. This attribute set is what that is. Each supported target system is listed as a pair which looks like <arch>-<os>, and maps to various attributes which are specific to that system.
crossSystemConfig is the platform string of the build target which nixpkgs uses. The supported possible values are not easy to find; the best method I found for doing so was perusing the config variables in the following file within nixpkgs:
rustTarget is the platform string of the build target which rust uses. It is often equivalent to crossSystemConfig, but not always. Supported values for this are easier to find, they are listed in here:
makeBuildPackageAttrs is an optional function which can be used to provide more configuration to the derivation which ultimately builds the project (ie naersk). This was needed because the windows build required some extra build-stage inputs, notably a special pthreads.
As can be seen, my project currently supports cross-compiling to both 32 and 64-bit linux on both intel and ARM processors, as well as 64-bit intel windows.
Following the buildTargets definition there are a couple of utility functions which you might find interesting, but which are documented in the source so I won't spend much time on them here. eachCrossSystem is important because it is used to generate the actual output of the flake, such that every cross-compilable target is listed as a separate target in the output package tree:
~/src/micropelago/domani :: nix flake show
git+file://...
└───packages
├───aarch64-linux
│ ├───cross-aarch64-linux: package 'domani-0.1.0'
│ ├───cross-armv6l-linux: package 'domani-0.1.0'
│ ├───cross-i686-linux: package 'domani-0.1.0'
│ ├───cross-x86_64-linux: package 'domani-0.1.0'
│ ├───cross-x86_64-windows: package 'domani-0.1.0'
│ └───default: package 'domani-0.1.0'
├───armv6l-linux
│ ├───cross-aarch64-linux: package 'domani-0.1.0'
│ ├───cross-armv6l-linux: package 'domani-0.1.0'
│ ├───cross-i686-linux: package 'domani-0.1.0'
│ ├───cross-x86_64-linux: package 'domani-0.1.0'
│ ├───cross-x86_64-windows: package 'domani-0.1.0'
│ └───default: package 'domani-0.1.0'
...
My nix flake populates the outputs.packages using eachCrossSystem, which in turn calls a callback for every build/target system pair. That callback then starts off by generating an appropriate rust toolchain for the pair:
fenixPkgs = fenix.packages.${buildSystem};
mkToolchain = fenixPkgs: fenixPkgs.toolchainOf {
channel = "nightly";
date = "2023-07-23";
sha256 = "sha256-LU4C/i+maIOqBZagUaXpFyWZyOVfQ3Ah5/JTz7v6CG4=";
};
toolchain = fenixPkgs.combine [
(mkToolchain fenixPkgs).rustc
(mkToolchain fenixPkgs).cargo
(mkToolchain fenixPkgs.targets.${rustTarget}).rust-std
];
As noted in the source, I would rather use fenix's fromToolchainFile here, so that I can keep all toolchain configuration in one place and still support folks who don't wanna use nix but do wanna help with development, but alas it's bugged (or I don't understand how to use it).
In any case, the important thing here is that the toolchain which I'm generating is composed of three parts, all generated from the same nightly channel version: rustc, cargo, and the rust standard library. The tricky bit here is that the standard library must be compiled for the target system, not the build system like rustc and cargo. This makes sense in hindsight: rustc and cargo will be running on the build system, but the standard library is compiled into the resulting binary, which needs to run on the target system. Obvious as it seems now, this distinction took me a while to sort out, and even longer to make sure everything was coming from the same toolchain version (a detail neglected by every guide I found online).
Finally the fun stuff:
naersk-lib = pkgs.callPackage naersk {
cargo = toolchain;
rustc = toolchain;
};
Simple enough, initialize naersk using the toolchain we generated using fenix.
naersk-lib.buildPackage (buildPackageAttrs // rec {
src = ./.;
strictDeps = true;
doCheck = false;
OPENSSL_STATIC = "1";
OPENSSL_LIB_DIR = "${pkgsCross.pkgsStatic.openssl.out}/lib";
OPENSSL_INCLUDE_DIR = "${pkgsCross.pkgsStatic.openssl.dev}/include";
# Required because ring crate is special. This also seems to have
# fixed some issues with the x86_64-windows cross-compile :shrug:
TARGET_CC = "${pkgsCross.stdenv.cc}/bin/${pkgsCross.stdenv.cc.targetPrefix}cc";
CARGO_BUILD_TARGET = rustTarget;
CARGO_BUILD_RUSTFLAGS = [
"-C" "target-feature=+crt-static"
# -latomic is required to build openssl-sys for armv6l-linux, but
# it doesn't seem to hurt any other builds.
"-C" "link-args=-static -latomic"
# https://github.com/rust-lang/cargo/issues/4133
"-C" "linker=${TARGET_CC}"
];
})
Now it gets interesting. buildPackageAttrs are any extra attributes for the compilation derivation that may have been specified in buildTargets, and are simply merged into the default attributes.
strictDeps I don't think is really necessary. As best I understand, it ensures that dependencies are passed into the derivation correctly for cross-compilation, but since we're able to cross-compile obviously they are.
doCheck tells naersk to not run tests. Since the resulting binary is for a different system than the one doing the building it's not really feasible to run tests (without using an emulator anyway).
The OPENSSL_ lines are specifically to support statically compiling the openssl rust crate.
The TARGET_CC line is a bit of black magic, but seems to be required because some crates compile code as a separate build step for macros (I'm not super familiar with rust macros, just hand-waving here). For my project the ring crate was the first culprit I hit when compiling, but this line also seems to have fixed some other issues seemingly unrelated to macros when cross-compiling to windows, so who knows. What the line actually does is set TARGET_CC to the path of a C compiler which will cross-compile to the target system.
CARGO_BUILD_TARGET is passed straight through to cargo, and tells cargo what system is being built for.
CARGO_BUILD_RUSTFLAGS are a set of flags which cargo will pass down to rustc. -C flags specifically are codegen options, which are documented at:
"-C" "target-feature=+crt-static" instructs rustc to compile targets statically. Similarly the -static flag which is passed in the link-args codegen option serves that same purpose. link-args are passed to the underlying linker, which is defined in the third -C instance to be the TARGET_CC binary we specified earlier. There is an open cargo issue to make cargo perform this part automatically, so maybe this won't always be needed.
If this seems underwhelming, given how crazy it actually is to be able to do something like this, then good! It took a lot of time to first get things working, then to boil out all the extra config lines and flags which I had added but which weren't actually needed, and to divine what I think is the minimal working flake.
And that's it! To compile the project normally it's enough to just do:
nix build
and then to cross-compile to another architecture:
nix build '.#cross-aarch64-linux'
Cross-compiling will take a long time, as it will in all likely hood require rebuilding the entire C toolchain, including C compiler, stdlib, rustc+cargo, and all dependencies, specially for that target system. This means it will also take a few gigs of storage out of your /nix/store as well. If that sounds dubiously worth it, just ask yourself: would I rather get this compiling on a real raspberry pi?
Hopefully this is useful! If you have any comments or questions about it feel free to email me at the address listed on the homepage.
Published 2023-11-27
This site can also be accessed via the gemini protocol: gemini://mediocregopher.com/