As some of you might be aware from the Dependency Audit v1 thread and #status-core, there’s a PR in review to ditch most of the package managers we use in favor of Nix (e.g. Homebrew, apt-get and npm install -g). While I still have your attention , let me say that this is a high-impact PR, and therefore we need as many devs as possible to test it and report any issues that they may find, and not wait before it reaches the develop branch before trying it out. I’m planning to merge the PR by the end of next week (Feb 22), and from then on, you’ll need to call make shell to enter this special child shell to work on status-react.
Why migrate to Nix?
We currently have a mess of different package managers, each with different stacks, guarantees in terms of reproducible builds, and impact on the user’s system. Nix has been coming up as a potential candidate several times over the last months (from people like @cammellos and @arnetheduck and a few others). This migration will allow us to focus on a single package manager across Linux and macOS, while at the same time:
gaining a predictable environment (i.e. we know exactly which tool versions we get as they’re identified by a cryptographic hash which comprises their dependencies and configuration);
better management of downloads, avoiding ad-hoc scripts and getting us garbage-collection for free;
having much less of a footprint/side effects on the user’s environment (we’ll build on Nix by opening up a child shell that’s fully configured for status-react, and Nix packages are kept separate from the user’s normal environment. We won’t be installing, upgrading or downgrading tools against the devs wishes);
achieving much better consistency between the desktop dev environment and our CI server (we’ll use the NixOS Docker image with the same script as for desktop dev environment);
What next?
In this PR, for the sake of keeping it as simple as possible, we haven’t yet made changes to Jenkins. That will be done in a separate follow-up PR;
A few larger dependencies are still installed outside of Nix, namely Qt for Linux/macOS, Android SDK and NDK. Qt certainly sticks out as a sore thumb, since it relies on the user to download it and correctly install it on his/her system. We can do much better than that by following the example of the Qt for Windows package which is delivered by Conan;
Update: the current PR will also include Qt and NDK, as well as some bundling tools we’re using for Desktop;
Conan itself could eventually be replaced by Nix, if we port the Qt for Windows (and associated cross-compiler toolchain) to Nix;
Please let me know of any issues you come across while testing this, and be prepared to send me the log output.
Just a heads-up, I’m also working on a Nix build for Nimbus and our toy project Stratus.
As part of this effort, I’m already close to producing cross-compiled Qt-based binaries for Windows and macOS from a Linux host. Sticking to using Nix on Linux will give us fully reproducible builds for all platforms that can be verified by end users to match the published source code. So far, I’ve started with an existing cross-compilation setup discovered here, but I might switch to the more standard approach described in the Nix manual. The current targets include Linux ARMv6, macOS and Windows, but we’ll also need to add iOS, Android, Linux ARMv7, Linux RISC-V and perhaps some more exotic platforms such as FreeRTOS.
Once this effort is complete, we hope to reuse the existing CI infrastructure of Status to test our builds, because at the moment we cannot track how our performance is changing over time (Travis won’t use the same hardware every time the tests are run).
So, please let me know when adding additional Nix builds to our CI becomes easy enough.
That’s awesome, thanks for the heads up @zahary! Looking forward to getting that into status-react so we can drop that prerequisite and streamline the existing scripts.
I’ve been running NixOS as my desktop daily driver for awhile, pretty elegant package management, I suspect the main issue we’ll have with this is adding and maintenance of packages that don’t exist or weren’t created with our intended use, so probably alot of upstream PR’s.
What packages or package managers are not covered by this approach?
Currently the big one is the Android SDK. We have a couple others that we still need to bring in, such as Qt (making good progress) and the NDK (we’ll bring it in once we solve the SDK one). So in the end I’d say it’s working out pretty well for status-react.
Currently we’re bringing in those package managers through Nix. Do you mean in the same sense as we’re replacing npm install -g with node2nix, which wraps a package.json in a Nix expression?
For the sake of transparency, here’s a report on my current progress with Nix:
Can now build and run the app on Linux based on Nix Qt 5.12 package (qt.full);
Currently working on building app on macOS. Today I could build it by leapfrogging my way around the gauntlet, but here are the issues that remain to be fixed:
qtkeychain tries to copy its output .qm modules to $QT_PATH/gcc_64/translations, which fails for lack of root access (since the Nix store is supposed to be read-only);
Current hack: build again and cmake won’t try to copy files again;
Solution: pass a local folder to qtkeychain CMake script;
CMake is not playing nice with Nix’s clang++ compiler for some reason (cmake complains that it can’t determine the features for the CXX compiler for subdirectories like reportApp - the root project seems to work just fine).
Current hack: I could get around it by setting the CXX compiler to /usr/bin/g++ just to see what other problems laid ahead. Also needed to change CMAKE_OSX_DEPLOYMENT_TARGET from 10.9 to 10.10 (I’m on Mojave).
One possible compromise for the compiler issue could be to use the host system’s C and CXX compilers for the time being (it won’t be any worse than the current situation, just not an improvement in that specific area);
Solution: move the PROJECT section of the root CMake project further below in the file and include the C language target. Rebuild Nix macOS SDK to target 10.9;
macdeployqt is not picking up on all dependencies from Nix store.
Solution: managed to write some bash glue code that’ll locate and provide 2 critical Qt libraries to macdeployqt, so we finally have a (mostly) working app on macOS built on Nix.
Good news! I was able to build the QtWebView module on macOS and will be contributing a PR upstream for that. That was the last major roadblock to a 100% automated Nix install for status-react, so we should be able to merge to develop soon.
tested build process on Mac and Arch Linux (native and Windows cross-compilation);
tested and debugged Nix caching issues;
did the necessary work to integrate qtkeychain Nix package in react-native-keychain to sidestep a nagging issue where it either builds on my machine or on CI.
noticed that some libraries pulled by macdeployqt from Nix deep in the dependency chain didn’t have their rpaths updated, so I’m working on a script to walk the dependencies and update the library links to point to the macOS package;
test again Nix cache on macOS after Jakub’s fixes;
work on migrating our Jenkins Linux build to rely on Nix;
create Nix expressions for android NDK, appimagetool, linuxdeployqt and NSIS to further reduce wget downloads/dependencies on apt-get;
continue working with @jakubgs to fix Linux build on CI server;
Nix targets macOS SDK 10.10. Need to get it to target 10.9, but build is failing when running tests of Go compiler.
Update: looks like upgrading the Go compiler package from 1.10.8 to 1.11.5 allows the build to proceed, although there are some warnings that some dependencies were built for 10.10 (?)
Note: I’m really enjoying how responsive the nixpkgs repo maintainers are. The qtwebview PR was merged in a couple of days, after having received thoughtful input. It bodes well for Nix.
The time taken by unattended make setup to set up a clean machine is at exactly 7 minutes on a laptop (includes downloading the 1GB Android NDK), which I’m sure is much less than the pre-Nix branches. We can make that process much faster in the future by splitting up the Nix expressions by area (e.g. not download Qt, Conan, appimagetool, etc. unless we try to build Desktop).
Following that, running making make prepare-android && make release-android takes under 7m30s, so we’re looking at building and deploying the Status Android app in under 15 minutes on a new system.
That’s super impressive! right now the issues aren’t about speed of downloads, but the time you waste jumping through the hoops to make it work. And it is literally hours.
Fix issue with qt5keychain build failing on CI but not on local machine;
Fix remaining issues with Linux CI build;
Implement target-dependent installing of Nix dependencies (i.e. only install exactly what is required for a given target platform).
pedro@pedros-thinkpad-x1:~/src/github.com/status-im/status-react$ nix-shell --argstr target-os 'windows'
these derivations will be built:
/nix/store/v40vb0k2cjd46hhr8469w81fqb8n2irk-env.drv
building '/nix/store/v40vb0k2cjd46hhr8469w81fqb8n2irk-env.drv'...
created 328 symlinks in user environment
[nix-shell:~/src/github.com/status-im/status-react]$ which linuxdeployqt.AppImage
[nix-shell:~/src/github.com/status-im/status-react]$ exit
exit
pedro@pedros-thinkpad-x1:~/src/github.com/status-im/status-react$ nix-shell --argstr target-os 'linux'
[nix-shell:~/src/github.com/status-im/status-react]$ which linuxdeployqt.AppImage
/nix/store/gz20grnz57hw15j55xwxw9mi4m6y4mdm-linuxdeployqt/bin/linuxdeployqt.AppImage
, let me say that this is a high-impact PR, and therefore we need as many devs as possible to test it and report any issues that they may find, and not wait before it reaches the 