The road to Nix, a functional package manager to rule them all

dentist · June 17, 2019, 7:55pm

Wonderful about Nimbus

pedro · June 18, 2019, 12:20pm

Daily progress report:

In the last couple of days, I’ve been finishing up scripts to automatically detect maven dependencies required in the Android Gradle build, and import them to Nix. Some challenges were:
- determining how to get missing dependencies that are not reported by e.g. gradle app:dependencies. I’ve opted for looping through the child projects (such as react-native-*, realm, etc) and querying for those dependencies, but that didn’t do the whole job. It was also necessary to query each individual .pom file with mvn dependency:tree to figure out the remaining dependencies. Currently, we’re not distinguishing dependencies in the context of their configuration (i.e. we could just take the ones meant for the compile configuration) so we end up with extraneous dependencies which are meant for e.g. unit testing;
- e.g. com.android.tools.build:aapt2 (which requires a platform-specific additional package not mentioned in gradle dependency reports);
- GitHub maven dependencies such as com.github.status-im:function (found out that those are provided by https://jitpack.io);

I’m hoping that once this phase will be finished, the gradle assembleRelease Nix expression will build successfully, allowing us to generate an apk that has reproducible build paths (/build/*), getting us much closer to the goal of Android reproducible builds.

pedro · June 19, 2019, 10:11am

Daily progress report:

Moved past the Maven repository issues, but left one workaround behind for the time being. Even though firebase-ads-15.0.1.pom imports firebase-analytics:[15.0.1,16.0.0), mvn :react-native-firebase:dependencies is reporting +--- com.google.firebase:firebase-analytics:16.0.3, which is clearly out of the range specified in the pom, and later fails the build. I’ve manually added the 15.0.3 package to the repository, which seems to be what we actually use.

The build has now progressed quite a bit more, and is failing at :app:mergeReleaseResources with:

java.util.concurrent.ExecutionException: com.android.builder.internal.aapt.v2.Aapt2InternalException: AAPT2 aapt2-3.3.2-5309881-linux Daemon #0: Daemon startup failed
Caused by: java.io.IOException: Cannot run program "/build/.gradle/caches/transforms-1/files-1.1/aapt2-3.3.2-5309881-linux.jar/0ee476bac04df5b90b90cf5bf4e1ceff/aapt2-3.3.2-5309881-linux/aapt2": error=2, No such file or directory
     at com.android.builder.internal.aapt.v2.Aapt2DaemonImpl.startProcess(Aapt2DaemonImpl.kt:82)
     at com.android.builder.internal.aapt.v2.Aapt2Daemon.checkStarted(Aapt2Daemon.kt:53)
     ... 8 more
Caused by: java.io.IOException: error=2, No such file or directory
     ... 10 more

vitaliy · June 20, 2019, 2:17pm

Having some weird issue with Desktop build. When running make startdev-desktop I get:

+ echo 'Building Detox.framework from /Users/vvlasov/Library/Caches/Yarn/v4/.tmp/8682eb383242d3ae44f37ccd8e906255.e77167f38aaa9591cda1b2cbeeea08aeb429c996.prepare/node_modules/detox/ios_src...'
Building Detox.framework from /Users/vvlasov/Library/Caches/Yarn/v4/.tmp/8682eb383242d3ae44f37ccd8e906255.e77167f38aaa9591cda1b2cbeeea08aeb429c996.prepare/node_modules/detox/ios_src...
+ mkdir -p /Users/vvlasov/Library/Detox/ios/73169b5508fdefdc30cf1fe418a0dea7d0538eed
+ /Users/vvlasov/Library/Caches/Yarn/v4/.tmp/8682eb383242d3ae44f37ccd8e906255.e77167f38aaa9591cda1b2cbeeea08aeb429c996.prepare/node_modules/detox/scripts/build_universal_framework.sh /Users/vvlasov/Library/Caches/Yarn/v4/.tmp/8682eb383242d3ae44f37ccd8e906255.e77167f38aaa9591cda1b2cbeeea08aeb429c996.prepare/node_modules/detox/ios_src/Detox.xcodeproj /Users/vvlasov/Library/Detox/ios/73169b5508fdefdc30cf1fe418a0dea7d0538eed
child_process.js:637
    throw err;
    ^

Error: Command failed: /Users/vvlasov/Library/Caches/Yarn/v4/.tmp/8682eb383242d3ae44f37ccd8e906255.e77167f38aaa9591cda1b2cbeeea08aeb429c996.prepare/node_modules/detox/scripts/build_framework.ios.sh
    at checkExecSyncError (child_process.js:616:11)
    at Object.execFileSync (child_process.js:634:13)
    at Object.<anonymous> (/Users/vvlasov/Library/Caches/Yarn/v4/.tmp/8682eb383242d3ae44f37ccd8e906255.e77167f38aaa9591cda1b2cbeeea08aeb429c996.prepare/node_modules/detox/scripts/postinstall.js:2:27)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)

Why would it run an ios-specific script?

pedro · June 20, 2019, 4:17pm

Is this issue nix-related? I don’t recall seeing this Detox.framework anywhere…

vitaliy · June 20, 2019, 4:20pm

Right! I just grepped the source and could not find it. Sorry, must be some local problem. Will investigate.

pedro · June 20, 2019, 4:24pm

Daily progress report:

After fixing the issues in the latest progress report, I’ve come across a build error in :app:mergeReleaseResource, with aapt2 complaining about a missing file/directory. Running with strace and looking at the sources has been inconclusive so far. However:
- taking the temp nix-build directory and creating a pure Nix shell on it, I was able to re-run the exact same gradle assembleRelease command and have it succeed. This would seem to point to environment differences.
- comparing the environment variables from both environments didn’t yield any significant differences.
- Running nix-build without --pure didn’t help though.

pedro · June 24, 2019, 2:31pm

Daily progress report:

Finally found out the issue that was causing aapt2 to mysteriously fail with ENOENT: the aapt2 being downloaded from a non-Nix source (an Internet Maven repo), it points to an dynamic library interpreter that is forbidden in a sandboxed Nix environment. The solution was to use patchelf --set-interpreter to point it to a Nix interpreter. This has allowed the gradle assemblyRelease command to run successfully to completion.
Will now clean up the existing Nix recipes from test code, and work on replacing the existing Jenkins logic with a call to the Nix build recipe.

pedro · June 25, 2019, 3:32pm

Daily progress report:

Started replacing Jenkins logic to use nix-build, as well as adding logic to clean up derivations once they’re build to avoid filling up the disk.

pedro · June 26, 2019, 1:30pm

Daily progress report:

Fighting with Groovy/Jenkins to get the Android keystore file recognized in the Nix script.
Cleaning up of branch.

pedro · June 26, 2019, 9:02pm

Daily progress report:

Managed to tame Groovy/Jenkins and get the first successful end-to-end sandboxed Nix build of the Android app!
The recent rebase seems to have introduced a new issue in the taoensso/timbre library (a single difference in the taoensso.timbre.*config* macro invocation), as well as the BuildID of a couple of the RN native libraries (libglog_init.so and libreactnativejni.so). These results were based on comparisons between builds done in 2 different CI servers.

pedro · June 27, 2019, 3:05pm

Daily progress update:

Removed the BuildID of the RN native libraries;
With the help of @yenda, fixed the Timbre library fork so that it generates reproducible macro expansions;
The last remaining file (which I had hoped would no longer have differences once we fixed the index.android.bundle issue) is resources.arsc at the root of the APK. This is a known source of non-determinism. The fact that we’re on an old version of the Google Android Gradle plugin doesn’t help either, so I’ll upgrade the plugin to at least 3.4.0 (which also implies upgrading Gradle to at least 5.1.1) and see if that helps.

pedro · June 28, 2019, 12:40pm

What an epic ending to a difficult week! Today marks the first time we have generated two binary-identical .apk files from 2 separate CI machines:

diff -s /home/pedro/Downloads/StatusIm-190628-104059-743f20-manual.apk.zip /home/pedro/Downloads/StatusIm-190628-102026-743f20-manual.apk.zip
Files /home/pedro/Downloads/StatusIm-190628-104059-743f20-manual.apk.zip and /home/pedro/Downloads/StatusIm-190628-102026-743f20-manual.apk.zip are identical

Daily progress update:

After upgrading to Gradle 5.1.1 and the Android plugin to 3.4.1, started running into build issues saying that :app:mergeReleaseResources task could not find aapt2. After some investigation, it turned out that :react-native-android:packageReleaseResources' was causing the file to be deleted from the cache because it was considered stale for some reason. It looks like the way we’re saving the ~/.gradle folder from one Nix expression to the release-android expression is not convincing Gradle, which causes it to consider them stale and trigger a rebuild. For now, I was able to get around it by disabling caching in Gradle for Nix builds.
The next step will be to guarantee identical builds between my dev machine and the CI servers, since the build environment uses a slightly different configuration at the moment (currently there are 1-byte differences in some files - such as /res/layout/abc_action_mode_close_item_material.xml - at the second-to-last byte in 5 files).
Even though we’ve reached the goal of reproducible builds, there is still a lot of work that remains to be done, namely:
- The issue with ClojureScript minification is still out there. Currently, I just disabled some aspects of minification, but the goal is to keep it at the :advanced level instead of :simple;
- I haven’t focused on ensuring that the remaining development environment works properly after these changes (e.g. that the developer is able to run make startdev-android-* successfully, and ideally reuse the same Maven local Nix repo instead of downloading dependencies to another local cache);
- Cleaning up and documentation;
- Testing other more compatible methods of reaching the same outcome (e.g. use yarn2nix instead of node2nix in order to keep compatibility with yarn.lock);
- Moving some new Nix infrastructure out of status-react, since it doesn’t need to be there (e.g. the tool to compute a local Nix Maven repository, which can be open-sourced);

pedro · July 1, 2019, 9:24am

Daily progress report:

I believe I found the reason for slightly different XML files between local build and CI build, thanks to apktool (which by the way is available as a Nix package): the different byte was in the resource ID, because the local build had an extra identifier coming from .env (DEV_BUILD=1). The solution is to accept a variant argument in release-android.nix and default to the same value as the CI server (nightly).
Now that build differences are resolved across machines, I’ll re-enable :advanced CLJS minification support in Google Closure compiler and try to get that to build deterministically.

pedro · July 1, 2019, 3:58pm

Managed to fix the minification issue once I found a compiler option for that: :stable-names true.

pedro · July 1, 2019, 4:04pm

I guess we should try publishing the app to F-Droid to see if the store is happy with the deterministic build.

pedro · July 2, 2019, 8:29am

Daily progress update:

Yesterday I tried installing the resulting app on a device, but got a loader error regarding libgojni.so. It turns out that the fix to remove the .gnu.version_d renders the library unloadable, so a different approach will need to be devised. After a preliminary investigation, this is my understanding of the situation so far:
- gomobile creates a temporary dir in the form of $TMP/gomobile-work-xxxxxxx/ where xxxxxxx is a variable hex number created by ioutil.TempDir. There are a few approaches we could use to fix this:
  - Patch the strings of the resulting library in the string table (or even remove them altogether);
  - Patch the sources of gomobile so that it uses a known directory;
- go build is leaving a build path on the string table (which you can see using readelf -V), which ends up affecting the a field in the .gnu.version_d header (which is why simply patching the string table isn’t enough). Normally this should not happen due to this logic in golang, so I’ll need to try to understand where things are going wrong.
  
  Truncated output of readelf -V:
```
Version definition section '.gnu.version_d' contains 1 entry:
  Addr: 0x0000000000015c7c  Offset: 0x015c7c  Link: 3 (.dynstr)
  000000: Rev: 1  Flags: BASE  Index: 1  Cnt: 1  Name: /build/go-build203396378/b001/exe/a.out
```

Running go build with the -x flag to print all commands, we can see the source of this path:

mkdir -p $WORK/b001/exe/
cd .
/nix/store/9642xkwmps282cy96s1sadis877dcr74-go-1.11.5/share/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -installsuffix shared -buildmode=c-shared -buildid=d6aCMiKF8dVHGgjqvwOI/Qx9Fub1et8xxoKugc2ST/ZiA8cufNSF4DIqGIOmys/d6aCMiKF8dVHGgjqvwOI -s -extld=/nix/store/61sbaavrlq3hdq7rx8m2wkcibz6qyvfj-ndk-bundle-19.2.5345600/libexec/android-sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android21-clang $WORK/b001/_pkg_.a
/nix/store/9642xkwmps282cy96s1sadis877dcr74-go-1.11.5/share/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/exe/a.out # internal
mkdir -p /build/gomobile-work/android/src/main/jniLibs/x86_64/
mv $WORK/b001/_cgo_install.h /build/gomobile-work/android/src/main/jniLibs/x86_64/libgojni.h
mv $WORK/b001/exe/a.out /build/gomobile-work/android/src/main/jniLibs/x86_64/libgojni.so

We can also see that the go compiler is leveraging the NDK’s clang compiler ability to generate reproducible paths with by passing it -fdebug-prefix-map=$WORK/b070=/tmp/go-build.

For future reference, here is where the a.out name is decided, and here is where the path gets computed.

pedro · July 3, 2019, 5:59pm

Daily progress update:

After trying several approaches to fixing non-determinism in the generation of libgojni.so, I opted for patching the Go compiler suite. Although that sounds heavy-handed, it’s a simple line of code in Nix to patch a source file in the Go compiler so that it looks for a temporary build location that we control in an environment variable we provide. If it doesn’t see that variable, it just falls back on existing behavior. I’ll be opening up a bug report at Issues · golang/go · GitHub, but for the time being, we already reap the benefits with little downside.
After the libgojni.so fix, I tested deploying the CI build to an Android phone and the app worked fine.
I conducted several tests building the same commit across different machines. We’re able to get identical apks on CI servers (where we provide the official Status signing certificate), but those won’t be exactly identical to the ones built on a developer machine, since they’ll be signed with the developer’s certificate (there’ll be differences in /META-INF/CERT.RSA /META-INF/CERT.SF /META-INF/MANIFEST.MF and /classes2.dex). This is assuming we build with the same arguments of course (notably build-number and build-type).
The rest of the day will be spent fixing CI builds for the other platforms due to the changes in this branch.

pedro · July 3, 2019, 6:02pm

Something I’ll be putting on my watch queue: Building Go with Bazel - GopherCon UK

pedro · July 4, 2019, 3:22pm

Daily progress update:

Fixed all CI builds, iOS still giving issues in fastlane;
Created a separate smaller PR with unrelated things to make upcoming PR smaller.