The road to Nix, a functional package manager to rule them all

pedro · May 10, 2019, 1:23pm

@yalu I’ve created a status-react branch called chore/nix-gc-roots that adds a Makefile target to prevent garbage collection of status-react. Feel free to give it a try and then call nix-collect-garbage -d to see if it works (it does for me).

frank · May 10, 2019, 10:30pm

i’ve changed CN cache to be first.
e.g.
substituters = https://nix-cache-cn.status.im/ https://nix-cache.status.im/ https://cache.nixos.org
i’m trying branch nix-cn-cache, it shows it will still access https://cache.nixos.org randomly which is the slow part. is this behavior correct?

frank · May 10, 2019, 11:03pm

failed at last.

pedro · May 11, 2019, 11:17pm

I guess it would be best for you to remove the slow servers from your list. Even a single server should be enough and will probably end up being faster in your case.

yalu · May 13, 2019, 10:11am

@pedro Awesome!!! Works!
feels so fresh and so clean
Thank you!!!

pedro · May 13, 2019, 11:10am

Great to hear that! Merged to develop.

pedro · May 13, 2019, 12:14pm

Daily progress report:

Investigate work required to get to reproducible builds (e.g. one step is to add SHA values to existing canary build which always builds same historic commit, this way we’ll know when an unknown external factor - like a download - has caused the artifacts to change).

pedro · May 13, 2019, 2:12pm

Great news, I think! As for reproducible Android builds, with the Nix effort, it looks like the only difference right now between 2 apks of separate CI builds is in index.android.bundle and specifically in how the calls to the ClojureScript logging library (timbre) are transpiled. If we fix this one issue, we’ll have bitwise identical Android builds:

from what I see here https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre.cljx#L370, the changing parameters correspond to ?file and callsite-id. callsite-id is computed from ?file in https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre.cljx#L558 and the unstable filename seems to be the reason why it is also changing.

Update: found out where the temp path is being created (in Leiningen): https://github.com/technomancy/leiningen/blob/f7e1adad6ff5137d6ea56bc429c3b620c6f84128/leiningen-core/src/leiningen/core/eval.clj#L238

pedro · May 13, 2019, 9:32pm

A good video by Sander van der Burg about replacing NPM with Nix, with an introduction to the Nix ecosystem at the beginning: https://www.youtube.com/watch?v=Rh-CSJL1Q-U

jarradhope · May 14, 2019, 2:39am

incredible job great work @pedro

pedro · May 14, 2019, 8:59am

I’ll be tracking the work/discussion on Android reproducible builds in Reproducible Android builds · Issue #8203 · status-im/status-mobile · GitHub

pedro · May 14, 2019, 4:28pm

Came across this post by the Pinterest engineering team about how they integrated Nix and Buildkite to end the duality between dev and CI machine setup: Continuous integration for iOS with Nix and Buildkite | by Pinterest Engineering | Pinterest Engineering Blog | Medium

Interesting to see the very similar path taken, namely with Fastlane and Bazel.

pedro · May 16, 2019, 11:15am

Daily progress update:

Having React Native being built through npm is causing build paths to get pulled in to the lib*jni.so modules. This is done by RN dependencies, so a simple commit in the RN fork won’t fix it. I’m going to try starting with a sample RN project and attempt to build RN from the ground up with Nix. That should enable us to have reproducible build paths, as well as a myriad other advantages, beginning with improved setup times (no need to compile boost, for example).

Update: I’ve set up a public Git repo where I’m working on a Hello Word Nix-based React Native app here: https://github.com/PombeirP/nix-react-native-test. I was able to successfully build it and deploy to Android, so clearly something else is the problem in status-react, most likely one (or several) of the RN modules that we import.

pedro · May 16, 2019, 9:09pm

In view of the goal of migrating from yarn install to yarn2nix (which should allow us to get stable paths inside RN dependencies), I think it is helpful to have an overview of the whole process happening when you run TARGET_OS=android make shell for the first time:

Overview of status-react setup

make shell is called
1. Pure dependencies are setup by nix-shell
  1. node
  2. yarn
  3. Android NDK/SDK, etc.
2. Impure dependencies are setup through prepare-for-platform.sh
  1. Create symlinks for npm/yarn pointing to mobile_files
  2. yarn install --frozen-lockfile
    1. react-native and react-native-* installed along with all other dependencies in package.json
  3. gradlew react-native-android:installArchives
    - NOTE: This is the point at which previous attempts with yarn2nix have failed, since there is a local build of the react-native fork happening in Gradle.
    - node_modules/react-native/android/com/facebook/react/react-native/x.xx.x is
      created (referenced as a maven repo in android/build.gradle). QUESTION: At which point exactly is it created?
    - node_modules/react-native/ReactAndroid/build is created (sub-folders: docs, downloads, generated, intermediates, libs, outputs, poms, react-ndk, third-party-ndk, tmp)
    1. android/app/settings.gradle
      - Configure project :app
    2. android/app/build.gradle
      1. apply from: project(':react-native-config')
      2. getVersionCode
      3. Configure projects
        
        react-native-background-timer
        
        react-native-status
        
        react-native-webview
        
        etc…
      4. Tasks from :react-native-android:
        
        androidJavadoc, androidJavadocJar, androidSourcesJar, preBuild, preReleaseBuild, compileReleaseAidl, compileReleaseRenderscript,
        
        checkReleaseManifest, generateReleaseBuildConfig, generateReleaseResValues, generateReleaseResources, packageReleaseResources, processReleaseManifest, generateReleaseRFile
        
        prepareLintJar, generateReleaseSources, javaPreCompileRelease
        
        createNativeDepsDirectories
        
        Boost
        
        downloadBoost, prepareBoost
        
        DoubleConversion
        
        downloadDoubleConversion, prepareDoubleConversion
        
        Folly
        
        downloadFolly, prepareFolly
        
        Glog
        
        downloadGlog, prepareGlog
        
        JSC
        
        downloadJSC, prepareJSC
        
        buildReactNdkLib
        
        make builds node_modules/react-native/ReactAndroid/src/main/jni/react/jni for 4 platforms
        
        packageReactNdkLibs
        
        compileReleaseJavaWithJavac, extractReleaseAnnotations, mergeReleaseConsumerProguardFiles, mergeReleaseShaders, compileReleaseShaders, generateReleaseAssets, packageReleaseAssets, packageReleaseRenderscript, processReleaseJavaRes, transformResourcesWithMergeJavaResForRelease, transformClassesAndResourcesWithSyncLibJarsForRelease, compileReleaseNdk, mergeReleaseJniLibFolders, transformNativeLibsWithMergeJniLibsForRelease, transformNativeLibsWithStripDebugSymbolForRelease, transformNativeLibsWithSyncJniLibsForRelease, bundleReleaseAar, signArchives
        
        installArchives
3. Pure shell is made available

igor · May 17, 2019, 8:57am

wow, we might want to have this somewhere in the docs also!

pedro · May 17, 2019, 9:24am

I’d agree, however I’m hoping to make this information irrelevant (at least in it’s current form) soon

pedro · May 17, 2019, 2:56pm

Snapshot of what gradlew react-native-android:installArchives builds:

pedro · May 17, 2019, 4:24pm

Daily progress update:

Got a first successful gradle build of our RN fork, even though it’s not anywhere near perfect (some parts of it need to be impure - as commands such as installRelease will attempt to write inside node_modules/react-native -, and need to find a way to keep the rest pure).

yenda · May 18, 2019, 7:45pm

@pedro I only skimmed through so it might be offtopic, I came accross GitHub - callstack/haul: Haul is a command line tool for developing React Native apps, powered by Webpack as an alternative react-native packager, maybe it can help you in your quest?

pedro · May 20, 2019, 9:59am

Thanks @yenda, that could be a promising way out (having glanced at the source code for a bit). I’m trying it on my sample RN project but ran into an error, looking into it:

./App.js
Module not found: Error: Can't resolve 'react-hot-loader' in '/home/pedro/src/github.com/status-im/nix-react-native-test'
resolve 'react-hot-loader' in '/home/pedro/src/github.com/status-im/nix-react-native-test'
 Parsed request is a module
 using description file: /home/pedro/src/github.com/status-im/nix-react-native-test/package.json (relative path: .)
   Field 'browser' doesn't contain a valid alias configuration

UPDATE: Managed to fix that. Apparently react-dom and react-hot-loader were missing from devDependencies.