Nimbus on mobile

Archive Software Engineering
1

Following up on the Nimbus on mobile unconference attended by @oskarth, @andrey, @yenda, @pedro, @mamy, @kim and @eugene. Also relevant for @adam and @cammellos.

My takeaway was that this is a long-term initiative that we’d be smart to kick off soon.

Here are the notes. Please edit if I mis-captured anything, but the topline is this:

We need to kick off a working group between Nimbus & Status core, with the objective being to build a proof of concept that allows us to test Nimbus on mobile, as well as familiarize the core devs with Nimbus – and the teams with each other.

Eric and Pedro have both expressed some interest, and it sounds like nearly everyone tagged in this post is relevant.

How would we like to move forward?

5 Likes
2

Thanks Rachel. The specific action point as far as I remember was for us to setup a regular call to get the teams closer to each other, with either one owner or one from both Core and Nimbus (eth1 track, not eth2).

There was some discussion regarding clean slate vs hybrid approach in the session. Afterwards, in conversations with Kim and Jacek, there seemed to be a rough consensus that a clean slate approach makes more sense. Of course, the specifics depends on bandwidth from e.g. Core team, since it is quite a commitment. I do think it is the right thing to do though, as it’s likely to be quicker and cleaner in the end. It roughly amounts to implementing all the interfaces that the Clojure/RN app needs.

I setup a call for Monday and invited people tagged here, let me know if someone else wants to participate. Please add things to the agenda as appropriate. Talk to you then!

3

I do think it is the right thing to do though, as it’s likely to be quicker and cleaner in the end. It roughly amounts to implementing all the interfaces that the Clojure/RN app needs.

Just to clarify, does that means than any status-react call will go directly through nimbus instead of going through status-go/status-protocol-go → nimbus?

4

@cammellos The clean state means everything is reimplemented in Nim so yes.

5

It’s more of a question of process, I did not formulate my question precisely.

It seems from the above post, that they don’t want to have an intermediate moment where status-go/status-protocol-go will interact with nimbus, so nimbus will clean off replace status-go/status-protocol-go, instead of a gradual shift (which I take is the hybrid approach).

If that’s so, that’s an insane amount of work, and high risk, as it sounds like an all-or-nothing approach, @oskarth care to elaborate? How come we were not involved in this conversation, given that the people involved in the conversation you mentioned have little understanding of the whole project?

1 Like
6

The conference was really just to start a conversation between core and nimbus team and no decision was made.

Personally I was in favor of the clean state solution.

The hybrid solution as I understood it was about replacing geth by nimbus, which would therefore be used by status-protocol-go and status-go (the part with the status-react api). To me it sounded like integration tests for nimbus with components that will eventually not exist in our stack.

Now if there is a way to just use the status-protocol-go part in nimbus and integrate with status-react directly that would be an iterative approach that makes more sense to me. I did not know enough about it to be able to make such proposal, for instance I don’t know if and how status-protocol-go interacts with the db managed by status-go.

7

Waaaaait, how close are we really to starting use of Nimbus on Mobile? Few months? More than few months? A year?

Premature optimization - or in this case compatibilization(just made this word up) - sounds to me like a waste of time usually. Yes, having a situation in which only parts of Status are done in Nim and parts in Go is not perfect, but mostly from aesthetic sense. From the point of view of performance it should not be a problem since both are linked in the same way, as dynamic libraries. I’d say porting things as they come one by one is a much safer and less chaotic process.

The decision is in devs hands ultimately, but from my point of view you’re trying to bite off to much at once considering the time it took to develop packages like status-protocol-go or whisper.

4 Likes
8

Right now the biggest hurdle to being able to cleanly switch from geth to Nimbus is that we never defined an abstraction layer between status-go and geth. I fear that we’ll repeat the same mistake if we go directly to Nimbus and start using its API directly. It goes without saying that even if we go with the clean slate approach, we should learn from past mistakes and design things in such a way that we can replace the node implementation easily.

1 Like
9

As far as strategies go, there are valid arguments for both. I’d look out for a few points which apply to both clean slate and via-go approaches:

  • keep all this behind a toggle - nimbus should still be considered experimental
  • start with whisper parts (ie don’t do eth stuff - it’s not ready)
  • nim code is cheap and easy to write and throw away (think python) - specially when porting existing code, it takes very little time to reach an MVP solution - you’ll reach clean-slate faster than you think, if you try it
  • the tough problems of speccing things out is probably what took the most time in status-protocol-go - not typing the go syntax - adjust timeline expectations accordingly
  • both approaches share a lot of the nitty gritty to get it to work - getting things to compile, sharing code between languages etc etc - thus it doesn’t matter that much and the decision can be delayed
  • don’t spend time optimizing the go-nim integration, such as threading - it doesn’t really matter since it’s a temporary solution
  • when doing clean slate, it would be wise to consider if the api between app and whisper can be simplified and layered somehow - what @pedro is alluding to - at the same time, it’s worth considering whether the offered whisper and status api primitives are the right ones and if they can be simplified/optimized
3 Likes
10

As @yenda already mentioned, this discussion was only just started, some exploration is needed here as, at least for me, there are several unknowns.

Obviously, from a technical “purity” point of view, the clean slate solution sure sounds the best.
Clean slate here means to have a direct interface between Clojure/RN app and Nimbus.
I believe this would also be the end goal of the Nimbus integration, no matter which path is taken.

However, I understand the concern of the amount of work that this might require before it can actually be (partly) tested.
But, the go/nimbus integration should also not be underestimated of work.
Work that on the long run will be thrown away (and hopefully we manage to do that).

I think it was @yenda who suggested this (?), but the clean slate way does not necessarly have to be so abrubt. We could have Nimbus and geth side by side available, and switch 1 API call at a time from one to the other. Practically not losing time in work that would be thrown away (the go-nim integration), yet still gradually starting to test Nimbus implementations.

About the unknowns, one for me is the currently used interface between status-go and Clojure/RN. I understood that this is still done with json-rpc calls? But not fully (callbacks?).

11

I had missed the last two messages when replying, but I think that the clean slate approach might actually contribute to have a better abstraction layer or a cleaner (not Whisper focused) API than going for the go-nim integration at Whisper level.

12

This is what the hybrid approach would do, keeping both implementations available so we can switch from one to another to do testing.

Correct, the callbacks from status-go to status-react are done through signals.

I don’t think the clean slate approach includes keeping both geth and Nimbus, but I might be wrong. Having multiple implementations is what forces you to think about the abstraction in the first place. Doesn’t mean we can’t design the abstraction anyway when doing clean-slate, we just have to be intentional about it.

13

The amount of effort and risk for a clean slate approach is huge.

We need to be pragmatic in our decisions and don’t solely base our decision on principles, but understand the complexity and technicalities of what such move implies, precedence and the engineering culture we want to establish.

Most of the proponent of the clean-slate approach seems to me have only a very partial understanding of the amount of code that needs to ported.

Pfs,multidevice, transit, datasync,discovery,whisper, wallet,multiaccounts, mailservers and more, really all of this in one go? From previous experience I have seen these kind of approaches failing spectacularly a few times, of course that’s not to say that it can’t be done, just that’s is really hard.

It is also worth mentioning that nim is a very young language, and I believe it’s fair to say that adds to the risk (as what happens if a library is not available for nim? we will have to re-implement in it), but that’s something that I don’t really know, as I am not familiar with the nim ecosystem, and I have nothing but respect for the language.

Not to mention that status-go will be a moving target, as we will have to likely make changes to it, which will complicate things.

Historically we have not been good at big-bang/clean slate approaches, many of the initiatives where this was tried have had mixed results (status-sdk, initial status-go move, new-protocol).

We had many discussions about this, and the overall company culture is that we prefer iterative approaches, for reason that should be well understood by now. Divide and conquer, integrate little.

Start with the minimum amount that can be verified, and move on from there (in this case it would be to integrate low level whisper api, make sure it works, and take it from there for example). There is of course a cost involved, compared to a clean slate, but risk is much better managed.

Basically all I am saying is that the clean slate approach should have rock solid arguments for it, and champions who understand the complexity of the effort, as we had many previous discussion on the matter, and both precedence and company culture are not inline with this approach, not to mention the high risk involved with this.

5 Likes
14

I believe there might be some confusion in the naming so to be absolutely clear about what we are talking:

  • Clean slate approach: No Nimbus - Go integration. Clojure/RN interacts directly with Nimbus. Thus at the level of status-react API.
  • Hybrid approach: There is a Nimbus - Go integration. Go sits in-between Nimbus and Clojure/RN. Where exactly this integration happens is to decide. But it would probably start as low as e.g. Whisper API level and gradually increase.

However, I think that the clean slate approach is not necessarily restricted to not having any intermediate solutions with both Nimbus & status-go / geth.

Integration will just be done at a different layer. And thus typically, the split up will be more difficult (and exist out of bigger blocks) for the clean slate approach.
But that too will depend on the current and future API and abstraction that is made.

I mean, it all depends on how intermingled these parts that you mention such as Pfs, multidevice, transit, datasync, whisper, etc. are.
It is true that I only have a very partial understanding of this, so perhaps the clean slate approach with Nimbus & status-go / geth split up doesn’t make sense in practice, and in that case the step(s) to take might be too big/risky.

So lets figure this out by at minimum having a look into the current APIs and have them documented (this is necessary work anyhow right?). We can begin with a simple list of the calls that are now done from status-react.

Would this be a good place to start?

I think perhaps it could (I didn’t mean to restrict it to Geth, but status-go/status-go-protocol in general), but you are probably a better judge on that.
As stated above, further investigation on the current API is needed to see if it makes sense to me.

Yes, that’s what I was trying to say. Working on a second implementation could improve the structuring of the API.
But the hybrid approach might not help as much here, as it might gradually make you re-implement the already taken path in Go. (I’m talking about the status-react API here, not the low level Whisper API)

2 Likes
15

Yes, seems like there is some confusion on what exactly hybrid and clean slate means, the way I see it:

The end result in both cases is identical (only nimbus will be there, no status-go/status-protocol-go), it’s only about the process, i.e how to get there.

  • Clean slate approach: No Nimbus - Go integration. Clojure/RN interacts directly with Nimbus. Thus at the level of status-react API.
  • Hybrid approach: There is a Nimbus - Go integration. Go sits in-between Nimbus and Clojure/RN. Where exactly this integration happens is to decide. But it would probably start as low as e.g. Whisper API level and gradually increase.

This sounds exactly how I understood it, I am advocating for the hybrid approach, slowly chocking status-go, and eventually get rid of it completely, so when you say:

However, I think that the clean slate approach is not necessarily restricted to not having any intermediate solutions with both Nimbus & status-go / geth.

That sounds like the hybrid approach to me, so probably we are not far off in views, but maybe there are some fundamental differences in the way we see it.
Whether we do it vertically (one endpoint is fully using nimbus, while the rest will be using status-go), or horizontally (one endpoint calls status-go, which in turns uses nimbus), I guess depends on what we will be tackling and in which order, both seems reasonable approaches and we might mix and match.
The end-result in both approaches will be the same (no status-go, only nimbus), which I believe we are all in agreement and it’s the company strategy.

Would this be a good place to start?

Yes, plus https://github.com/status-im/status-react/blob/develop/src/status_im/native_module/core.cljs which are some endpoint that are only exported natively, on top of that there are the asynchronous signals status-mobile/src/status_im/signals/core.cljs at ebc2ff04cd9d27269729e3208b1fdeac46bd2cb3 · status-im/status-mobile · GitHub which the app is reacting to.

1 Like
16

Me after writing this post

image

(+1 for hybrid approach)

6 Likes
17

We’ll be using this thread to share regular updates on the progress of moving the app to rely on Nimbus. Today we had our first status-nim meeting (notes available here, thanks @mamy!) and the next will be tentatively scheduled for Oct 14th.

Next step will be to do some janitorial work on status-protocol-go to ensure that geth types/implementation details do not leak out of the very lowest layers, making it simpler to later implement a Nimbus transport layer. I’ll be scheduling a quick meeting with @cammellos and @adam to go over that.

cc @kim

18

Coming a bit late to the discussion :wink:

Very good point.

geth sucks in being a library. I guess it has never been a goal for them to provide a Go library for the Ethereum stack but rather provide a command line Ethereum client.

Not sure how nimbus is built but in geth there are plenty of places where we don’t have as much control as we needed to efficiently implement business logic.

The end result in both cases is identical (only nimbus will be there, no status-go/status-protocol-go)

I saw this statement a few times in the posts above and I wonder why is it so*? Why do we need to choose one instead of plan for interfaces, modularity and encapsulation?

I think that a vision where we have a few components like p2p stack, storage, chat logic etc. where some are written in Nim and some in Go is totally fine. The important part is how the client can interact with these components.

This is the mistake we have been repeating over and over – building these giant monoliths where to run a stupid unit test you need an Ethereum node and business logic is tangled with low-level p2p bullshit. Nimbus on mobile effort seems like a good opportunity to break out from this loop.

*One valid argument is to simplify the build process. That makes sense however, (1) we already figure out that and (2) having just one output skews the perspective. Also, using this argument, I could say that we should get rid of React Native – that would simplify a lot of things :slight_smile:

Next step will be to do some janitorial work on status-protocol-go to ensure that geth types/implementation details do not leak out of the very lowest layers, making it simpler to later implement a Nimbus transport layer.

That sounds fine. One more thing that I mentioned during the call is to figure out how Go and Nim (and Java and Objective-C) can communicate. So far, we have been relying on C bindings. Maybe there is a better way?

19

Not quite the same, but we just dusted off stratus to make sure it still builds (thanks @stefantalpalaru!) - if you’re looking for more prototype-level whisper code to experiment with, there’s some in there.

Hopefully it should work well with the 1.9.5 geth/whisper implementation.

20

Daily update:

  • After having made a first pass at wrapping Geth data structures/APIs in status-protocol-go yesterday, I’m working today in updating status-go and status-console-client to use the updated API. Lost a bit of time since I started on recent Geth 1.9.x branches, only to realize that out-of-the-box it is incompatible with Geth 1.8.x due to PoW changes, so messages would have no chance of arriving to the destination. Rebased on existing Geth 1.8.x branches.