IFT Request: Status Backend Design Specification

Our Mission
1

Tl;dr - Define architectural backend for Status and the specifications of the modules required to deliver current functionality and beyond

The IFT would like the Status App teams to come up with an ideal architecture specification for the Status backend (status-go). This work would be a partial effort from all parties involved (No 100% FTE allocation), i.e. it doesn’t hinder current user growth and retention efforts.

With it, it defines

  • the modules (e.g. draft specifications) it needs to consume to replicate current functionality
  • the manner it creates business logic around these modules to surface features
  • the process in which it surfaces these features to a front end
  • the orchestration mechanism that facilitates all of this

The resultant deliverable needs to satisfy the following org constraints:

  • defines clear product/business deliverables that are made possible by underlying changes; the delivery of those product goals, in a layered, modular, and spec’s manner is the “Definition of Done”[1]
  • facilitates the ability to efficiently consume project capabilities (inside IFT and out), develop business logic around them, and surface them to the front-end.
  • logically built to be efficient, from both a performance perspective as well as maintenance and contribution perspectives.
  • allows for easily built, Status compatible, front-end clients based on surfaced features
  • can efficiently consume Nim packages, as the initial targeted consumed modules would be Nwaku (networking), Chat SDK, Nimbus (potentially light client proxy), Logos Core, and Codex (large data storage and retrieval)
  • should prioritize internal (or recommended) dependencies (IFT) over external ones.
  • clear separation of concerns, who owns what across the stack should be intuitive and understood.

The Status team should leverage key resources within the IFT to confirm compatible architectures.

A draft of these deliverables should be ready to present and discuss at the All Hands in Croatia.

[1]: A note on this one for clarification. The business case drives the tech. We most definitely don’t want to find ourselves in a position where we’re just “refactoring code” as engineers and that effort is completely divorced from the features we’re providing to users in order to grow adoption and retention. These two things go hand-in-hand.

Some Motivation as to why (very non-exhaustive)

The Status backend suffers from significant historical and technical debt, and lacks a number of properties deemed vital to its long term success.

  • Many of the CCs that made the decisions that lead to the current architecture are not here anymore. The lack of documentation and knowledge passing has lead to current maintainers not fully understanding the codebase they’re responsible for.
  • the separation of Vac/Waku from Status and no strong communication afterwards left portions of the codebase without clear ownership (chat protocols).
  • incorporation of new protocols and the development of new features and business cases for Status is slow and difficult to reason about
    • e.g. identifying issues with Waku performance being caused by Waku or Status’ implementation being difficult, no integration of Nimbus or verification proxy
  • It has been difficult to abstract away key protocols within the codebase
    • e.g. Chat SDK and communities
  • It is effectively impossible for the community to “build on Status,” limiting our ability to grow a developer community or any permissionless innovation of the ecosystem.
  • the start of liblogos, a core to web3 clients

The IFT needs a function within itself that fosters an efficient road from new infrastructure functionality to user facing access to it. Having such a thing serves multiple purposes:

  • valuable user feedback and live performance testing to new functionality within our infrastructure projects
  • quick adoption of novel features to test new business cases
  • separation of concerns

Status should be the vehicle for this. So let’s start mapping a road to make it one.

As always, feedback, requests for clarification, and criticism welcome.

edits:

  • altered wording around the timeline and quality of expected delivery
  • added historical ownership motivational reason
  • typos
5 Likes
2

I’m really glad to see a push for clearer backend direction and long-term architectural thinking. Thank you for sharing it. There’s a lot to align on, and the direction outlined in your post opens the door for meaningful discussion. Which is really great to see.

For the sake of alignment vis-à-vis the actual implementation within status-go could we visit a few points just to balance out some observations in this post? This could be a good opportunity to polish the context as presented above and perhaps apply some minor tweaks for accuracy on the architectural reality.

As stated, I really support the mission of defining an ideal backend architecture. Maybe to help us work towards a practical roadmap I have noted a number of points that jumped out at me, I believe those not familiar with the status-go code base would benefit from understanding them.

The lack of documentation

Agreed. We have fits and spurts of motivation on this, but it always finds itself deprioritised after a time. Formalising specification is always a worthwhile goal, and I welcome efforts to improve it collaboratively. I understand this is a shared issue across many IFT teams so working together we can definitely benefit each other.

Chat protocols.

This is an area we certainly need to smarten up. As many will know in Vac’s absence Status CCs have actively worked on the chat protocol for years, though the work of encapsulating it has never been a priority.

You’ll likely remember that 5 years ago we chose to de-encapsulate the chat protocol by migrating the chat repo in to the main status-go repo, we can debate whether that was a mistake, I’d argue it was. For those that don’t remember see the details from the repo:

“incorporation of new protocols and the development of new features and business cases for Status is slow and difficult to reason about”

This point is probably meant to be read with a pinch of salt, as the observation obviously doesn’t actually track with the experiences of Status devs. Those reading this literally can be assured that we have many many many examples where we’ve implemented new features and new protocols in rather impressive timeframes. A handful of examples include :palm_up_hand: :

  • Chat Identity protocol and user profile feature.
  • Wallet on-ramping
  • Wallet swaps feature
  • Actually the entire wallet product is a testament to fast and easy to reason development.
  • Local transfer, sync and pairing protocol and feature
  • Double ratchet encryption algorithm implementation for group and 1:1 chat
  • Hash ratchet encryption for communities encryption
  • Communities history store, transport and restore.
  • Push notifications and local notifications.
  • etc etc

“no integration of Nimbus or verification proxy”

Yes, this is a critical and highly desirable feature, though the issue with implementing it within status-go has never been architectural.

For the benefit of those new to the project, we have in fact attempted 3 times with concerted effort to integrate Nimbus / Fluffy. The real obstacle has consistently been the extreme difficulty of integrating Nim-based components into a complex user-facing cross-platform Go/mobile stack.

This is a very large and technically complex subject, and the details of which are understandably occasionally forgotten. For the benefit of institutional knowledge I’ve already drafted a highly detailed analysis of the extensive and technically intricate issues we have experienced with this venture. And I’m currently consulting with our fellow CCs on each point, and I can share it once we are confident of the content.

The unavoidable reality is that the go language is near peerless for cross compilation over multiple demanding platforms, Nim regrettably is not. We need to work towards resolving this, I’d love to see Fluffy working within Status it would be amazing. In any case to move forward, any future plans for integration need to take on board the lessons learned on this matter.

“It has been difficult to abstract away key protocols”

The Chat SDK could be used within the current structure, but improvement to the architecture would certainly make this a better developer experience. There is always room for improvement, and currently there are some unfortunate tight couplings between the core products within Status(e.g. Messenger ↔ Waku ↔ Wallet), but they don’t present a current blocker to building a chat SDK.

Communities are relatively self-contained. Its current immaturity though is regrettable and due to a rushed implementation under pressure from a poor understanding of the complexities by a former program lead. Communities grew too fast, with baked in assumptions, overly opinionated, and wasn’t resourced properly to evolve.

“The community can’t build on Status”

This isn’t a technical or architectural limitation. It’s a product focus issue. The only real extensibility feature we had, the DApp browser, was removed under instruction by the same former program lead. Dapps were/are an exceptional way for us to encourage community building, we would write about it regularly. See here for examples Status app - Dapp integration news.

More broadly speaking 3rd party building hasn’t been prioritised as a use case. That’s a strategic call, arguably not an architectural limitation. On a product level we have not implemented the required hooks for 3rd party developers to build from.

That said, we are working right now on improving chat extensibility incrementally. It’s a product-driven effort, and like all things Status, it’s proof-oriented: build, test, validate. Architecture will follow function.

For more details see our first stepping stones to allow a more extensible chat

In Summary

I really love the initiative, and after reconciling it with what implementation work actually looks like, we’ll be able to make some great progress. I think you will find strong support of the idea within Status, we all want a future-facing status-go that can work cleanly with Logos technologies (personally I’m dying to see Codex able to be used within Status).

It will take real effort though as it needs to account for cross-platform constraints (desktop and mobile) as a first-class requirement. It’s one of the hardest and most constant forces shaping the Status codebase.

3 Likes
3

It is my opinion that the integrated project (in this case nimbus, but also Waku or Codex), should be the one producing C-bindings, and a Golang wrapper to allow integration in a Golang codebase.

I believe expectations should be set in this matter, whether or not they are in line with my opinion.

2 Likes
4

Yes, I would welcome expectations in this matter also. I agree with your opinion. It would be a rare exception for an upstream lib author to not consider integration convenience as part of their core offering, as this is the key to wide adoption and portability.

5

Do note that in the case of Nimbus, there is a clear counter-argument here which I did not think of initially.

I think in the case of Codex and Waku, and I can only talk really for Waku, it make sense to own and maintain some popular wrappers to encourage adoption.

Not sure it makes as much sense for Nimbus, due to the fact that client diversity is driven by programming language diversity.

6

Agreed, that is an excellent point. Nimbus specifically will likely never be native. It could benefit the wider conversation for us to compare the ease with which we can integrate and use the native language implementation (geth) verses Nimbus, the contrast is as different as night is from day. That’s not to say geth is favoured for functional reasons, it isn’t, Nimbus is clearly preferable in that respect, eg geth has no equivalent to Fluffy.

Though I find myself unsure how aware the wider IFT is with the titanic implementation and maintenance burden “efficiently consuming Nimbus” within Status requires. There are strong, non-architectural, reasons for why we haven’t achieved this yet, despite multiple attempts.

It is A LOT of work (even with the perfect architecture) and this is not a burden of Status’s making or failing. It is reasonable for part of this load to be carried by other teams.

1 Like
7

This (and every other difficulty you have mentioned herein) is all based on the assumption that the ideal backend is written in Go.

This assumption doesn’t have to exist and I need the Status team to at least consider this. I am aware of the amount of work to change things, and I constantly put that into combat with the amount of work to work cohesively as an org towards what we can all call an “organizational north star”. It isn’t simple.

I know that there are sentiments that we (the IFT) are submitting a proposal for work with the intention you (Status App teams) come up with a solution that we already have in our heads. This is partially true, but requires significant conversation and collaboration to explain.

The reality here is that I have submitted a request for work in order to give the Status team (the current domain experts) an opportunity to come up with a solution that satisfies the above requirements and constraints on their own. This leaves room for solutions that I (we) have not come to ourselves and presents opportunities for collaboration and discussion (the all hands).

In the event the proposed solutions are not satisfactory, and subsequent discussions don’t lead to an agreement on a way forward, then we have a way forward that can be technically justified. I don’t see any other prudent way to go about things. We wouldn’t be able to evaluate the submitted proposals if we didn’t do this.

2 Likes