How Stimbus or what is needed to start this work

Archive Software Engineering
1

Following up on @jarradhope’s timely explanation on Why Stimbus, I’d like to kick of a discussion on how we can organize integration that will align the organisation.

Who works on this, the expected outcome and way of working are some of the first things to be decided on. Here’s a proposal following separate and a collective conversation with @arnetheduck @andre @iurimatias @oskarth and @kim to understand the work required. Full notes here.

Please share any thoughts on who needs to be involved and in what capacity (contributor, stakeholder) and what responsibilities this team should have (i.e. what you expect in terms of deliverables, communication and way of working).

Proposal: Production team Stimbus

Set up an independent team to work on a shim layer that can be interfaced with through an API and gradually phase out status-go.

Contributors

1 or 2 members from the teams:

  • Core
  • Desktop
  • Vac (Waku)
  • Nimbus

For contributors, Nimbus integration needs to be their number 1 priority and main allocation. Contributors bring their knowledge of requirements for respective clients, but beyond that do not represent interests of any client team. As the layer needs to be client agnostic, independence of the production team is key. Anyone with relevant expertise can join this effort as long as there is agreement on other critical work not being at risk (Mobile, Desktop, ETH2, Vac, Governance, etc.)

TBD is how the team will be lead. Could be an individual or rotation, can also be decided by the team.

Stakeholders

Responsibilites

  • Collect and curate common features for shim layer (check lib-status on what’s necessary)
  • Specify features for shim layer
  • Assure review by stakeholders of feature specs
  • Integrate high prio features (Messaging > nim-waku, shim layer)
  • Assure quality control of implementation
  • Experiment with naive Lightclients node of Eth1 on Desktop

Planning

Start work

Within 2 weeks

Delivery

Critical features used by clients (replaced status-go) by end of calendar year. Critical features TBD

1 Like
2

One thing I feel fairly strongly about, is that rewriting status-go should not be the main focus of this effort.

I appreciate the reasons exposed in Jarrad’s post, I think it’s very valuable to have sinergy between teams, and definitely I agree with the ultimate goals. But there are many ways to skin a cat.

From a user perspective, there’s no value on whether the backend is in status-go, nim, or a mix of both.
As a matter of fact a user will be negatively impacted, as we will take away resources from new features, and introduce QA issues and bugs, long before we can ripe any benefit from the move.

Just to give you an example, as I have actually worked on some of this and actual understanding of the effort required:

  • PFS, effort required roughly 3 months, 1 dev

  • Move application logic to status-go (that’s a rewrite essentially), started on July 2019, we ended roughly on December 2019 (2 devs were actively working on it, not full time, as we had other v1 priorities). This took an incredible amount of understanding of the two codebases to make a smooth transition. Inevitable QA issues were introduced, although in this case the benefits easily overweight the drawbacks, in terms of performance, ease of development, as testified by the fast progress made by the desktop team.

  • Datasync, roughly 2 months, that’s probably the least technically challenging, as simpler than the previous two.

All of these are separate components and have clearly defined interfaces, as they used to be in separate repos and have been added to status-go in some cases for operation reasons.

Other parts that will have to be rewritten is wallet, transactions, extended keys etc.

I am all for using nim, and I would definitely not write new stuff in status-go, write it in nim, and integrate some critical parts (waku for example), the rest I would just package as C code and integrate.

I just would like to make sure that the goal of this effort is not to “phase out status-go”, as that as little value in itself, rather, we should go back to Jarrad’s post when it comes to motivation and deliverables.

Namely “Stimbus forces alignment across the organisation. Stimbus is the focal point in where the 3 teams come together, it aligns Mobile, Desktop and Nimbus”, I doubt that requires rewriting status-go, that looks more like engineering-driven thinking Things You Should Never Do, Part I – Joel on Software.

Another thing worth mentioning is that we already had plenty of discussion about this
Status.app and I am surprised that the work done already by pedro does not seem to be mentioned, at least explicitly.

3 Likes
3

Thanks @cammellos really helpful to get the rough implementation effort of relevant activities.

Happy to take gradually phase out status-go out of the proposal. I do not have an opinion on this and think such decisions should be left to the production team. What I want to make sure we achieve here is a shared understanding of expectations for this production team. If those expectations include that removing status-go should not be the objective that sounds good to me. That said, would love to hear what others’ expectations are

1 Like
4

While gradually phase out status-go isn’t a goal in and of itself, there are a few reasons why this eventually will likely happen naturally - some of them are technical (multiple thread models, multiple garbage collectors, different string implementations), some of them are social (simplicity of tech stack, developer focus, hiring, mechanical sympathy between parts, audit requirements on security-critical code etc) - another way to put this is that I’d generally agree that it doesn’t really need to be a goal in an of itself, but can indeed be left to the production team itself.

Perhaps more importantly though, and why this gets brought up in discussions around the unification, is we should make room for it inh the architecture of the transition, acknowledging that we want to retain the ability to make case-by-case decisions for individual pieces, that no stakeholder gets blocked by this discussion as the project matures, and ensuring that there exist a clear path to putting code and logic where it most naturally makes sense.

Just like the migration to status-go improved the clarity of communication and responsibilities, so should the introduction of stimbus be done such that the API’s and modules we define act as conduits of understanding between the teams involved - providing an unambiguous definition of what each component needs from the others.

It’s also worth remembering that the protocol definitions and specs that we’ve written dramatically lower the cost and increase the flexibility of choice, insofar as implementation details are concerned - by being rigorous here we ensure not only that it’s cheap for us to nimbly move between implementations, but also for anyone else that might be interested in the fruit of our work - the code itself, be it in status-go or status-nim is in many ways byproduct of going doing the hard work of, based on a user story, understanding the problem, gathering requirements, balancing the tradeoffs and picking a solution. This flexibility is key to managing the uncertainties of our field and the rapidly developing landscape.

1 Like
5

Moving a conversation on team lead for Stimbus on Discord to keep the discussion open and in one place.

@iurimatias proposes to take on a role as teamlead for Stimbus as it relates to Desktop. @jarradhope outlines reasons for why this makes sense, given that @andre and someone from Nimbus have input on the requirements, and sign-off on the implementation and deliverables.

iurimatias

I propose myself as the team lead for stimbus as it makes sense due to desktop

fizzgig

@iurimatias I’m not opposed to this, seems to make the most sense to me. Reasoning: The Desktop team needs to familiarise themselves with the codebase and has the most bandwidth to deal with it. As @cammellos highlighted that this is a waste of time from the mobile app teams perspective, and they should be focused on user-facing features. I couldn’t agree more, the mobile app team needs to be laser focused on user growth, particularly with user onboarding and retention. Having that said I think @andre and someone from Nimbus should have input on the requirements, and sign-off on the implementation and deliverables. I know @arnetheduck had some good suggestions on ensuring a gradual change that is non-disruptive. Also I want to re-emphasize that this is on the lower priority to shipping Desktop.

Also, adding to responsibilities based on comments by @arnetheduck and @jarradhope

  • Assure sign-off by stakeholders
  • Allow for case-by-case decisions for individual pieces. Limit dependencies between feature implementation
  • Ensuring that there exist a clear path to putting code and logic where it most naturally makes sense
  • Provide audit proof documentation
  • Define API’s and modules along with required input, output and interfacing with other components
  • Specify features for shim layer by defining user story, understanding the problem, gathering requirements, balancing the tradeoffs and picking a solution
6

Input from @oskarth

  1. Priority for this team should be on getting nim-waku used in one of the clients, this is the critical path
    for Waku v2, one of the most immediate upcoming bigger protocol/researchy upgrades, is developed in Nim
  2. Doing above while ensuring mobile all gets a stable surface that isn’t impacted by more experimental work
  3. We need clarity on resources, what the team is going to be in terms of specific individuals and how much time they are allocating (e.g. I know Samuel is likely to start with 1d a week, say)
  4. As for API the whole shim idea Jacek was pitching makes sense to me, but I haven’t thought about it in too much detail
  5. Happy to speak more to Vac side of things and be interface point for research integration
  6. Concern I heard from Core, is reimplementing things that already work with little benefit to end users, as well as breaking things. Doing 1,2 and 4 and possibly other actions should alleviate this concern
  7. At the same time, I don’t see any reason for non core resources to not work on porting certain features, if they deem this to be necessary (cohesion, mechanical sympathy, learning/onboarding, etc)
  8. As for how to achieve 2, outside of 4, it seems reasonable to me that (a) at least one core resource is in team (b) we use a PR workflow with one reviewer each from <core,desktop,(nimbus),((research))> [latter two can be done at discretion, probably - e.g. for Nim related code structure and research integration points]
7

Cross posting the way forward as discussed during the sequel of Core Dev call #32. Meanwhile @iurimatias has shared an architecture proposal here. First call for the workgroup is being planned. As the work has started following the agreement below, I’ll close this topic.

a. What, Why, Why now
Consensus on valid arguments to integrate Nimbus:

  • A Create synergy between teams
  • B Support Nim-Waku features

Other arguments

  • Removing Infura: Can technically be done without integrating Nimbus
  • Preparing for ETH2: Assumes there wouldn’t be an option to use go with ETH2 which is debatable

b. Responsibilities

  • not discussed

c. Deliverables

  • Initial architecture proposal by end of next week
  • Discuss and analyze this week

d. Way of working

  • Iterative
  • Bi-weekly call
  • Stakeholders to sign off on specs (see list on Discuss + @cammellos)

a. What, Why, Why now
Consensus on valid arguments to integrate Nimbus:

  • A Create synergy between teams
  • B Support Nim-Waku features

Other arguments

  • Removing Infura: Can technically be done without integrating Nimbus
  • Preparing for ETH2: Assumes there wouldn’t be an option to use go with ETH2 which is debatable

b. Responsibilities

  • not discussed

c. Deliverables

  • Initial architecture proposal by end of next week
  • Discuss and analyze this week

d. Way of working

  • Iterative
  • Bi-weekly call
  • Stakeholders to sign off on specs (see list on Discuss + @cammellos)

e. Who

  • @iurimatias (lead)
  • Core contributors of the desktop team in rotation
  • @deme (represent Nimbus team)
  • @vitaliy (represent mobile requirements)
  • @samuel (go knowledge)

Solution requirements

  • Lower impact on existing clients
  • Gradual approach - No big bang switch (To be explored if Shim layer can provide this, depends on implementation, e.g. use of 2 layers a) native Nim layer and b) C-interface