Desktop paths forward, a choose your own adventure?

Archive Software Engineering
1

So, there has been a back and forth on how to move forward with Desktop. There has been experimentation among different cohorts in the organization, and a lot of back and forth on what the tech stack should be. This thread is an attempt to bring the conversation to a more public forum so that everyone can follow and stay up to date / opine if they wish. I’ll add two recent “summaries” to give a bit of context, and others can add more if they feel it doesn’t summarize things enough:

@jarradhope said:

The Embark team has been getting familiar with the current tech stack and running different experiments, having identified Nim/Qt as a potentially simplified and impactful technology stack, demonstrating a proof of concept with some screens implemented, status-go integration and basic chat.

Andre has raised some great counterpoints around the risk of a new implementation, the risk of having bugs, split teams on different codebases, potentially long time to market.

At the same time we’re looking to productionise a desktop application sooner rather than later, so we need to choose wisely, however we don’t appear to have clarity on which approach is best.
I reached out to Volodymyr, as last year he was for a C++/Qt implementation yet he is the goto man for react-native-desktop, he made the good point that we did have a working desktop in the past, albeit with a simpler interface and Volodymyr himself wasn’t convinced which direction would be best. So it sounds to me we don’t have enough information to make the call.

Volodymyr suggested that perhaps Volodymyr and Vitaliy push status-react and react-native-desktop as much as they can within a fixed period of time (1 month?), to see how far they can get.

I think this is a really good idea.
Another consideration, we want to replace status-go with a clean interface, and integrate Nimbus in the application. This should be sdk lib that other developers might want to use.

So another idea might be that, while Volodymyr and Vitaliy are working on the status-react desktop stack, we can have people working on this status-go replacement, and they can have Nimbus folk overseeing their code quality as they learn Nim.

That way in the future we will have a clear picture of what direction to go, and we’ll have significant progress on integrating Nimbus into the app (mobile and desktop).
Let’s think about this, if there’s any other ideas on how to proceed please share, then let’s come to an agreement on the best path forward, I would be happy to join a call next week, (keep in mind Andre has well-deserved time off and we should respect not to disturb him) - we should aim to be on the same page by end of next week, how does that sound?
enjoy your weekend !

and @vitaliy said:

I am not at all entitled to hard opinions, some thoughts that could be very wrong:

  • react-native-desktop path: benefits of code reuse, but difficult to integrate and to maintain w.r.t. mobile code. Definitely worth to try nevertheless. Would be a pity to waste so much effort.
    On the other hand, the stack is extremely complex and prone to breakage at various places. I don’t know how we can tackle this. It’s inherent to ReactNative i think. There is a major update coming soon (JSI/Fabric etc.), so this should help make things more performant (but more complex again maybe?)

  • Nim/Qt: it’s great to have a client from scratch, helps to throw light at various parts of the whole client<->status-go interaction. And will help in improving the Nimbus client interface.
    Wondered about Nim vs C++ choice specifically, apart from it being a powerful lang in its own right. Meaning, how sure can we be it’s not going to cause unexpected issues along the way? E.g., i was looking through Status’s Nim repo to get a feeling of how it’s used, and quickly came across Prevent a mysterious crash in NBC · status-im/nim-faststreams@6ce0472 · GitHub. Also nim-qml library might need to be updated/fixed, but the stronger version of the same argument applies to react-native-desktop-qt anyways.

My Opining

We have made a stance that Nimbus will eventually be the backend of Status. We fund a relatively large team to work on all of the things, and have a large budget to do security auditing on it, for a reason.

Whatever option is the one that makes it easy to incorporate Nimbus work into our future application development cycle is the correct one forward. This is my personal top priority. There are tradeoffs of course, and potentially wasted work from what we have, but the end goal is to have something that unifies the work being done now, and what we have made clear we will be working on in the future.

I think it is reasonable to do what @jarradhope mentions and see what is what at the end of that. I do not want to waste too much time discussing and arguing while not working forward. We as an org have a tremendous amount of hard work ahead to do what we’ve all set out to do. If there exists dissent and argument at the end as to which to do, fuck it, do both.

That being said, I really do appreciate the input from all the domain experts of Status. This org is one that has many opinions, all of which are valuable to me. So I’m at a cross roads of how much do we talk vs how much do we do?

7 Likes
2

Just to give a bit more context that I believe is not captured in the post.
This is the post that started the conversation:

I browsed through the nim-status-client repo and it’s really cool!
Understand that going status-react+rn-desktop route would have it’s own quirks. The repo and processes around it are complicated, there are lots of unknown unknowns, etc. Of all the desktop options, i now think i’d prefer a stock Qt (C++/QML route) though.
To be honest, if i were to find out about the whole thing a week ago, i’d gladly ask if i could join the effort. But the whole thing didn’t play out nicely, i think. Why would there be a private repo? I agree that decision making need not be collegial at all times, but at least some form of transparency would be welcome.

It’s not just a question about the tech stack, but also how we handle decisions and how much we value experience and expertise in the company.

I have to say that the way it came down is a bit embarrassing, as I would expect the developers who are most intimately familiar with the project to be heavily involved with the decisions. Granted that it might be a slip up, and things do happen, but it’s worth reflecting on how it came to happens.

I’d expect the discussion to be less centered about what’s going to happen with desktop, but how are we going to make those decisions, if we get better at this, any new initiative will benefit from it, instead of just the ones we are currently working on.

This is crucial especially in initiatives that have a large impact and require a considerable amount of effort, for example integrating Nimbus, which is technically a very challenging and delicate process, that requires some thinking ahead on what’s the best approach ( already discussed here Status.app ).

This goes back to “How much do we talk vs how much do we do?”, generally, I think it’s important to distinguish output vs outcomes, rather than doing and talking.
Banging on the keyboard like code monkeys might not take us very far, while sometimes having a conversation is just what we need to re-align ourselves and head in the right direction. Sometimes of course is the opposite.

Often I have the feeling that as an organization we value outputs more than outcomes, and we tend to create work for ourselves unnecessarily, without really evaluating the impact of the work produced (that’s a natural tendency to some extent, but having organizational awareness of it it’s important).

That’s my 2p, tech stack wise for desktop UI I don’t have a strong opinion, instinctively I would lean towards the technologies that the majority of dev’s in the team are most familiar with (I take that’s JS), to leverage their expertise, re-using status-react is also an option worth considering (and I am sure it has been considered), but other will have a better understanding of the pros and cons of this approach, as it has been already worked on in the past.

Nim/QT sounds like a fine choice to me, ultimately I think this is a decision for the destkop team, which best understand which technology will be a best fit for their team.

For Nimbus integration I believe it should be gradual and we should strive by slowly “chocking” status-go bit by bit, instead of having a parallel effort, as replacing a moving target requires much more effort, similarly as how we discussed previously.

8 Likes
3

Well, we don’t have enough resources for that :slight_smile:

4

Hi!
My thoughts on technology selection:

react-native-desktop-qt + status-react

Pros:

  • We already know that it can work. Status used this Desktop version for months.
  • Code reuse! It is possible.
    @andrey created first desktop UI in a week I believe (correct me if I’m wrong). It worked but had own, non-mobile UI and reused logic.
    When we decided to use Mobile UI instead of old desktop -
    it took me alone only couple months to run it. Significant part of this time was spent on custom two-pane navigator for landscape orientation and dealing with breaking changes in master.
  • same technology stack and same sources means devs can be moved between desktop and mobile without learning curve. It is much easier to reallocate resources. Also there won’t be difference in hiring for Desktop or for Mobile. I believe that shouldn’t be underestimated.

Cons:

  • You might think this one is the main - maturity of react-native-desktop. Yes, it can’t be called production-quality. There are two problems we will encounter: lack of features/bugs in rnd itself, lack of desktop support when we use 3rd-party modules that has native parts. One part of solution is allocating resources to rnd but the second very important one - is making some tradeoffs. Desktop UI can be simpler. If we don’t have something - we can add it or question its neccessity)

  • I think in fact this is is the main problem - integrating of Desktop in the same codebase would impact Mobile and the whole team. When desktop existed, biggest problem were changes in Mobile that break Desktop. And for successful code reuse there should be a mind shift - every movement in Mobile codebase should take impact on Desktop into account.
    This part feels quite challenging for me.

Creating new client in C++/Qt/QML

Pros:

  • new codebase:
    • It should be cleaner and nicer than since Desktop already has clear vision on what it should achieve in the end. Mobile codebase was evolving for a long time and probably many things you would do differently now.
  • new technology:
    • Very solid technology that proved UI capabilities on all platforms. Thats a lot.

Cons:

  • new codebase:
  • new technology:
    • increased learning curve and therefore impossibility to quickly reallocate resources between Mobile and Desktop
    • New hired developer should be for Mobile or for Desktop.

Creating new client in Nim/QML

Here is a problem - I don’t know much about this technology. So I can’t fill “Pros” section :frowning: @iurimatias I’d ask you and your team to add things in comments. I’ll update my post.
Cons are similar to Qt/C++

Pros:

  • new codebase:
    • It should be cleaner and nicer than since Desktop already has clear vision on what it should achieve in the end. Mobile codebase was evolving for a long time and probably many things you would do differently now.
  • new technology:
    • ??? (will fill from comments)

Cons:

  • new codebase:
  • new technology:
    • As far as I know, Nim/QML bindings is a separate project that isn’t mature and that will be one more area that needs attention.
    • increased learning curve and therefore impossibility to quickly reallocate resources between Mobile and Desktop
    • New hired developer should be for Mobile or for Desktop.

My opinion: until now I can’t say what path is better. But after creating this comparison I feel like first path is safer at least.

And to engage more people in discussion lets do something like instagram challenge with mentions:
@andrey and @roman, I’d be happy to hear your thoughts on desktop path forward!

6 Likes
5

i’ll suggest using GitHub - asticode/go-astilectron: Build cross platform GUI apps with GO and HTML/JS/CSS (powered by Electron) , it’s simple when u know golang/js/css/html5.

6

@volodymyr.kozieiev, do you mean ui related stuff was often broken by changes on mobile side? The rest of it almost likely can be easily tested. The thing is (but i’m just guessing now, not 100% sure) that we will want to have almost same layout on tablets as on desktop. This means that even if we ditch rn-desktop we will have this kind of problems in rn-status, thus we need to solve them anyway.

so my two cents:

  1. Almost likely we would need the same or very similar layout for tablets/desktop, thus even if we try to implement it for rn-desktop it shouldn’t be in vain even if we ditch it at some point. So imo the fist thing we should try is still rn-desktop.
  2. Assuming most of logic was moved to go side (and will stay outside UI part even when we move to nim), it shouldn’t be a big problem to implement UI using a different framework, if we detect other parts of logic which we don’t want to reimplement we have to move them to that backend.
  3. If we ditch rn-desktop I would choose framework based on technology’s maturity rather on team’s expertise, because as far as I understand regardless which technology we choose from nim-qml/cpp-qt-qml/your-option list we actually don’t really have a huge team expertise/experience in implementing desktop UI using those stacks, so then we would want at least do not bother with framework’s bugs. But I agree with @cammellos that should be desktop team decision.
2 Likes
7

@frank as I remember we can’t use electron based stuff because it’s impossible to implement proper webview there and we need it for a dapp browser.

8

Hi!
Following the @volodymyr.kozieiev thoughts on technology selection.
As mentioned, while the react-native-desktop-qt is not yet production-ready, a version of Desktop app could be built for macOS and Windows using the actively maintained react-native implementations from Microsoft:

macOS could be also achieved with Catalyst as @andrey mentioned earlier in the chat Support Project Catalyst (running iPad apps on macOS) · Issue #131 · react-native-community/discussions-and-proposals · GitHub
And Linux support could be extended later using the react-native-desktop-qt
These approaches now have already good support for some community packages.
Example: GitHub - react-native-webview/react-native-webview: React Native Cross-Platform WebView

Edited:
Two major libraries that are used widely for interactions in react native (also by React Navigation) will come soon with support for desktop, the will be rewritten in C++ which will also enable possibility to reuse in the react-native-desktop-qt:

2 Likes
9

@roman, not only. Things like adding a new 3rdparty module to mobile that wasn’t included in desktop build. Or updating only ios/android part of react-native-status when interface changed. But of course that can be fixed by proper automated testing.

10

just want to share my vision of the desktop application,
first, it’s never been a priority and clojure core developers have never worked on it full time. What we’ve been trying to do is to run status-react code on desktop, this approach has several drawbacks. We add conditions and complexity to mobile code. On mobile we have lots of functionality and libraries we don’t need on desktop (camera, gestures, biometric etc ). Instead we could create a separate code base for desktop and reuse as much code (business logic and UI components) as possible from status-react. With re-frame framework it should much easier to do. Next react-native-desktop-qt , it is not yet production-ready for sure and needs lots of effort to maintain it, also react native is changing every day, and even in status mobile version updates is not smooth and easy. Last time i was able to run desktop app based on react-native-desktop-qt it was laggy and has different issues with rendering and had memory leaks, so we can say for sure 1 developer is not enough for such project, we can just look at react-native-windows repo

image
image772×318 10.3 KB

, and here I agree with @Ferossgp, Microsoft actively working on this repo and even added macOS support, so its worth a try at least, status-react has also changed, app storage was moved from realm.js to status-go, shadow-cljs instead figwheel, we’ve spent a lot of effort to support this in desktop. So my suggestion would be

  1. run a new clojure project on windows and mac using react-native-windows
  2. integrate status-go and implement simple sign in flow using UI components and code from status-react repo
  3. implement chat using UI components and code from status-react repo

i would say if 1 and 2 went smooth, we can have a fully functional chat in 1w by 1 dev, I’ve already done that trick twice :slight_smile:

In status-react repo we have 200 contributors :partying_face: and more than 5500 commits, we had lots of contributions from bounty contributors, and our codebase still clean and good, and application is production-ready and has zero bugs or crashes in runtime (i believe its because of clojure and re-frame), developing status react we have hot reload with the persistent state, repl, lisp (clojure), re-frame, and removing any part is a step back

5 Likes
11

There are good arguments for both approaches here. If there was ever a time to simply try both approaches, this is it. In the end, the users will decide which approach works better and the two teams will be able to learn from each other and keep each other accountable in terms of delivering the best user experience. This iterative approach may work much better than the big bang approach where we plan it all out and commit to one well defined path as there are many unknowns along any given path.

I end here with a summary of a quote about IBM in the early 1980’s and how they choose to try pursuing the PC even though they already proved with near scientific rigor that the mainframe will be the way forward. Then in launching their own PC, they had two separate teams work on it. In the end it worked out as their approach to PCs became the standard for years to come.

The future, IBM knew, and could prove with scientific rigor, lay with the central station, the ever-more-powerful mainframe into which a huge number of users could plug. Everything—economics, the logic of information, technology—led to that conclusion. But then, suddenly, when it seemed as if such a central-station, mainframe-based information system was actually coming into existence, two young men came up with the first personal computer. Every computer maker knew that the PC was absurd. It did not have the memory, the database, the speed, or the computing ability necessary to succeed. Indeed, every computer maker knew that the PC had to fail—the conclusion reached by Xerox only a few years earlier, when its research team had actually built the first PC. But when that misbegotten monstrosity—first the Apple, then the Macintosh—came on the market, people not only loved it, they bought it.
Every big, successful company throughout history, when confronted with such a surprise, has refused to accept it… IBM, the overlord of mainframes with as much in sales as all the other computer makers put together and with record profits, could have reacted in the same way. In fact, it should have. Instead, IBM immediately accepted the PC as the new reality. Almost overnight, it brushed aside all its proven and time-tested policies, rules, and regulations and set up not one but two competing teams to design an even simpler PC. A couple of years later, IBM had become the world’s largest PC manufacturer and the industry standard setter.

12

A few random thoughts on this subject:

  • how much manpower is needed to maintain react-native-qt once it works?
  • is it satisfactory to ditch linux and only support mac/win?
  • are we willing to adopt nim-qt? this wrapper will likely end up needing contributions from us - trivial and mechanical unlike native-qt, but there nonetheless.
  • what would extensions/plugins look like, across mobile and desktop, if different frameworks were used? most of the web world already makes the distinction between desktop and mobile versions of sites, so this may not be a such a big problem, but…
  • Qt has android/ios support as well, what would it look like / what would the implications be of a qt version on mobile?

There are a few things that have changed that make this less of a dramatic choice than it used to be:

  • The protocol is described in a spec, separately - it no longer resides in the UI
  • Much of the actual logic is now in status-go - this trend would / should continue, with the UI becoming thinner and thinner, in terms of functionality and logic - ie the UI should focus on being pretty and productive while the logic lives elsewhere
  • Much of the difficult design work of crafting a good user experience is independent of the actual framework used - ie with sketches, user flows, screens etc done, maintaining parity becomes much less of a deal.
4 Likes
13

I would tend to agree. It might be worth making a post mortem on how this decision came to be and how we can avoid it in the future. “It” being the lack of transparency, surprise aspect, potential double effort, etc. Considering we’ll be working together for a while, and keep making decisions of this kind, getting this out of the way in a fashion where everyone feels heard seems like it’d be worth the effort.

@andre @iurimatias @hester what do you think? I don’t have a lot of the facts of how this came to be, so it’d be best if people familiar with the matter kick start it, and then we could bring in other people to fill it in. Shouldn’t take too much effort, and we can use Core dev calls for this too.

5 Likes
14

I’ll spin off a separate thread on decision making here, we can then also discuss on Core dev call if that makes sense. Please bear with me.

My initial thoughts on this are: This was not a flawed process, we simply don’t have a process, nor clarity on who makes decisions and how they are made. I think we can also improve our decision making skills (identifying decision, gathering information, propose resolutions, etc.). Until we reach the point of having structured community input, pitches and voting in place we need clarity and process.

Takeaways from #standups on Discord:
a We need to create clarity around who decides
b Whoever decides needs to collect input publicly (b can’t happen without a)
c Decision and motivation need to be communicated publicly
Sometimes b might not be feasible or convenient, but should be the exception, not the rule. c should justify why b has not been followed in such cases

2 Likes
15

This was not a flawed process, we simply don’t have a process, nor clarity on who makes decisions and how they are made.

I think that’s a bit of sugar coating, we might not have a process true, but there’s an explicit hierarchy in the company, and some people were involved or were in the known, which to me counts as a process . if there’s not one that is spelled out explicitly.
At any point anyone could have stepped in, but no one did, and the lack of responses in this thread is louder than anything.

Kudos to @oskarth to jump in, I would have expected someone (anyone in the known) to do similar as soon as the issue crop up, instead of having a single sided conversation between us.

1 Like
16

I can’t tell if you’re referencing me, I’ll respond as if you are. The answer is simply because there’s no decision to be made yet, I am exercising patience and if it’s not obvious by now I’ll state it clearly - I don’t value enforcing my will over the people having to work with their codebases - and I prefer decision-making to be bottom up. The times I’ve exerted my position is when I’ve absolutely had to, and if you haven’t noticed, I’ve even backed off on technical decisions and implementation because these days I can delegate to the absolutely talented and wonderful team that we are. There’s absolutely no need for any management layer to enforce a violent decision here.

Why is there no decision to be made yet? The situation as it stands to me is that Embark team and Vitality and Volodymyr were given a month to push their respective implementations as far as they could, why? So we would then have tangible material artifacts of 2 desktop implementations, which will allow us to make a more informed decision collectively.

The month to push of each implementation will be up next week, and I expect to see 2 workable demo desktop implementations.

From there, I am looking forward to us resuming the discussion about the implementations as an organisation. What are the pros and cons of each direction. Maybe one of them will be a clear winner.

Now, here is the reality check, the Embark team isn’t likely to switch to Clojure. It’s a foreign language to them and it’s a large codebase for a team that hasn’t grown up with it. So the rn-desktop implementation has to be damn good and there need to be large support from the core team to win over Embark team.

If core cannot make a compelling case, then the core team will be put into a position to decide whether they want to continue allocate 2 resources to continue maintaining rn-desktop.

If we end up doing 2 implementations, ultimately the market will decide which client is best.

Either way, I am indifferent, both implementations have great arguments for them, which is why we’re doing it this way. I think giving both sides a month to prove their efforts is the most fair approach to all involved, it respects the efforts of all, gives everyone else time to cool down, weigh the pros and cons, and bring clarity to collective decision making. Surely you would agree?

If a decision is demanded of me after end of next week, be assured I will give it.

3 Likes
17

I am getting a feeling that this conversation is beginning to have several focal points.

Just to reiterate:

  1. I was most appalled by abrupt switch to nim-qt, without any involvement in a related decision-making process, with a secret repo, with myself unknowingly working on making sure that status-react works for Embark folks.
    In hindsight, i think i was not properly aware of the fact that this is up to Embark team to decide. But i would have liked to know the rules of the game in advance. My reaction overall was quite emotional, apologies if it wasn’t appropriate.
  2. I was happy to find out that Volodymyr’s suggestion to have a 1-month effort was supported by Jarrad, so that we can push it as far as we can. But this wasn’t the original plan i think. I think originally the intention was to make a decision together, then follow it. Not to follow 2 competing paths.
  3. My understanding of your words @cammellos (“the lack of responses in this thread is louder than anything”) is that there are no responses from Embark team themselves here. Probably i’m wrong, but this is also a part of an issue i think.
  4. To be super-honest, I do think that nim-qt is really interesting and cool, and Embark folks are making really cool progress, and it’s nice to have an alternative client, this will help clarify API for sure. It’s just that on a human/emotional level, I don’t like how it originally came to be. We need no sugar coating, what happened is what happened, there was in my opinion some unhealthy attitude involved. I for one got over it already.
3 Likes
18

Thanks @vitaliy that captures my thoughts accurately, as I mentioned earlier, it’s not about technology but people.
I shied away from pointing out who’s role is to respond to this as I don’t think is appropriate for me to say explicitly, only to nudge.
So when I said "At any point anyone could have stepped in, I specifically wanted to keep it vague, as my knowledge of things is very partial, and it’s all a black box from an outsider perspective.

2 Likes
19

My 5 cents:
I don’t want to “win”, “lose” or “compete” against anyone in the same company :slight_smile: So how I see things - me and Vitaliy are doing timeboxed research to check rn-desktop potential. Outcome of this research can be used for a decision re-evaluation if needed.

4 Likes
20

Hi everyone,

Sorry I haven’t replied earlier, I wanted to give things a chance to settle down a bit.

First for some context, when the Embark team took on desktop, we had the choice to choose our stack which didn’t necessarily had to be status-react, however due to lack of any real alternatives status-react became the default path while we explored other possible alternatives.

So we did several experiments (which included electron, node-ffi, nodegui, clojure+javafx, and even forking firefox), the goal which was in part for learning purposes as it helped us better understand status-react & status-go itself, and in part to explore other stacks, until the last moment we were still going with status-react as the default route (and thus still inquiring about the progress of rn-desktop-qt) before realizing the nim+qt stack was a solid choice for desktop and helping out with nimbus integration.

Now, the part that is important to understand is that on Embark we were used to do things a bit differently and I was not fully aware about the culture in Core and lot of this misunderstanding seems to be caused by that.

For example due to competitive reasons we were used to do things in private repos until they were minimally ready, this was particular true for experiments like it was the case here or something that needed to be proposed.

And myself in every organization I ever worked in, even the most flat ones, typically communication always flows in a similar manner. i.e the lead communicates with the team, his/her peers or relevant people (for e.g @petty for security), as well as their own manager. Then it flows from there to the other teams. So on this aspect, from my perspective, everything was ok as I was communicating this way.

However in practice 1) as I learned this is not how we’re suppose do things, I understand that now 2) I didn’t communicate well enough to everyone it seems, and information wasn’t flowing as I thought 3) I made the mistake of assuming that communicating this in the PM meetings was enough.

So for what it’s worth, I would like to apologize to everyone affected, in particular @vitaliy and @volodymyr.kozieiev for any misunderstandings.

10 Likes