Core-UI proposal
There are two deadlines:
- code freeze for audit (Release Candidate)
- actual release (V1)
For this reason we propose to evaluate with Corey which part of the code have to be frozen for RC, and which can be worked on until V1, so that we can optimize planning. RC should include:
- anything to do with the way we manage keys
- underlying chat protocol
- the way the final binaries gets created
We consider that we are:
- breaking protocol compatibility between beta and v1
- dropping data between beta and v1
- moving communication protocol to status-go
The main changes proposed by Core UI team are:
- removing realm in status-react and using status-go as backend
- changing protocol encoding on the wire
The above implies that we are cutting scope by not:
- working out UX for migration/support of legacy accounts, then implementing it
- working out application issues between legacy and new accounts
- migrating data from status-react to status-go
- testing compatibility between beta and v1
We will break develop branch
We will iterate on release candidates and consider v1 ready once two RCs tagged from develop are tested compatible and feature full.
FAQ, unknowns and risks
Why remove realm?
(some arguments are repeated a few times in this document)
- 1000 lines of dependencies removed from the yarn.lock file, a bazillion line of unauditable code
- smaller memory footprint of the app (one less db server to run)
- performance improvements:
- less work on the js thread
- less back and forth between status-react and status-go at startup
- opens the door to further performance improvements in the future:
- by getting results from status-go in transit, which is the fastest encoding format for clojurescript
- potentially even clojurescript datastructures directly without using the bridge if we mimic the way realm was behaving
- get rid of ubuntu-server for desktop which was blocked by realm so far
- realm was a bottleneck at some point and writes were moved to an async queue that was managed by cljs, now
- development of alternative UIs/client is much easier
Unknown regarding performances of status-go vs realm
Besides bemchmarking of performances between the 2 databases (sqlite on go side, realm on react side) which will be addressed as a first step, the performance is influenced by different factors:
- realm was synchronous and status-go communications are aynschronous:
- previously we had to move realm writes to an asynchronous queue to avoid blocking the js thread when writing a lot of data (but still on the unique UI thread). This is not necessary anymore in status-go because the writes are asynchronous. Error handling won’t be worst than with realm where we can’t do much more than logging if writes fail (for a while it just crashed the writing loop and stopped writing anything to the db until app was restarted). We will have to make sure write errors on status-go side don’t have a similar effect, though error handling is go is suppose to be pretty solid.
- some queries are passed to the code and made in-line. Because status-go is asynchronous this is not possible and the follow up code will have to be done in a callback. However this is mitigated by the fact that these kind of queries are mostly used in messaging code which is moved to status-go anyway. Further work in the PR removing realm for status-react should uncover the potentially remaining use cases, but there performance impact is expected to be minimal.
- the login phase will be simplified:
- currently it requires starting a node as well as a bunch of back and forth between go, realm and js thread. This is due to the fact that js thread needs to start the node, decrypt the realm db to get the list of accounts, try do decrypt the account db, login in status-go, restart the node with the right account config.
- once accounts data is moved to status-go, getting the accounts list and login only consists of a native binding call to status-go to login. This reduce the load on js thread as well as the back and forth required to login.
- moving the messaging is already part of v1 scope but it is good to remind that messaging is one of the few remaining bottlenecks of the app even on high end devices
- the data could be exchanged with status-go in transit format to make the conversion an order of magnitude faster https://swannodette.github.io/2014/07/26/transit-clojurescriptas and save cycles on the UI thread.
Security issues with moving data from realm to status-go
- the security risk has already been taken as some part of the app like installations was already moved to status-go
- status-go already handles the keys and there is no more sensible data it could possibly work with
- sqlite + sqlcipher is more battle tested than realm and is a SQL database
- realm is 1000 lines of dependencies (in yarn.lock file, not LoC which is much higher!), removing that is a security benefit not issue
- realm is blocking removal of the ubuntu-server in status-desktop which is a security risk we have been looking forward to remove for a while
We could remove realm later, or bit by bit
Removing it after unfortunately is not possible without a break in compatibility of installed versions.
Removing it bit by bit would be a better strategy, essentially migrating each piece of data bit by bit (of course no migrations is better, so a clean break is easier).
But eventually to get all the benefits you would have to actually remove Realm, and that means that users that go from a version with Realm say v1, to a version without , say v1.5, will lose their data (they could probably keep their data by installing v1.4 first, and then v1.5, so there’s an upgrade path, but it’s not the smoothest).
So it only makes sense to remove it now before V1, considering there is a consensus on the fact that the time for a breaking change is now or never.
We could move as much as possible before v1 and then consider it acceptable for users to lose things like extensions and browser data:
That is not worth the trouble because moving extensions and browser data is by far the simplest thing in this proposal, it is at most a week of work for the two of them together.
How do we decide what information should be persisted in status-go? For instance what about UI related data like opened window?
If opened window is an information that has to be persisted, then the persistence is done in status-go. status-go will replace realm, no exception.
An absolute rule is that no migration should cross the status-go/status-react api boundaries to avoid any issue with migrations in the future. This means that no migration of persisted data should involve status-react.
status-go is now responsible for managing persisted data, and it is exposed to status-react via services APIs.
As a consequence this means that it is also harder for status-react to make changes to the persistence layer, since go has now to be involved. However that also means that such changes will be discussed more carefully in the future in the form of API updates.
We still don’t know how many things we need to change in status-go
That is the purpose of this document. Besides messaging which was planned for v1 anyway, there isn’t much left to do, and if we follow this plan, we should know much more by the end of next week.
The plan
STEP 1: Addressing lack of benchmark for raw performance of using status-go vs realm
Procedure
- Create a
browser table in status-go and an API that offers the same capabilities as its realm counterpart
- Schema of
browser table in realm js is as follow:
{:name :browser
:primaryKey :browser-id
:properties {:browser-id :string
:name :string
:timestamp :int
:dapp? {:type :bool
:default false}
:history-index {:type :int
:optional true}
:history {:type "string[]"
:optional true}}}
Array of string can be replaced by string if there is no equivalent in sqlite, but status-go has deliver it as a json array of string to status-react
- The API for the benchmark should include get-all-browsers which will return an array of browser, and add-browser which will add the browser passed as argument to the table
- Optionaly status-go can take and return transit data
STEP 2: Removing realm and identifying potential painpoints
- Estimation: 1 day
- Proposed participants: @yenda
Procedure
Consequences
- no persistence until status-go persistence is added
STEP 3: POC of status-go accounts database and simplified login
Procedure
- create a database on status-go side to manage accounts
- add native bindings to get accounts and login/logout from an account
- login should: start a node with the user config
Next steps
If
- Step 1 is demonstrating no big raw performance differences between status-go and realm,The plan
- Step 2 didn’t identify blockers for moving from an asynchronous to a synchronous db
- Step 3 made reasonnable progresses (possibly completed and went beyond expectations)
Then we’ll fully focus on removing realm and breaking migrations and protocol backward compatibility for V1.
See ## References for the list of endpoints that will have to be implemented.
There is a few places where we already know we can make shortcuts for instance:
- when importing a key, status-go will already have access to all the default and user added tokens from it’s multiaccounts database, so it can directly check the balance of all the path and return the ones with a balance to status-react, as opposed to status-react having to trigger that
- multiaccounts is not yet persisted in status-react, so if we know we are moving away from realm, we won’t double the work and hence reduce the scope
Reference
Current API for realm queries/writes, that we will have to implement in status-go.
Will be refined after step 2
[NEW] means it doesn’t exist yet in status-react either
Multiaccounts [NEW partly step 2][required]
Estimated 2 weeks
-
previously status-react was doing login using the address of the account
-
with multiaccount status-go should use a unique account identifier and pass that identifier in the account list to status-react
-
this would be an unsolved problem in status-react in the future when we start having multiple chat accounts
-
Read
-
Write
- save-multiaccount
- delete-multiaccount
- save-account-settings
-
a new database to list all accounts is needed
-
will contain the basic information required to show a list of accounts and each account settings.
-
will be encrypted by the app
-
individual multiaccount settings could later on be encrypted per multiaccount, but currently status-react doesn’t do it either. If that doesn’t require much more work on status-go side, I would strongly suggest to do it already.
New login [NEW partly step 2][required]
Estimated part of ### Multiaccounts
With multiaccounts the login would work slightly differently. status-react will provide the multiaccount id and status-go will have to do the following things:
- signal an error if the password is wrong or something went wrong or
- start the node based on the multiaccount configuration and
- load the chat accounts and start listening for new messages and
- load the wallet accounts and start watching for new transactions
Browsers [required partly step 1]
Estimated 2 days
- Read
- Write
- save-browser
- remove-browser
Dapp permissions [required]
Estimated 2 days
- Read
- Write
- save-dapp-permissions
- remove-dapp-permissions
Part of moving messages to status-go:
Contacts
Putting this one first because it is a bit special, technically not needed for messages and chats though required for features like block user to work smoothly
- Read
- Write
- save-contact
- save-contacts
- delete-contact
- block-user
Contact recovery
Installations
- already in status-go
- remove from status-react completely?
Mailservers
-
this should be mostly removed from status-react
-
gaps would be sent along with chats
-
mailservers should be provided along with account as a setting
-
mailserver-topics should be completely handled on go side
-
Read
- get-all-chat-requests-ranges handled by status-go
- get-all-gaps becomes part of get-all-chats
- get-all-mailservers becomes part of account settings
- get-all-mailserver-topics handled by status-go
-
Write
- save-mailserver
- delete-mailserver
- save-mailserver-topic
- delete-mailserver-topic
- save-chat-requests-range
- save-mailserver-requests-gap
- delete-mailserver-requests-gaps
Messages
-
should be part of status-console-client already
-
only mentionning methods that are needed and might not already be part of console-client yet
-
signal for changing outgoing message status
-
Write
Transport
- should be part of status-console-client integration already
- only mentionning methods that are needed and might not already be part of console-client yet
Chats
@cammellos currently working on that
- Read
- Write
- save-chat
- clear-chat-history
- deactivate-chat
- add-chat-contacts
- remove-chat-contacts
