Performances: before moving communication protocol to status-go

1. Replace status-react side polling by signals based subscriptions for ethereum calls

  • new whisper messages and fetching of transactions are two exemples of actual react side loops that can be replaced by much more efficient subscription model.
  • the first MVP for this will be newBlockFilter which will send a signal everytime a new block has been mined, a very important information for amount of confirmations required.
  • filters for eth logs events and whisper will be coming next
  • ultimately these signal based feedback would allow us to efficiently replace web3 polling while having similar functionalities such as for instance promises on transaction receipts, which is useful in the case of token transaction failure which the current wallet doesn’t understand.

So this is a status-go feature that will not only benefit the communication protocol but other features as well

@igor will start this next week

2. Whisper filters and keys management on status-go

  • first we will move chats from status-react realm db to status-go sql db
  • this will allow filter and key management on status-go side
  • currently whisper filters and keys are requested on status-react side
  • every time user logs in he needs to recreate the whisper keys for public chats and register whisper filters
  • with these changes status-go will start the whisper filters and create the symmetric keys for the chats when user logs in
  • this will remove lots of back and forth between status-react and status-go on start up
  • this will remove polling loops for whisper filters as it will be replaced by signal based pushes for new messages as implemented in 1.
  • an api will allow the user to make chats queries with pagination

Required API

  • tbd
  • will probably involve a higher level API with some create chat primitives for the different types of chat which will manage the keys and filters accordingly

3. Move messages from status-react realm db to status-go sql db

  • second we will move messages to status-go in two passes
  • first the messages will be saved once they have been received in the filter, without looking into the communication protocol payload, only whisper payload, before they are even passed in signal to status-react
  • then status-react will save the “decrypted” payload through a call to status-go, which replaced the current save message in realm
  • an api will allow the user to make messages queries with pagination
  • this is the critical part of the MVP that will allow us to evaluate how much faster/slower status-go/sql db over IPC with json->cljs conversions is compared to realm-js over bridge with js-object->cljs conversions

Required API

  • add message data(id data)
  • get messages (chat, from, to, cursor, limit) -> (cursor, [messages])
  • delete messages (chat or user or id)

4. Move mailserver management to status-go

  • this will be the most efficient in terms of js-thread management as it will allow us to control how much messages are passed to status-react at startup time without clogging the thread
  • the whole message requests cycle will be handle on status-go
  • historical messages will be passed to status-react through signal in reasonably sized batches, with grouping per chat

Required API:

  • will require a way to fetch based on user action (fetch gap for instance), but otherwise mostly integrated with the chat api

5. Blacklist and whitelist management

  • for blacklist it is a simple list (or rather hashset for efficiency) of public-key, all messages signed by such keys are dropped
  • whitelist is a tribute-to-talk specific thing so it is only used if the user have tribute to talk activated and only for certain kinds of messages (requires looking into the payload).

Required API:

  • add/remove public key from whitelist/blacklist
  • use/don’t use whitelist (enabled/disabled ttt)

6. Move the rest of the protocol to status go

I don’t think it make any sense to start this step before the above, as they are much more granular and lower hanging fruit in terms of application performances

@andrey @igor

3 Likes

about 2)
https://github.com/status-im/status-go/tree/feature/leave-join-public-chats
I have already worked on status-go for that (haven’t moved the db, but state about filters is all handled in status-go), but did not finish as had no bandwidth. Still lot to do but might be a start.

1 Like

Speaking of this - we have whisper support in nimbus now but were discussing if the general whisper RPC API is optimal - any desiderata that would make it easier to use or redundancies that could be removed from it?

2 Likes

edit: investingating with repl it turns out realm terrible perfs here are indeed caused by the remote debugging. overall the transaction for joining #status] with a brand new account is "Elapsed time: 170.000000 msecs" which is still significant enough to cause a visible lag.

  1. will have a huge impact on performances. Here is a profile of joining #status chat for the first time (around 20000ms). The events basically cost nothing. The big red block is realm reads (message exists? for deduplication) and the big green one are realm writes (writing new messages to db).

Profile: https://ipfs.io/ipfs/QmPrryyNRo7AuAzPGXZxu6wuMwcDAywoc37DipJdcoX12b

note: I wonder if it isn’t the js remove debugging that cause such a use perf hit from realm because it has to communicate between chrome and the emulator?