A Status API proposal

[yenda]

During the Basel meeting, the idea of moving the communication protocol to status-go was brought up again. Weither some code should be on go or clojurescript side is a recurring topic of discussion at Status. Historically the rule of thumb was that status-go should only contain patches required to work on mobile and everything else should be on status-react side. The idea was that it should be easy to swap ethereum implementation.

This post is a proposal for the definition of a stable, documented interface exposing Status primitives for managing accounts, wallets, settings, chatting, etc. intented to be used by status-react and third party developers to easily build on top of ethereum protocols.

Design principles

Stable, Growing API

The API, past the initial experimentation phase, should remain stable so that developers can confidently build on top of it (cf Rich Hickey talk on accretion https://www.youtube.com/watch?v=oyLBGkS5ICk). For instance, even if the underlying implementation of the chat protocol is changed, the API does not and third party developers only have to update the implementation of the status API to be compatible with the status network.

Extend web3

Whenever possible, the API will be exposed by extending web3.status object
We would have for instance: web3.status.chat with chat primitives such as addContact, sendMessage…

Implement as a library

status-api, written in go, would be included in status-go.

Client/Server analogy

status-react is the UI, anything that would be done server-side in a web application should be done in status-go/status-api. This also mean our implementation of the status API would be the goto requirement for someone who would want to develop a CLI or alternative UI for status that would be fully compatible with the status network.

Goals

Improve testing

Both sides of the API can be mocked to test them independently, with reproducible test to evaluate the impact of changes on performance, battery, and network consumption. This could make @lukasz job much easier.
Possible objective: have automated battery test for each commit of status-react and status-go

Improve performance

web3.status calls being higher level, this will reduce the interactions between status-go and status-react. We shouldn’t have cascades of callback anymore as it will happen on go side.
Many computations will be moved away from JS thread, enabling multithreading capabilities.
We would remove realm, which was a bottleneck for status-react, and rely on status-api for persistence.
@jan.herich could elaborate on that point.
Possible objective: no more lags in the application for chats with huge amount of messages

Reduce security risks

We would further reduce the amount of js dependencies, reducing the threat that they represent.
With the client/server approach, we could also minimize the attack surface of the remaining dependencies.

Make codebase more accessible

Such a change would make the codebase more accessible for both go and clojure developpers
Go is an easier language with more developers and traction for doing the kind of things that happen in the backend compared to clojurescript on react-native
Status-react will have less domain specific code and less backend-like code. React-native being already a niche in clojurescript world this will lower the barrier of entry for wannabe contributors.
Possible objective: reach X contributors for status-api and Y contributors for status-go by the end of Q3

Portability

We don’t know what mobileOS might come out tomorrow, with better security guarentees. With a well defined API, we make it easier for ourselves to develop for new plateforms.

Risk analysis/Feasability

Integrate status-api as a library for status-go

This was mentionned as the preferable way to do it by @adam

Extend web3 (Low risk)

This was already done for our whisper extensions

Replace Realm (High risk)

This should be clarified and tested as early as possible.

• Can we do everything we need with leveldb ?
• Can we encrypt the database for each account ?

Future work

Project management

• create swarm

Status-go

I suppose status-sdk could be use as a starting point ?
@adriacidre @pedro @adam

Status-react

• mock status-api for use in status-react
• create re-frame library for status-api
• trim status-react of functionnality now handled by status-api
• build status-react with mock of status-api

[cammellos]

To add to that,

the move will be iterative, pushing down bit by bit into status-go, so it would be good to lay down a step by step plan (if we decide to move the logic to status-go, that is).

The best starting point to me it seems moving the transit implementation, as it’s already at the boundaries with status-go, and take it from there.

Other questions/concerns:

We know there is at least one thing that needs changing to allow that (generation of message IDs is clojure specific, so we’d have to change that), but are there other things that you can think of that might need changing?

As far as I know LevelDB does not support secondary indexes and no encryption, are there extensions that we can use to enable this behavior? Secondary indexes might not be necessary as we can potentially code the information we need into the key (i.e. chat-id/timestamp/message-id), or denormalize , but is this the road we want to take?

[roman]

status-react is the UI, anything that would be done server-side in a web application should be done in status-go/status-api. This also mean our implementation of the status API would be the goto requirement for someone who would want to develop a CLI or alternative UI for status that would be fully compatible with the status network.

It would be great to elaborate boundaries here. What is “UI” here, what is “server-side”. It’s a bit vague for me.

We would remove realm, which was a bottleneck for status-react, and rely on status-api for persistence.
@jan.herich could elaborate on that point.

This can be done without moving any app logic to go side.

We would further reduce the amount of js dependencies, reducing the threat that they represent.

Which exactly JS dependencies except for realm (which can be removed without this transition) will be reduced? Can’t this be done other way?

Go is an easier language with more developers and traction for doing the kind of things that happen in the backend compared to clojurescript on react-native

this argument is very subjective and quite far from being convincing, IMO. Also, it mentions only one side of the coin.

web3.status calls being higher level, this will reduce the interactions between status-go and status-react. We shouldn’t have cascades of callback anymore as it will happen on go side.

do we have any data which would prove that these interactions are the bottleneck atm?

Many computations will be moved away from JS thread, enabling multithreading capabilities.

again, this can be handled this on clojurescript side as well

[cammellos]

@roman

It would be great to elaborate boundaries here. What is “UI” here, what is “server-side”. It’s a bit vague for me.

I’d say the boundaries at the bare minimum is sending and receiving messages (shh filters management etc).

The problem is that sending messages is stateful (and it will be even more when we implement a double ratchet), so the API itself needs to have some state management (which programming language it is it’s not really relevant, I do see an advantage in using go just because is there, but clojure would also do).

I’d be happy if we could extract the core logic of sending messages in a separate library, and interact with it through a set of API calls such as getMessages(chat-id, from, to, limit), subscribeToPublicChat(chat-id) in the UI, something on those lines.

[divan]

This is a great idea. I was more than once willing to build alternative UIs for the chat (one using C++/Qt, second using terminal UI library, to chat from term) and realized that it’s impossible with a current setup without reimplementing chat layer from scratch.

One question regarding Implement as a library part — what exactly does it mean?. status-go already is implemented as a library (from the React-Native perspective, I mean). Chat protocol should run on top of existing whisper/devp2p stack, so this library cannot be separated from status-go itself. Does it make sense?

Another thing is the serialization interface to use in for this API. We currently use JSON whenever possible, but as you mentioned problems with “chats with huge amount of messages”, maybe there is a sense to explore faster and smaller encoders/decoders? Anyway, that would be way easier to research and benchmark on a Go side.

[jan.herich]

Generally, I like the Idea, but I have few concerns, things I’m worried about:

@yenda

Improve performance

web3.status calls being higher level, this will reduce the interactions between status-go and status-react. We shouldn’t have cascades of callback anymore as it will happen on go side.
Many computations will be moved away from JS thread, enabling multithreading capabilities.
We would remove realm, which was a bottleneck for status-react, and rely on status-api for persistence.
@jan.herich could elaborate on that point.
Possible objective: no more lags in the application for chats with huge amount of messages

While there is potentially huge benefit in RN App not having to care about low level persistence at all and just calling “server” API to do it (think updateAccount, getContacts), I don’t think that such “method” oriented API is the right approach for flexible, modern UI, especially the read part (queries).

In the modern web development, people are gradually moving away from such approach (mostly done via REST routes) towards data based data based queries (GraphQL, OmNext…). The huge advantage of such model is ability to declaratively express which data the client side needs, without any or very little changes in the server API.
This is great for mostly 2 things:

1. Different clients needs different data (mostly amount of them), one can easily imagine that listing of contacts for cli client will probably need less data then listing of contacts for mobile app/desktop app. We don’t want to force clients to load unnecessary data and strip them later, and we also don’t want explosion of API routes/methods like getClientsWithFullData, getClientsWithShortSummary, etc.
   Data based queries solves this in a nice way and each client can express which data is needed, like [{:clients [last-name first-name birthdate photo summary]}], or [{:clients [last-name birthdate]}] without any new endpoints.
2. We don’t want the server API changes to become bottleneck - It’s true that currently, we don’t really have anything like data based queries in the application, but since there is no server/client interaction it’s not nearly as problematic as when we would split the current interaction model into server/client.

GraphQL
Om Next

[divan]

@jan.herich I agree with your point, but I’m afraid this is simply not a case for us. GraphQL solves a specific problem – mitigating overhead on keeping in sync backend and frontend in the case of large and versatile API surface and only when frontend is updated much more frequently.

Think about Facebook UI case – they experiment with UI often, introduce new elements, that require new endpoints, and while updating UI might be easy, deploying updated servers could be longer – there is a visible asymmetry here and GraphQL solves exactly this. I don’t think this is a case for our chat API.

Moreover, we’re not going to change chat API on daily basis – ideally, it should be small and simple. Even if we’ll have an explosion of “different clients”, it’s not like they will need vastly different responses and types of data from “backend”. They will want to use 99% of the same endpoints.

Plus, the data we deal with is really small. Overhead of sending “birthday” field (via in-memory!) is really negligible compared to implementing GraphQL like logic complexity.

[adam]

@divan

One question regarding Implement as a library part — what exactly does it mean?

To implement it as a standalone library, separate from status-go project. There is a couple of motivations behind this:

• status-go has a lot of legacy code and bad testing practices which would be inherited when writing status-api as a subpackage,
• allow status-api to work with various clients as long as they support required protocols,
• smaller codebase, easier to test and benchmark,
• more options how to design the application: embed status-api in status-go or link status-go and status-api with status-react during compilation.

---

The biggest problem now is that I can’t play with the messaging protocol without running the app. Even if we don’t decide to write it in Go, it should be a totally separate library that I can wrap in a CLI or whatever and use with my geth node supporting required protocols.

From my point of view, the most important are:

1. 100% unit tests coverage and correctness of the messaging protocol, free of race conditions,
2. Ability to test and benchmark the library with and without mocks locally (without mocks means using a local Ethereum cluster with e.g. Whisper nodes; the case is to make sure that the protocol makes sense and not worry about whether it’s Whisper or something else underneath),
3. Clear interface understood as a set of methods and data types. I like protobuf files but Go interface type and structs should be easily readable for everyone as well.

I suppose status-sdk could be use as a starting point ?

I don’t think so. There was never a real discussion about the architecture, transport, serialization etc. It should be designed from scratch.

Extend web3

Isn’t web3 a bottleneck? It only provides a one-way communication and uses polling for many things that we need to bypass with signals. Shouldn’t we think about a full-duplex communication channel like WebSocket? At the same time, it’s a lower priority than a clear interface.

---

I also don’t think that GraphQL has a place here. GraphQL might be a good solution to explore the blockchain (@dmitrys posted a link to Introducing The Graph. Last month we announced The Graph and… | by Yaniv Tal | The Graph | Medium) but in our case, the traditional RPC model should work better. The question is whether it should be through HTTP or some full-duplex channel like WebSocket.

[divan]

@adam

To implement it as a standalone library, separate from status-go project

I would love it to be a case, but how is it going to work? How will it function if it’s not a part of status-go? I.e. let’s say the client calls a function SubscribeToPublicChat(id) – what happens next in this library?

I completely agree with the rest of the points.

[divan]

Just realized that having chat protocol abstraction at Go side will allow us to test PSS protocol (alternative to Whisper) much easier. Even allow switching dynamically if needed.

[adam]

let’s say the client calls a function SubscribeToPublicChat(id) – what happens next in this library?

Two options:

1. status-api will be a dependency of status-go and it will be instantiated with a Whisper service and whatever is needed,
2. Allow to instantiate status-api via C bindings and somehow allow communication with status-go.

I would opt for (1). At the same time, status-api will have a CLI version which will be able to communicate with clients using JSON-RPC via IPC/HTTP/WebSocket. (2) seems super complicated and limits status-api to our very specific case.