Transfer contacts between multiple devices idea

Messenger
1

In Prague, we’ve talked about options to synchronize chats/contacts/history across multiple devices (i.e. between Status Desktop and Status Mobile). Many of us are reinstalling nightly builds daily, and sometimes it requires re-creating account, so all the chats/contacts are lost, and we need to add it again, just to repeat the process two days later.

Proposed solutions

The proper solution, as we see it, would be to store all that information in encrypted form in decentralized storage, like Swarm and IPFS. The problem with it is in the current state of both – neither Swarm nor IPFS are supported, and the implementation of light Swarm client is still in progress.

One suggestion has been made during Status Desktop meeting to adopt an approach similar to Whatsapp Web. Quick recap: web/desktop versions show a QR code, which user needs to scan with already authenticated Whatsapp application on the mobile phone, and desktop version automagically syncs with the mobile app. However, the underlying mechanism heavily depends on a centralized server. Basically, it’s web app opens WebSocket connection to the server, generates a unique key and provides this key via QR code to the mobile app. The mobile app, upon receiving the key, contacts server and asks to transfer account’s messages and data to the WebSocket associated with that key. More details here.

But the idea of using one authenticated and configured app to quickly bootstrap another one is tempting, and actually, QR codes might be good enough to do the whole job alone.

Using QR codes for data transfer

An idea is based on two premises:

  • amount of data to sync is relatively small
  • every phone is enabled with a camera nowadays

In essence, the idea is to make one device (Status Desktop, for example) to take all data need to be synced (contacts and chat names), encrypt it and encode into the series of animated QR codes. Another authorized device (Status Mobile) open up camera, scan QR codes, and get the whole payload, decode and decrypt it, and will use it as a bootstrap values for an empty account.

Let’s crunch the numbers.

Data to sync

Assuming we want to sync only contacts and chats, let’s put a pessimistic average of 50 chat groups and 100 contacts. Chat groups can be identified by their string representation (i.e #status-core-desktop) and contacts by their keys hashes (65 bytes or 130 hex characters). Let’s assume the average chat name is 20 chars long.

Summing this up: 50 * 20 chars + 100 * 130 + data schema overhead (5 chars per each entity = 150 * 5 = 750) = 1000 + 13000 + 750 = 14750 = 14.4KB.

Now, 14.4 KB of ASCII characters, with HEX subset being the majority of the content, it should compress pretty efficiently (to 1K or less), but QR codes encode asciinumeric and binary differently, so the benefit of compressing and feeding binary to QR encoding is under question.

QR Code Encoding capacity

QR codes have version range from 1 to 40. Version 40 offers 177 rows max and 31K pixels. With high error correction rate it offers 1307 alphanumeric bytes capacity, and for low error correction mode – 3009.

That’s how QR code version 40 for 1276 bytes looks like:
tCz0X

It looks almost like a noise, but my smartphone camera can perfectly read its “Lorem ipsum…” content. In order to fit 14.4K of data, we’ll need at least 12 such frames with QR codes. To adapt it to average consumer screens, we’ll have to choose less QR version or decrease error correction level.

Animating QR codes

Now, the idea is to display multiple QR codes in a sequence, probably in a loop to allow client code to restore missed frames if any. On 15 FPS this will give us data rate around 1307*15=19.1KB/sec which is more than enough to transfer all the contacts.

The idea is not new, there are projects exist to demonstrate this approach. Here is a good example with quite an old device and camera:

They claim to have even higher throughput, ~43KB/s.

I believe we can experiment even with higher fps, that will allow us to choose smaller QR version.

Alternatives

The animated QR approach seems to me to be the easiest one to implement, as QR encoding implementations are abundant and battle-tested. One can use native mobile OS QR reading API (in Status-React, for example) to feed data frame-by-frame to the library in status-go to decrypt and decode data. However, it’s worth to note the alternative options, which I outline briefly.

  1. Colored QR codes
    It’s not widely implemented, but QR codes can use colours to increase the encoding ratio. Most of the color QR codes use color simply for aestaetic look, but it’s possible to encode additional data with color.

By the way, QR codes should not necessarily be black&white and boring. We can have this, for example:
generate-2

  1. Specialized visual coding protocols
    There is some amount of literature and implementations of non-QR code based visual code data transfers, such as ThruGlassXfer or coding designed for aircrafts. Not to mention another 2D barcode matrix coders.

  2. Custom solutions similar to Apple’s Particle Cloud paring mechanism used to sync with Apple Watch. It’s patented and implementation details are super cool:

FInal thoughts

I personally think the naive QR code animated approach would be quite easy to implement. But I’d love to hear thoughts and criticism of this idea. I do understand limitations of it, but it can be seen as a temporary or even “fallback” solution after the decentralized storage one is implemented. Also, if it’ll proove to work great, it’ll offer much better security as there is no network involved at all.

5 Likes
2
1 Like
3

another option is to use whisper mail servers. while this is not as reliable and efficient as storing data in swarm/ipfs. reliability can be improved by verifying that data reached known mail servers. and efficiency, by generating backups every week or so. so that the reader doesn’t have to traverse all history to find recent backups. i realize that there are other technical questions but it seems like a viable option.

or we still can use a swarm gateway. i believe that we even discussed how to integrate it so that it will be transparent for an app. and we can swap it with light client when it will be available.

1 Like
4

I suggest using Identity contract, where the same Identity can have multiple messaging keys, one per device, and one device could request other to upload the content (contacts and chat history) to other, the handshake could happen through whisper and the upload/download could be done through a direct encrypted connection, such as UPnP. Status nodes also could provide this service.
Adding a key is done through the identity management, similar to Universal Logins proposal.

1 Like
5

Thanks for the post, excellent work and very interesting!

Just a few things about ipfs/swarm:

To keep the devices in sync, a form of storage (other than the device itself) is not required, think of any distributed data store, to backup on the other hand, a form of storage i(other than device themselves), is advised.

I think we need to distinguish the two use cases that are conflated here:

  1. Backing up
  2. Syncing another device

The distinction is important as they have different properties, especially when it comes to encryption.

When you backup data, you are encrypting the data for a device you don’t know about (essentially the use case is losing your device and having to start from scratch), which essentially makes it inherently less secure. Also, in this case you might want to use a form of (decentralized) storage, other than the device itself for redundancy.

Syncing on the other hand happens between two devices that you currently (hopefully) own, and does not need any form of storage, in this case the encryption can be much stronger as you can encrypt data for that particular device only.

Currently we have already a way to sync messages and eventually contacts (https://github.com/status-im/status-react/pull/6658), simply the devices are paired and the syncing happens through whisper in chunks, much like the way you described it (as a interesting point, the photo is a lot of data, so much that we can’t sync many contacts at once as it would hit whisper size limit).

2 Likes
6

@dmitrys you’re right, I should have mentioned this.

@cammellos thanks! You’re absolutely right, backup and syncing have different properties. I’m not sure I completely understood why encrypting for the unknown device is “less secure” and for trusted device is “stronger encryption”. To me, it sounds like the opposite (less trust – more secure encryption is needed, more trust – less secure).

Awesome to see that work on sync via Whisper already has been started. That invalidates my proposal to use QR codes as a temporary solution, definitely.

However, I think we had another use case that can take advantage of it – it’s sending new/working bootnodes from one device to another – perhaps related mostly to Revolution mode idea: if one device has working peers info, and other have only peers/bootnodes that are blocked, we wanted to let people easily share peers IPs and enodes with other people.

@exiledsurfer thanks, great read. I don’t see, however, how it can be applied here :confused:

1 Like
7

@divan I see your point, here though it’s not the amount of trust you put on your devices, as you don’t have to make any concession (i.e. you want to use the strongest encryption that is available), but in case of unknown devices your options are more limited.

Essentially it boils down to the difference between this two statements.

When backing up:

Any device that the user will own in the future will/should be able to decrypt this data

When syncing on the other hand:

Device B (and device B only) should be able to decrypt this data

So just to give a concrete example:

backing up

If you are backing up data, most of the implementation use a single encryption key (in our case it would be the public key, or a passphrase).

If you use multiple keys for different chunks of data, which would increase considerably security, then likely you would give a file (in whatever shape or form, could be a qr code), to the user, as remembering multiple keys (they could be in the 100s), it’s not feasible, this file effectively becomes your single key, and can only be deleted when the backup is discarded/not needed. Deleting the keys is an issue here as it’s likely not stored on the device itself, so the software has no control, it’s the user that needs to delete the key if a new backup is generated.

Any of the keys generated this way is generated solely from entropy of the owner of the device being backed up (non-interactive).

syncing

When syncing, you can just use a ratchet to encrypt chunks of data, discarding decryption keys as soon as they are used. In this case entropy from both devices is used to encrypt the data, as it’s an interactive process.

Basically backing up is operationally more complex and part of the complexity is due to the fact that is more generic and its use cases are broader, syncing being narrower in scope.

2 Likes
8

Ah right, now it’s clear.

As for the QR idea itself, I actually decided to try it out and wrote a simple proof-of-concept solution to test with a real smartphone.

Here is a quick demo how it works:
https://www.youtube.com/watch?v=-MnegPaMkgE

The code is here: https://github.com/divan/txqr. Apart from the lib, it provides two command line tools:

  • txqr-ascii - for encoding file and rendering animated QR frames in the terminal, and
  • txqr-gif - which encodes the file into an animated GIF with QR frames

I used gomobile to build simple Swift iOS app that reads those animated QRs and decodes content (expects image for now). Apparently, iOS already can scan QR codes natively, so that’s a very straightforward solution. It just shows the resulting image and prints downloading/decoding stats, nothing more. Sources are here: https://github.com/divan/txqr-reader

So I experimented with different frame rates, QR code sizes, amount of bytes in each QR frame, and, while more experimentation needed, with this naive solution, I got the following:

  • the fastest result I got for downloading 15KB file was 3.1 sec.
  • the minimum acceptable FPS on my iPhone XS seems to be around 10FPS (7 is safer). The native iOS camera scanner seems to have higher latency (I saw numbers from 10 to 60ms per next returned frame with recognized QR code), so perhaps decoding code optimization will yield better results, but that’s what I got – the faster the animation, the more skipped frames.
  • too small amount of data in each frame results in huge encoding overhead and large download times. too large amount results in a ​higher probability that QR code won’t be recognized properly (due to motion, camera aberrations, etc) – each square converges to a single pixel, basically.

Here are my experiments’ results for two FPS values – 3 and 7:
53

So, my first conclusion is that this approach is definitely viable way to send some small pieces of data within a halted networking conditions. If this amount is less then 5KB, that’s almost perfect case. We might think about passing bootnodes info, for example.

For larger amounts of data (tens of KB or more), perhaps more optimizations are needed. And there is definitely a lot of space to optimize in my implementation.

3 Likes
9

That’s very interesting, never thought about using qr code like that. Nice idea.

Regarding sync, I am looking forward a solution that connects through network Status clients, so they can exchange any information, inclusive contact list.

I am linking this as it’s related to the same problem-solution of Status.app

1 Like
10

Animated QR code as a wireless protocol to transmit data is pretty cool. It feels much harder to eavesdrop on than anything else

11

I actually reimplemented txqr with fountain codes and did a couple of blog posts about it:

Pity that Status failed to create a culture of experimentation, exploring new ideas and knowledge sharing.

2 Likes
12

Wow that’s super cool!

13

I just saw this in action in a production app (Ledger Live for Android). They use it to transfer accounts from the Desktop app to the mobile app, avoiding the painstaking job of manually adding all accounts to the phone.