IPFS alternatives (SNT utility)

rachel · May 20, 2019, 1:58pm

As I mentioned on core dev call #18, we’re having issues with IPFS storage for SNT utility features. Chu’s had trouble getting Tribute to Talk to work in 90% of cases.

I learned from @barry that he’s also having issues publishing to IPFS, so am creating this thread to track how we fix this.

Problem
The IPFS gateway we use is unreliable.* Any feature for which we’re storing data on IPFS is affected—Stickers, Tribute to Talk and the liquid funding DApp.

*Could be something to do with Infura API blocking certain IPs?

Potential solutions

Host our own IPFS gateway
Risks are that anyone could upload a large file and bring this down. Accessing data via Cloudflare may lead to slow load times. And also IPFS’ hashing algorithm is not deterministic.
Use Swarm feeds instead
In their latest release, Swarm has fixed bugs that made it an unviable option in the past. @yenda and @jarradhope will be with the team in Madrid for their hackathon this weekend and can test/ask questions.

Current plan
The TtT team is going to experiment with Swarm further (@yenda) and check in with @jakubgs about the set-up of our own IPFS gateway—this sounds like ongoing work. TtT team will discuss both options again after the Swarm hackathon this weekend.

Notes from our call here; updates to come.

jakubgs · May 20, 2019, 3:25pm

I can definitely look into deploying our own IPFS gateway, but I assume it would have to scale.

Could you give me a rough idea of how much traffic it would have to handle initially? Just a ballpark figure for numbers of requests would be nice.

yenda · May 20, 2019, 3:33pm

It would say it should be similar to mailservers in terms of numbers of requests, except they will be much smaller. However we might also or instead want to deploy a swarm gateway with our own swarm cluster.

barry · May 20, 2019, 4:03pm

It should only need to scale if we expect to upload a lot of content frequently to the nodes. In theory content that is frequently called will start to cache around the network on other IPFS nodes and will even reduce demand on our nodes as other nodes that are closer to requesting clients will deliver the content.

There should be a good amount of info on setting up a cluster here: https://cluster.ipfs.io/

jarradhope · May 20, 2019, 4:08pm

I also want to point out that IPFS multihashing function is non-deterministic depending on the DAG generation when chunking. So if we verify content downloaded (and we should [by running the hashing function locally and seeing if it matches with the query, combined with filesize in manifest]) then we’ll get false negatives.

This for me is a another strike against ipfs as you can’t really be sure what your asking for is what your getting, which to me almost defeats the point, will be asking how swarm tackles this (I suspect it probably isn’t an issue because not many implementations, but I’m curious if it is well specified)

jakubgs · May 20, 2019, 7:10pm

There already is one:

Thanks!

arnetheduck · May 20, 2019, 8:19pm

link? Post must be at least 20 characters, imagine that.

jakubgs · May 20, 2019, 8:35pm

You could just embed some fitting meme image, it counts as text.

yenda · May 20, 2019, 9:13pm

Is it updated to swarm 0.4 already?

barry · May 21, 2019, 12:53am

Is there a working browser client that can upload to swarm and then disappear from the network without having the content garbage collected?

yenda · May 21, 2019, 8:44am

The content is garbage collected by removing the chunks that have not been requested for the longest.

For feeds it should be safe because they would be checked everytime the user logs in or someone checks the user profile, and even more if we use it for things like public profile in public chats.

jakubgs · May 21, 2019, 11:35am

We use the ethdevops/swarm:latest image:

github.com

status-im/infra-swarm/blob/master/ansible/roles/swarm/defaults/main.yml#L2


      
          ---
          swarm_cont_version: '0.5.7-alltools'
          swarm_cont_image: 'ethersphere/swarm:{{ swarm_cont_version }}'
          swarm_cont_name: 'swarm-node'
          swarm_cont_vol: '/docker/{{ swarm_cont_name }}'
          # listen port which will be incremented for every dontainer
          swarm_cont_port: 30399 
          # address to listen on
          swarm_cont_addr: '0.0.0.0'
          
          # permissions
          swarm_cont_uid: 0

Let me verify, for example the container on node-01.gc-us-central1-a.swarm.test was created 13 days ago, and is:

 $ docker run --rm -e PASSWORD=123qwe ethdevops/swarm:latest --version 
...
Running Swarm:
Swarm Version: 0.3.5-stable Git Commit: 8bbe72075 Go Version: go1.11.1 OS: linux
localhost swarm version 0.3.5-8bbe7207

Nope, looks like latest points at 0.3.5: Docker
There is an explicit v0.4.0 tag, but using it would stop automatic updates.
I guess we could use edge if we want to be edgy.

jakubgs · May 21, 2019, 11:44am

Oh wait, let me retract that. I verified this on node-01.do-ams3.swarm.testand it’s 4.0 there:

[email protected]:~ % d exec -it swarm-node /swarm --version
swarm version 0.4.0-c3b317a4

That one was updated 22 hours ago. It appears the GC host in the swarm has run out of space, despite having --store.size specified, which probably blocked the container updated. I’ll fix that.

jakubgs · May 21, 2019, 11:57am

Okay, now all hosts are running 0.4:

 $  ansible all -o -a 'docker exec swarm-node /swarm --version' 
node-01.gc-us-central1-a.swarm.test | CHANGED | rc=0 | (stdout) swarm version 0.4.0-c3b317a4
node-01.do-ams3.swarm.test | CHANGED | rc=0 | (stdout) swarm version 0.4.0-c3b317a4
node-01.ac-cn-hongkong-c.swarm.test | CHANGED | rc=0 | (stdout) swarm version 0.4.0-c3b317a4

oskarth · May 21, 2019, 2:50pm

@rachel Do we have a Github issue for the IPFS problems? I can’t find a simple problem description or debugging trail for what we tried, if it is a specific gateway problem or IPFS general problem or whatever.

Update: No issue exists, conversation continued in #status-core-ui and https://github.com/status-im/status-react/pull/8129. Issue created here IPFS Gateway reportedly unreliable (PLACEHOLDER ISSUE) · Issue #8251 · status-im/status-mobile · GitHub

jarradhope · May 21, 2019, 5:58pm

github.com/ipfs/kubo

Content added via mfs and ipfs add have different hashes

opened 02:55PM - 17 May 18 UTC

closed 05:05PM - 18 May 18 UTC

achingbrain

add: ```sh $ echo 'Hello, World!' > hello.txt $ ipfs add hello.txt added Q…mYAXgX8ARiriupMQsbGXtKdDyGzWry1YV3sycKw1qqmgH hello.txt $ ipfs object data QmYAXgX8ARiriupMQsbGXtKdDyGzWry1YV3sycKw1qqmgH | xxd 00000000: 0802 120e 4865 6c6c 6f2c 2057 6f72 6c64 ....Hello, World 00000010: 210a 180e !... ``` mfs: ```sh $ echo 'Hello, World!' | ipfs files write --create /hello.txt $ ipfs files ls -l hello.txt QmPbFXUdgtyrCjExNAjugJTro6Lx4GG4MuvXTiHcZQkiif 14 $ ipfs object data QmPbFXUdgtyrCjExNAjugJTro6Lx4GG4MuvXTiHcZQkiif | xxd 00000000: 0802 180e 200e .... . // huh, weird, only dag-pb bytes in the data field $ ipfs dag get QmPbFXUdgtyrCjExNAjugJTro6Lx4GG4MuvXTiHcZQkiif {"data":"CAIYDiAO","links":[{"Name":"","Size":22,"Cid":{"/":"Qmd8q2S2ByCjpRxQvYNrHhC3C6oZ9N9CEbtCxnwakgw6ea"}}]} // ok, let's look at the linked node $ ipfs object data Qmd8q2S2ByCjpRxQvYNrHhC3C6oZ9N9CEbtCxnwakgw6ea | xxd 00000000: 0800 120e 4865 6c6c 6f2c 2057 6f72 6c64 ....Hello, World 00000010: 210a 180e !... ``` So `ipfs add` gets you one DAGNode with type `file` that contains the data: ```sh ipfs object data QmYAXgX8ARiriupMQsbGXtKdDyGzWry1YV3sycKw1qqmgH | unixfs {"type":"file","data":{"type":"Buffer","data":[...]},"blockSizes":[]} ``` But `ipfs files write` leaves you with an empty DAGNode of type `file` that contains a link to a `raw` DAGNode that contains the data: ```sh $ ipfs object data QmPbFXUdgtyrCjExNAjugJTro6Lx4GG4MuvXTiHcZQkiif | unixfs {"type":"file","blockSizes":[14]} $ ipfs object data Qmd8q2S2ByCjpRxQvYNrHhC3C6oZ9N9CEbtCxnwakgw6ea | unixfs {"type":"raw","data":{"type":"Buffer","data":[...]},"blockSizes":[]} ``` Does anyone know what the thinking behind the difference in structure was?

DAG Builder isn’t specified, multi dag builders for same content! hooray!, but client doesn’t know what dag builder was used so have fun verifying

petty · May 23, 2019, 1:19pm

This seems like such an oversight…

yenda · May 24, 2019, 4:54pm

Could someone on go-team push the timeout for the feed chunk exploration to 5 sec and make a docker image that we can use for our swarm cluster? @igor
I discussed with the person that developed that algorithm and the current 1 sec is arbitrary and has an error rate that is too high

igor · May 27, 2019, 9:15am

@adam or @pedro can you take a look, it looks like a super small thing to do?

rachel · May 28, 2019, 2:28pm

Update on the situation regarding Tribute to Talk:

To ensure stability for the MVP, we’ll revert back to using the “dumb” version of this feature and storing tribute amount in the contract (at some cost to user: Tx required for setting + privacy sacrifice).

Eric’s feedback on using Swarm for a public profile proof of concept is here.

We agreed it’s best to experiment with allowing users to store ‘lesser’ elements of their public profiles on Swarm, such as their photo. Pending how that goes, we could then start incorporating SNT utility features such as ENS name and TtT setting in the public profile as well.

@jakubgs this means that we won’t need the IPFS gateway for this particular feature. From my POV you can pause work on that, but as IPFS issues have also affected @barry and @cryptowanderer and will probably impact sticker market as well, I don’t believe it’s a wasted effort. Following up on how we’ll ensure reliability of stickers in #status-core-ui.