[Proposal] Fix the transactions list

In the context of tribute to talk we are relying on token transactions from the wallet but unfortunately it doesn’t work very well at the moment.

How does it work:

  • when the user logs in a polling loop starts with a 15 second cycle.
  • every iteration the etherscan api is hit for the list of transactions
  • every iteration an event chain gets the current block, then queries eth logs for incoming and then outgoing token transactions, then groups transactions per block and then requests block info for every block containing token transactions.
  • every iteration all found transactions and token transactions are merged into app db
  • refreshing transactions list shortcuts the loop

What is bad about it:

  • the condition for querying etherscan and ethlog is odd and rarely true, which is why token transactions don’t always appear in the wallet
  • there is a 100000 last blocks limit on token transactions, which is why older token transactions never appear in the wallet
  • every time the user logins everything needs to be refetched, there is no persistence
  • it is expensive in CPU and network resources
  • it relies on 3rd party services

What we should do to improve:

  • get rid of the loop entirely as the polling us made unecessary by some existing rpc methods
  • persist transactions to avoid requesting everything everytime users logs in
  • remember oldest checked block and query whole history iteratively over time
  • remember latest checked block to only query from there
  • use ethlog filters instead of ethlog and receive new transactions as signals instead of polling
  • rely on a signal for new current block to update number of confirmations dynamically in subscriptions

What we need from status-go to be able to do that:

What will it bring?

  • less third party services (no more need for etherscan, and infura not needed in full node mode)
  • less bandwidth consumption
  • more reactivity for pending/incoming transactions
  • more reliability for transactions list
  • better performances as some of the work done in this loop is expensive
  • code much easier to reason about

Time estimate:

  • one week 2 go devs (implementation and review)
  • one week 2 clojure devs (implementation and review)
  • two days QA

Nice idea, without looking at the code in any detail the steps makes sense to me.

Just want to give some pushback on the time estimate: would this really take 4022=160h engineering time? Naively it seems like it can be done quite a lot faster than that.

1 Like

I’m trying not to be overly optimistic in Dev estimate here.

I don’t know enough about the status-go side but I suspect implementing rpc websockets might be slightly challenging.

On Clojure side there is a lot of work:

  • delete most of current code
  • generalize current ethlog rpc call, and make a ethlog filter version
  • getting rid of etherscan means renouncing to a lot of non hex encoded data about transactions so transitioning will imply getting a lot of hex conversions, it might be worth a more general converter that would reuse the work on about spec
  • handle signals for current-block and new filter matches
  • implement persistence of transactions
  • bug fixes

So I am not sure about status go side of things but for the Clojure side I don’t think it is overestimated. And I think it is really worth it because right now the current transaction list is an unreliable waste of bandwidth which is a shame for a wallet app. Also as mentioned the package includes some developments that will be reusable elsewhere.

1 Like

I think we have these methods implemented in status-go already, they were needed for some DApps, so can you double check if they are available or not.

What we didn’t do is to expose them in a push (instead of polling) way.

Also i noticed that each time we open wallet we request in a loop all tokens balances from their contracts, and prices from cryptocompare, would be great to move all that to status-go as well, similar how we try to do with messages, so instead polling or request all contracts, we could just request wallet balance from status-go

@andrey that is what this thread is about, eth_getFilterChanges allows you to get the latest changes in registered filters for eth_logs and new blocks mined, and it needs to be sent by status-go as a signal.

@igor I will test to day, if it is already working then all is left to do is a signal so that we don’t have to pull for it.

1 Like

@andrey Do you think this could be why the wallet is slow to load for some users?

Also @yenda if we are to persist transactions, remember oldest checked block and eliminate reliance on etherscan, are we making it easier, harder or neutral to make improvements to tx history UI?

Complaints about our confusing/defunct tx history were one of the top concern’s out of last week’s wallet calls. I can not currently trust my Status wallet like I can my bank account, because the tx history is too difficult to read and too unreliable.

Among other issues, gas shows up as a 0 ETH transaction, which is confusing. Transactions can be very slow to appear. There’s no user-friendly information about the contract/user transacted with. Compare this with MetaMask, where they recognize certain contracts and even methods to a degree:

I would say neutral or slightly easier as we won’t have to rebuild the transaction history from scratch every startup, which means we can add up some logic on top of transactions that won’t impact transactions as much. Examples of such logic would be:

  • if a transaction has 0 eth and some data, we show it as a contract call and try to decode the data in a human readable form
  • if a transaction has 0 eth, some data but no log we can assume that the underlying contract execution failed. for erc20 tokens that would usually mean you didn’t have enough tokens or you didn’t pay enough gas

Regarding performance I think it might affect the wallet slightly more than the rest of the app as there is a forced update of transaction list when going there. It is beneficial overall anyway because again:

  • you don’t fetch whole history over and over again
  • you don’t have a loop running but rather just listen to updates from status-go

The wallet will then be more responsive and transactions much faster to appear. Currently if you are unlucky and poll for a transaction right before its block is mined you’ll only see it up to about 30 sec later (I’ll be surprise if it is more but it is already very long from a UX perspective considering most wallet would already tell you it’s been confirmed 3 times)

@andrey @rachel we should plan working on this next week, as a first iteration without status-go involvment we can:

  • persist transaction in db
  • ensure failed token transactions are shown as such
  • token history not limited to 2 weeks but fetching older history either over time or when user scrolls down. I would opt for the later, it will add a delay for the user when scrolling down the list of transactions similar to what happens in chat when you go further back in history, but the amount of API calls to infura over time would be much much smaller. 99% use cases won’t require checking out transactions older than 2 weeks, and most of those would be persisted anyway. So the idea would be that you have all history since creation or 2 weeks prior to recovery of the account and from then on all history is fetched and persisted, so if you go away for 2 months the 2 months are fetched when you relogin, similarly to what is done now with message gaps on mailservers.

If status-go can be involved we can get rid of the loop and use signals instead which will free up the js thread a bit more.

1 Like

@oskarth @jarradhope do you have any thoughts on this?

The suggestion fixes a newly documented issue (lack of history for token transactions), should offer performance improvements and also paves the way for us to improve Tx history UX.

sorry missed this, yes definitely seems alot better.

1 Like

the subscriptions were merged to status-go codebase: https://github.com/status-im/status-go/pull/1455
they should support the filters that are needed for this functionality