In the context of tribute to talk we are relying on token transactions from the wallet but unfortunately it doesn’t work very well at the moment.
How does it work:
when the user logs in a polling loop starts with a 15 second cycle.
every iteration the etherscan api is hit for the list of transactions
every iteration an event chain gets the current block, then queries eth logs for incoming and then outgoing token transactions, then groups transactions per block and then requests block info for every block containing token transactions.
every iteration all found transactions and token transactions are merged into app db
refreshing transactions list shortcuts the loop
What is bad about it:
the condition for querying etherscan and ethlog is odd and rarely true, which is why token transactions don’t always appear in the wallet
there is a 100000 last blocks limit on token transactions, which is why older token transactions never appear in the wallet
every time the user logins everything needs to be refetched, there is no persistence
it is expensive in CPU and network resources
it relies on 3rd party services
What we should do to improve:
get rid of the loop entirely as the polling us made unecessary by some existing rpc methods
persist transactions to avoid requesting everything everytime users logs in
remember oldest checked block and query whole history iteratively over time
remember latest checked block to only query from there
use ethlog filters instead of ethlog and receive new transactions as signals instead of polling
rely on a signal for new current block to update number of confirmations dynamically in subscriptions
What we need from status-go to be able to do that:
on status-react this could transparently use private-rpc-call as it is already the case for other rpc-calls, the callback being called everytime there is some updates from the ws
Nice idea, without looking at the code in any detail the steps makes sense to me.
Just want to give some pushback on the time estimate: would this really take 4022=160h engineering time? Naively it seems like it can be done quite a lot faster than that.
I’m trying not to be overly optimistic in Dev estimate here.
I don’t know enough about the status-go side but I suspect implementing rpc websockets might be slightly challenging.
On Clojure side there is a lot of work:
delete most of current code
generalize current ethlog rpc call, and make a ethlog filter version
getting rid of etherscan means renouncing to a lot of non hex encoded data about transactions so transitioning will imply getting a lot of hex conversions, it might be worth a more general converter that would reuse the work on about spec
handle signals for current-block and new filter matches
implement persistence of transactions
bug fixes
So I am not sure about status go side of things but for the Clojure side I don’t think it is overestimated. And I think it is really worth it because right now the current transaction list is an unreliable waste of bandwidth which is a shame for a wallet app. Also as mentioned the package includes some developments that will be reusable elsewhere.
Also i noticed that each time we open wallet we request in a loop all tokens balances from their contracts, and prices from cryptocompare, would be great to move all that to status-go as well, similar how we try to do with messages, so instead polling or request all contracts, we could just request wallet balance from status-go
@andrey that is what this thread is about, eth_getFilterChanges allows you to get the latest changes in registered filters for eth_logs and new blocks mined, and it needs to be sent by status-go as a signal.
@igor I will test to day, if it is already working then all is left to do is a signal so that we don’t have to pull for it.
@andrey Do you think this could be why the wallet is slow to load for some users?
Also @yenda if we are to persist transactions, remember oldest checked block and eliminate reliance on etherscan, are we making it easier, harder or neutral to make improvements to tx history UI?
Complaints about our confusing/defunct tx history were one of the top concern’s out of last week’s wallet calls. I can not currently trust my Status wallet like I can my bank account, because the tx history is too difficult to read and too unreliable.
Among other issues, gas shows up as a 0 ETH transaction, which is confusing. Transactions can be very slow to appear. There’s no user-friendly information about the contract/user transacted with. Compare this with MetaMask, where they recognize certain contracts and even methods to a degree:
I would say neutral or slightly easier as we won’t have to rebuild the transaction history from scratch every startup, which means we can add up some logic on top of transactions that won’t impact transactions as much. Examples of such logic would be:
if a transaction has 0 eth and some data, we show it as a contract call and try to decode the data in a human readable form
if a transaction has 0 eth, some data but no log we can assume that the underlying contract execution failed. for erc20 tokens that would usually mean you didn’t have enough tokens or you didn’t pay enough gas
Regarding performance I think it might affect the wallet slightly more than the rest of the app as there is a forced update of transaction list when going there. It is beneficial overall anyway because again:
you don’t fetch whole history over and over again
you don’t have a loop running but rather just listen to updates from status-go
The wallet will then be more responsive and transactions much faster to appear. Currently if you are unlucky and poll for a transaction right before its block is mined you’ll only see it up to about 30 sec later (I’ll be surprise if it is more but it is already very long from a UX perspective considering most wallet would already tell you it’s been confirmed 3 times)
@andrey@rachel we should plan working on this next week, as a first iteration without status-go involvment we can:
persist transaction in db
ensure failed token transactions are shown as such
token history not limited to 2 weeks but fetching older history either over time or when user scrolls down. I would opt for the later, it will add a delay for the user when scrolling down the list of transactions similar to what happens in chat when you go further back in history, but the amount of API calls to infura over time would be much much smaller. 99% use cases won’t require checking out transactions older than 2 weeks, and most of those would be persisted anyway. So the idea would be that you have all history since creation or 2 weeks prior to recovery of the account and from then on all history is fetched and persisted, so if you go away for 2 months the 2 months are fetched when you relogin, similarly to what is done now with message gaps on mailservers.
If status-go can be involved we can get rid of the loop and use signals instead which will free up the js thread a bit more.
The suggestion fixes a newly documented issue (lack of history for token transactions), should offer performance improvements and also paves the way for us to improve Tx history UX.
