Replacing Mixpanel - A short term brainstorm

Archive UX and UXR
1

If we gut MixPanel from the Status application tomorrow, what could we use as proxies for replacement metrics? Medium term we plan to replace MixPanel with Methrics (see here), but what’s stopping us from removing MixPanel today?

Brainstorm Exercise
What are tools that we can use today (or with a trivial amount of work) as proxies for the most important metrics:

  • Active users (Daily transacting users)
  • Wallet usage (# transactions sent / % of users that have > 0 in wallet)
  • Chat usage (daily chat usage)
  • DApp usage (users who use DApps)

Are there dashboards in place today that can help as proxies for the above metrics? e.g. Android/iOS dashboards/metrics, Whisper message volume, etc. If we can get close enough to our most important metrics, then there should be no reason to keep MixPanel around until V1.

The metrics above are examples, may not specifically be the most important for each team

Why should we remove MixPanel?
There’s been many discussions on this topic, most of which are long lost in Slack. My personal concerns summarised are:

Having MixPanel in the application has been useful in understanding user behaviour, but goes against the fundamental principles of why Status exists - self sovereignty of data.

2

  • DTU: Take https://etherscan.io/address/0x744d70fdbe2ba4cf95131626614a1763df805b9e and remove known exchange sources to get rid of traders

  • DTU: Looking at interactions with Status app specific contracts, like SNT going into our ENS contracts

  • Chat usage: Whisper volume metrics in our cluster, correlate to user growth (sans spam)

  • DApp usage: Do UXR / let this be up to DApp itself.

  • Wallet: similar to DApp. Can also look at ETH txs for wallets that have SNT in them, and then remove what seems like exchanges/traders

Unclear to me how easy it is to filter out trader/exchange traffic, but seems worth exploring.

UXR KISS solution to validate user flows Why You Only Need to Test with 5 Users

Can also consider putting generic “StatusApp” string in data field. This would leak some bits of information (this tx came from Status) but if it is for SNT it might not be a big deal. Just a random idea.

2 Likes
3

Did some more digging into DTU, and it strikes me that SNT-SNT is unlikely to be trader volume. This data can be found here https://etherscan.io/token/0x744d70fdbe2ba4cf95131626614a1763df805b9e (download CSV export at bottom)

Etherscan only supports exports of the first 2000 records from a certain date, but with some naive investigation we can see how long it took for the 2000th transaction to go through in a month:

> tail -n 1 snt-2000-apr.csv
"0xa5736f1a50fde0bffc5d11d3b26b7c1500b7779341e631403f97eeb858a710de","5371492","1522736143","4/3/2018 6:15:43 AM","0xd551234ae421e3bcba99a0da6d736074f22192ff","0x2e46956565cebdcbbb39ecd22af02e1916a2fe37","66561"
> tail -n 1 snt-2000-may.csv
"0x3e98df7cda4c5c4cfa674c1ca96ec77beeca2e4b54797a38c0d9db33985bb1c2","5545255","1525291318","5/2/2018 8:01:58 PM","0xfbb1b73c4f0bda4f67dca266ce6ef42f520fbb98","0x3f000ab24d1b9403bf8e3d6ccc6af69772ae5c34","22980"
> tail -n 1 snt-2000-jun.csv
"0xb33e6219834a3a0618aba928fb75b1aa40f6f72d46d07cdb3ac94f43ee9fd858","5737352","1528215664","6/5/2018 4:21:04 PM","0x6e84ac1fe7f17950ac82b8fe9b92d14ec8098456","0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be","10523.9332"

I.e. it took ~54h to reach 2000 SNT txs in April, 44h in May, and ~112h in June. I’m not quite sure why there’d be fewer total SNT transactions in June when we enabled mainnet, so that’d be worth digging into. One easy thing to do is to dedup per from address (hypothesis: May has a lot of txs from same address) but it turns out there are about ~800 unique from addresses in both May and June from what I can tell. A lot more can be done with this data, and we can setup some kind of export script and do basic analysis of this.

4

For interactions that involve smart contracts, events can be fired in the contract, they are similar in nature to mixpanel events but stored on chain and can be indexed for faster parsing, they are also fairly low cost in terms of gas usage compared with on chain storage.

5

What would be concrete examples of when we took the output of mixpanel (ie the stats), had a discussion around it, and acted up on it - beyond trivia / pat on the back / obvious stuff (“users use dapps!”), that we couldn’t have figured out otherwise / would have been excessively costly to figure out through other means?

Another question also: what’s the interest in knowing DTU to begin with? What do we do with that particular number, except to check off key results? Knowing what the metrics are used for in detail might help drive a better discussion as to what replacements or proxies we can look at.

Also, what can be replaced with reasoning, experience and other types of (smaller) studies like what @oskarth points out (which by the way reminds me of its equivalent in application performance testing, which similarily points out the diminishing returns of larger sample sizes and advanced profilers: How do I profile C++ code running on Linux? - Stack Overflow and https://poormansprofiler.org/, reason being maths / Bayes)? Can we use the 80/20 rule to tease out some other clever hacks?

Here are a few other coarse indicators off the top of my head:

  • Dapp developers customize / test their dapps for status (outsource the success decision to others)
  • number of complaints per time period, on google play/appstore
  • For chat: more specifically, look at ingress whisper traffic to whisper cluster from leaf nodes - I think we’ve gutted whisper enough in the name of efficiency for this to be fairly reliable - @divan can confirm? this effectively is traffic analysis btw - anything that works for us here is a failure :slight_smile:

I once suggested “downloads on release day” to @Chad but that was rebutted with the fact that an unknown percentage of those users drop off / don’t actually use the dapp, making it a bad metric for more advanced stuff like retention and engagement. I’d still think there’s at least some correlation here with the popularity of the app / DTU since it’s a prerequisite to actually transact, but this ties back to the question above - is engagement/time spent in app something we’re optimizing for? I honestly can say that I have no idea how good such a metric is, but it’s used as trivia in virtually all the app stores out there as a sorting option, so someone must have thought of it as useful for something (verifying authenticity of app?)

Here’s a questionable one as well, that points out a weakness:

  • Collaborate with dapp developers - effectively most dapps rely on web resources which mean that they already track the dapp usage numbers for us… longer term, this is something we’ll move away from (with better storage options / swarm / ipfs / whatever), but a reality today, and one that our users should be made aware of at the very least.

A way of looking at this particular thread is that anything that we can find out about our users is potentially something that we should work towards thwarting - the mere fact that we can use it for “good” means that someone else can use it for questionable reasons - an explicit goal of anything we do in this field has to be to help users understand these inevitable leaks that are part of the service we’re providing.

1 Like
6

What would be concrete examples of when we took the output of mixpanel (ie the stats), had a discussion around it, and acted up on it.

Example 1. DApp usage is high considering how deeply hidden within the UI it is placed today (3 clicks deep). This means that we should be focus on improving our DApp development documentation and everything DApp related (see hackathons, DevRel hiring, etc. etc.)

@Chad can probably point to a lot more clear examples where we have used it for product improvement rather than just pat on the back stuff.

obvious stuff (“users use dapps!”)

It may be obvious to you, but it isn’t for everyone. Some people think that Status is just messenger, or just a wallet, or just a DApp browser. Saying ‘obvious stuff’, implies that every user of Status sees the application from your perspective.

what’s the interest in knowing DTU to begin with? What do we do with that particular number, except to check off key results?

It’s a guiding point that let’s us know if we are actually building something that people use or not. What if 0 people used Status, how would we know? Also, I’m not particular tied to DTU, happy to use something else as a proxy for “Do people use the Status app”.

Also, what can be replaced with reasoning, experience and other types of (smaller) studies.

That leaves decision making to the loudest voice in the room. I’m sure you can imagine a community member chiming in with: “Our wallet flow sucks, we need to improve it”. How does it suck? Why do you think it sucks? Qual research is applied during the design phase to validate the improvement, but in order to even engage with the conversation about whether the wallet flow sucks, quant metrics are incredibly useful.

This has happened repeatedly at Status over the past 4 months (I’m sure @Chad can point to specific Slack threads where metrics were used as the objective truth on a subjective topic).

is engagement/time spent in app something we’re optimizing for?
This relates to which metrics are most important for Status, which is great!.

I personally don’t see ‘time in app’ as something we want to optimize, as it encourages shady/dark ux tactics to keep bringing users in. If someone uses Status to send money, and it takes < 10 seconds, then thats a great success!

DTU’s has been thought of as the main metric purely because it is a safe measure of daily utility of the Status. Considering the breadth of tools people have access to in Status, we assume the time frame is daily (vs a weekly/monthly, for example Google Maps is used more on a weekly/monthly basis since most people don’t need a map in their daily routine). If our goal is to bring DApps/messaging/wallet to the masses, then DTU is a good proxy that we are providing utility for users in one or more of the three areas.

As mentioned above, I’m more than happy to collab/brainstorm on alternatives. DTU was the best we could think of 7 months ago and lot has changed since then.

3 Likes
7

A different way of looking at things that might be interesting - privacy metrics:

8

Summary from a F2F conversation about this w/ @oskarth @arnetheduck Dustin and @carl

In Berlin during ECDC, we spent ~90 minutes burning the midnight oil discussing the pro’s and con’s of Mixpanel usage and metrics in Status. It was acknowledged (I hope) that although qual/quant UXR can be used for measuring app usability, long term qualitative metrics are crucial as lagging indicators of feature adoption/utility, and team success.

We discussed varying spectrums of the tension between:
(a) Status as a completely secure, 0 bits of leakage, application that has complete darkness on usage. And…
(b) Using usage to help make Status a useful to fulfill our mission of mass adoption of Ethereum in the world.

Proposal

  • Remove MixPanel in the next release (or the one after, whatever is most feasible)
  • Dustin (to join as a core contributor mid-July) will own the creation of our own open Status dashboard, in the same vain as the above privacy metrics outlined above.
  • The deadline for having these new metrics will be 1st of September, which gives us a few weeks of data in order to start planning for Q4.
  • The solution may include an explicit opt-in (i.e. not as part of the onboarding flow) in order for users to explicitly provide us with data.

Everything else around the key metrics we should measure in the proposed solution are not clear at this stage (other than the high level metrics outlined above)

Next Steps

  1. Further feedback on above from others passionate about this topic. (cc @rachel and @Chad, happy to discuss over VC if easier)
  2. Continue collecting proxies for the important metrics outlined above (e.g. I still haven’t checked what info we have in our iOS and Android dashboards)
9

I’m supportive. It sounds like the custom dash will be roughly equivalent to our MixPanel implementation, and the SNT voting feature will be a helpful supplement.

I also question the importance of this to users.

We’re in probably the optimal context for privacy to be trending, but I know maybe one person who broke up with Facebook after Cambridge Analytica.

Will complete darkness be a selling point to convert people from other messengers? Can we make the benefits of privacy felt in product?

And what about the downsides? Darkness has actually raised questions from a couple of my friends about the potential for criminal use.

Our core mission is to popularize Ethereum—beyond the early community who value privacy as much as we do. I think that using data to see problems and opportunities within our product, will be more impactful to our growth than stronger restrictions on product tracking. Edit: @naghdy’s points above summarize why I feel that way.

We do need to uphold our principles, and I may be understimating the public interest. I just want to know how we can leverage this as a benefit.

@oskarth challenged me to think about the privacy requirements for someone like Julian Assange or Edward Snowden to use Status. Becoming the default messenger of someone that high profile would definitely be a marketing coup. :slight_smile:

1 Like
10

I am also supportive of the proposal above, but want to outline briefly a difference in my thinking.

Darkness is very definitely a selling point in my experience, and always leads to interesting questions about criminality etc. that can be used to make important and far-reaching points about the socially-constructed nature of both law and utility. Which leads me to:

Our core mission is to popularize Ethereum—beyond the early community who value privacy as much as we do.

Which I disagree with (also incidentally why I don’t think mass adoption should be a core principle). There is a fundamental trade-off in this world between autonomy and convenience. The internet as we know it now is like it precisely because many well-intentioned and intelligent people got so caught up in the narrative of making sure everyone could participate in the creative connection enabled by a collaborative web of information that they did not think very critically about the trade-offs being made to make it convenient enough for everyone to access.

I would say that our mission in this context is to make using Ethereum as convenient as possible while not compromising on the kinds of autonomy enabled by such distributed networks.

It’s not about popularizing Ethereum or ensuring growth to me. It is about building networks that incentivise collaborative value creation, rather than relying on the number of users or the data they contribute in order to remain sustainable and profitable.

DTU and wallet usage seem to me to be metrics related only to web 2 stuff that ought not really impact our thinking too much. In this vein, I do really like Jacek’s suggestion about DApp developers, testing, and tracking complaints, because they a) encourage community participation, which we can incentivise and b) have far more to do with collaborative value creation, rather than simple usage, which I think will be a far more powerful financial/social/network effect going forward.

2 Likes
11

Issue to remove Mixpanel Remove Mixpanel · Issue #5169 · status-im/status-mobile · GitHub

1 Like