What are our criteria for releasing security-critical features?

Archive
1

I did something nasty just now. I rejected a PR merge that a lot of people put a lot of time and effort in and we universally seem to find a good idea. The github issue in question is: Feature/emojis signing

We all want emojis and we’ve all wanted them for quite some time. Do a quick search on Slack and you get 336 results of conversations mentioning emojis. I bet most of them are around emojis as a signing phrase. @jarradhope even made a demo of it: https://rawgit.com/jarradh/ca8bfc407d7d19b8a5d69021fc442ef6/raw/48d9d2781709d959b7df9e8dea2807df8e5dddc9/emojiguard.html

Being one of the few products that actually has something in place that acts as a phishing guard is in itself quite unique and a showcase of how we value security.

While we have consensus that emojis are the right thing to do; we have not gotten the design right.

Here’s a rough table of issues we found in UX testing and how the latest implementation, to the best of my knowledge addresses them. https://docs.google.com/spreadsheets/d/1jAaPeoxVWUIy2zjdRWjCeRPPauUCZlSb3nb-NGJdHBM/edit#gid=0

In short, there are a few issues for which the emojis provide a solid solution and we are thereby introducing an improvement. The most important being:

The difference between the ‘seed phrase’ and ‘signing phrase’ was not clear to a number of participants.
Emoji signing phrase and Seedphrase are now conceptually and visually distinct.

Still not distinct enough for 100% of users, but let’s leave that for now.

There are about 6 issues that I’ve either seen remain in testing or imagine from a usability engineering perspective to expect to come up as issue sooner or later:

  1. The copy used does not indicate what one should do when not recognizing the emojis.
  2. None of the participants described intended behavior with regard to the phishing guard.
  3. No evidence about the effectiveness of the signingphrase
  4. The pattern does not prevent phishing for password.
  5. [Sign contract transaction] without introduction might create confusion for lay users. Nowhere in the app is there any mention of ‘Contracts’ yet.
  6. The critical moment does not show the Wallet address and amount for verification.

Although I find these issues highly important, they are not by definition a reason to reject a commit; Because all these issues also exist in the current implementation.

My main reasons to reject are process related:

  • We should only release security-critical features if we have validated them thoroughly.
  • After validation we should not make any changes to core UI elements without thorough analysis of the impact as changes my invalidate any findings we have about how secure our solution is.
  • “It’s not going to make things worse” is not sound logic unless we can prove it. Honestly, this was my rationale up until this morning. Until I realized False security is worse than no security. I.e. a few improvements actually do make things worse.
  • When we approve a commit knowing that issues remain we need to at least have a follow-up process defined (e.g. A github issue to resolve remaining issues)
  • When we approve we need to make sure that the release is ‘contained’ and any issues are not unknowingly transferred to for example Desktop.
  • The rationale that something is built is not a sustainable logic in a scaling organization. I could build something. It will be full of security holes, ugly and malfunctioning.

To kick of discussion, please share any criteria you’ve used in the past to judge when (not) to release new features.

Let me end this with some wise words from @Chad:

Security features need proper process or validation, otherwise it is going to end up on wall of shame 2.0

3 Likes
2

How do we identify in the first place what a security-critical feature is?

It might not always be obvious (as your example highlights) and depends on the viewpoint you are looking from (design, implementation, …).

Same questions apply to privacy probably.

3

Very important question. We have a doc coming from UXR side that classifies tasks as Essential or Security-Critical.

The only way I know to do this is by looking at the consequence. Which is why I specify Consequence for issues in usability testing. If a consequence is that a user may lose all their assets, I currently classify this as Security-critical. I often treat related tasks as critical as well as usually adverse consequences are the result of compounding factors.

Would welcome all eyes on this as indeed we need to asses risks from multiple viewpoints.
I’ll add Privacy-critical to the sheet.

Classification Description
Essential Necessary for successful use, this can be a task related to a frequently used function (e.g. send a message) or one that would block usage (e.g. create an account).
Security-critical Tasks in which people are vulnerable to lose assets, unintentionally expose their identity or other undesirable information

This is far from complete and arbitrary. Please add any feedback.
We also need to define what success means for each.

E.g. for Signing phrase "User does not continue transaction when after x times of signing a transaction a different signing phrase is displayed.