Soon we will introduce profile pictures in both the mobile and desktop app. Concerns have come up about objectionable content (read: promoting child pornography by adding certain profile pictures), the moral implications as well as risk of getting suspended from the app or play store.
Please share your thoughts on potential mitigations.
There are a lot of other topics on chat moderation. It’s a broad topic for which moderation mechanisms depend on what they’re aiming to moderate. I would like to hear more thoughts on a specific case: ‘Objectionable’ profile pictures in public chats
Profile pictures in public chats
Soon we will introduce profile pictures. Profile pictures add another type of user generated content (i.e. ‘USG’) to the application, in addition to:
text (all chats)
images (1:1 and group chat)
stickers (all chats - adherence to App store and Play store T&C required on submission)
urls (whitelisted for security reasons, currently limited to YouTube)
emojis (all chats)
In the #profile-pics channel as well as in other places, concerns have been expressed about the potential impact of profile pics. Here’s a summary:
Concerns
Users might include content that violates store policies resulting in getting suspended
Users might include content that users might deem objectionable, deterring after joining a public chat
Users might include content that core contributors might deem objectionable, e.g. child pornography
@cyanlemons@michaelb@JoRain@maciej@samuel@cammellos@Ferossgp, @iurimatias as you’ve been involved in or are working on developing this feature, please comment if this summary is lacking or inaccurate. Going forward, I consider these concerns as valid risks to be addressed and not the topic of debate. Although they make for great topics to discuss elsewhere, I’d like to focus this topic on discussing mitigations.
Mitigations
Require users to accept terms and conditions addressing content as required by Play store policies:
“defines objectionable content and behaviors (in a way that complies with Play’s Developer Program Policies), and prohibit them in the app’s terms of use or user policies”
Ensure that the feature can be removed if needed
Introduce “a method for filtering objectionable material from being posted to the app” (i.e. hide identified content by default by including some sort of library to classify images)
Identify and hide profile pictures altogether by default
One could argue that if we hide profile pictures by default the feature doesn’t add too much value to begin with
Show profile pictures only for people you add as a contact
We’ve seen issues in the past with disjointed identities and it being unclear when a certain name, identicon or image would be visible. This may have changed
Show profile pictures only in community chats and group chats that users have explicitly opted to join and can be moderated
In time, introduce and transition to avatars as profile ‘picture’
Allow users to flag (i.e. react to) objectionable messages; whereby if a threshold of ‘flag reactions’ is reached, the message and sender will appear in the UI under a ‘message flagged as inappropriate’ label. Illustrated below:
Voicing a personal preference for this solution number 8. As it can also apply to some of the death threats popping up lately. A major challenge with this would be to find a threshold and enable community governance on a threshold. Next to that cancel culture is real; creating a new chat key to participate is possible but it does come at a loss of database currently and transferring an ENS name would need to be supported in the app.
1 (T&C) and 2 (be able to remove the feature) IMO are needed regardless and 7 (avatars) has additional benefits (reduce netwwork load, potential for a marketplace)
Require users to accept terms and conditions addressing content as required by Play store policies:
I think we need to be very clear this only applies to public facing content. For instance, if you are in a private 1:1 chat with someone, you can send whatever and we can’t stop you. I also think we need to make sure this only applies to our app store variants of the app, and that downloading the APK imposes no such restrictions.
I think we should also be careful about #8, as it seems almost like Twitter. As a platform that claims to enable the free flow of information, I think it would be rather jarring to download the app as a new user, with some libertarian-esque image in mind, and to see “Message flagged as inappropriate” in the first channel they join. Another thing to consider is that it’s a DOS/censorship vector. Create some bot script that creates a new keypair, submits a “flag reaction” to the network, switches the proxy (to evade IP rate limiting) – and rinse & repeat.
Agree, I find as they are now public chats can be containers for all kinds of shit content. I would appreciate if there was a mechanism to not subject all participants to this torture and together flag things down.
Likewise as Hester I prefer this approach, mostly because it scales beyond just profile pictures.
Otherwise, moderated communities can’t come soon enough
I think we should also be careful about #8, as it seems almost like Twitter. As a platform that claims to enable the free flow of information, I think it would be rather jarring to download the app as a new user, with some libertarian-esque image in mind, and to see “Message flagged as inappropriate” in the first channel they join.
@cyanlemons a tweak I can imagine is that the message would not be shown as Message flagged as inappropriate, but more specifically +1000 people flagged this message as inappropriate. Such that it’s clear that flagging was done by users, not the platform
Generally these measures will be not very effective (not arguing against or pro) in public chats, since creating an account has basically no friction, so it’s very easy to overcome by just posting the same with a different account, any moderation issues will have the same flaws in free-for-all public chats.
Would you say the same holds for profile pictures? My thinking is that the barrier of creating a new account and adding a profile picture over and over becomes unappealing.
Generally, I agree that none of these mitigations are effective, definitely not on their own or as a preventive measure. What I believe we’re looking for is a strategy of multiple mitigations to address the below concerns. They can’t be solved for public chat, they can only disincentivize or complicate malicious behavior.
In case of the first concern, being able to point to a mitigation as “robust, effective and ongoing UGC moderation, as is reasonable and consistent with the type of UGC hosted by the app” (Play store policy) could be sufficient. Even though it ultimately is not effective to prevent malicious UGC
Would you say the same holds for profile pictures? My thinking is that the barrier of creating a new account and adding a profile picture over and over becomes unappealing.
It’s the same, while it’s true that is slightly more convoluted if done manually, it’s trivial to automate and have a scripts.
To conclude this topic for now. Below was agreed in chat to move forward:
Continue as planned, release profile pictures in public chat in 1.10
Include basic terms and conditions directing to (Play store and App store privacy policies)
Disable the feature in a new release if needed or introduce the functionality to ‘Show profile pictures from contacts only’ to offer a way for users to self-moderate the content without them needing to block other users entirely
