A Natural History Of Trust & Safety

In the beginning, there were harms.

No, not yet. First, there’s an idea, a product, a company. You’ve made an app where people can express themselves in some way (text, images, videos, something else, some combo of them…) and with some ingenious and distinctive features (everything is ephemeral… or permanent … or upside down … or run through a filter … you get the idea). Congratulations. You’ve raised some money and are gardening the community and it grows. Maybe it’s big enough that someone calls it a platform.

Much of the company is thinking about questions like these: How do we make the product better? How do we make money? How do we tell more people we exist?

Then, without warning, come the complaints. They come in all shapes and sizes, from all directions. Probably to an email address like, “help@ourstartup.com” or eventually, “abuse@ourplatform.com.” They range from things that are fairly simple and not too painful, like, “How do I log in?” or “Why is the ‘post’ button oval instead of rectangular, I think that’d be nice,” or “Why doesn’t your app work on my Android phone from 2011?” or “Who are you and why are you emailing me?” These are real problems, and there’s honor in solving them, but they’re not Trust & Safety problems. The team that handles these kinds of problems are usually called something like “user services” or “help desk.”

And then there are problems like, “Someone told my phone company they were me and took control of my account, can I have it back, please?” or “I clicked on an innocuous looking link and now my account is selling a cryptocurrency but I don’t know what that is” or “I’m 16 years old but I found a line of code that lets me destroy your entire site, can I have some money and I’ll tell you which line.” The people who handle these problems are called something like cybersecurity. These are also important problems. We’re getting closer.

But then other grievances come in and they’re more like this: “My ex posted naked photos of me from 8 years ago on your website, my children’s classmates are turning them into dank memes, help” or “People are using your product to share recipes for arsenic smoothies and I heard some kid drank one” or “one of your power users posted my address and told everyone to not send pizzas to my house and then 700 pizzas arrived at my house, along with a SWAT team, help.” Many people, if emails like this regularly arrived in their inbox, would quietly close the email tab and slowly back away from the computer. These people are not cut out for Trust & Safety work. It’s best not to try and make them.

But some people, maybe just one person at the company, they will hang in there with these messy, human, winnerless problems. They can’t walk away. With some strange mix of empathy, curiosity, and urge to ease some stranger’s pain, they will debate, with others if possible and just themself if necessary, how to help. Eventually, they will make a decision, take action (or decide not to), and live with the consequences. They will take down someone’s words. Or ban a user account. Or suppress it in search results. Or write someone an apologetic email that they can’t help. One way or another, they intervene. Then wonder about the implications of what they’ve just done. This is how your Trust & Safety team starts.

Slowly, or all at once, these grievances become their whole job (as a system evolves for managing all the complaints, they probably start calling them “tickets”). They may be one of the less technical people at the company, or not. But for whatever reason, they care. They didn’t choose this set of problems, the problems chose them. Maybe because they’ve suffered something similar (or dissimilar). Maybe they’ve been an investigator or first responder or caregiver before. Maybe because these problems are so different than most of the other problems you see at a tech company — darker, squishier, ambivalent. Novel in certain details but also ancient. It scratches an itch in them that other tech problems don’t. The number of tickets only ever goes in one direction — up. It never goes down. There may be lulls and slow days, but the number never goes down.

It starts with one person who cares trying to do rough justice for people who are suffering where your product is part of the reason. But that’s not where it ends. The gravitational force of precedent emerges. After the first decision comes a second. And now you are forced to wonder — were the two consistent? Do the two data points trace a shared principle? Either way, your platform now has common law. You can’t prevent this. The only question is whether you recognize and acknowledge it.

The one person doing rough justice is all at once a miniature legal system — rules, precedents, evidence, reasoned adjudications, appeals. Expectations of due process and transparency. The appeals are coming back to the same person who cares. That feels wrong. They need help. You hire another person, or they shop for software tools to help in their work, or you hire a firm that has done this before, has contractors nearby, or far away, or really far away.

Now you have a team. The number of tickets continues to climb. The problems mutate with the times (your CEO testifies to Congress, or your product is accused of helping people seeking illegal abortions, or illegal firearms, or both). The Trust & Safety team strives for fairness and consistency. They’ll never achieve this. Academics will dissect and journalists will mock their endless Keystone Cops attempts. They adopt a sardonic tone in their Slack channel or get Sisyphus tattoos together. The problems they work on are novel in their expression, because your technology is at least a little bit new. But they’re not really problems at all. The harms and the grievances manifest from the product, in your community, but they are not in the end problems that can be solved. They are a byproduct of being human which is, after all, not a problem to be solved, but a condition to be managed. The tickets continue to climb.

Alex Feerst is the William Blake of Trust & Safety. Originally posted on Medium, reprinted here with permission.