Coral · 2016–2023
Signal, Not Verdict
The approach that shaped how trust and safety teams now think about AI in moderation.
The result
36%
of warned commenters self-corrected McClatchy controlled study
In 2017, no one had integrated an AI toxicity signal into a comment platform. I designed Coral's moderation experience from the ground up, including the first production integration of Google's Perspective API, against the direction the industry was heading.
Role
Lead Product Designer
Sole dedicated designer. End-to-end ownership of the moderation experience.
Timeline
2016–2023
Mozilla era through Vox Media acquisition.
Scale
120+ newsrooms
23 languages, 18 countries. WSJ, Financial Times, The Verge, NY Mag.
The Situation
A comment box looks simple.
By the mid-2010s, newsrooms were shutting down comment sections entirely. The problem was not comments. It was what happened when they were left unmanaged. Moderators at The Washington Post were reviewing more than a million comments a month. The New York Times had a team of 14 people moderating 12,000 comments a day, on only 10% of its articles.
What slipped through landed hardest on the readers newsrooms most needed to keep: women, readers of color, people participating in good faith. Publishers who shut comments off were not solving anything. They were giving up on the readers that mattered most.
They wanted to do their job fast.
More importantly, they wanted to do their job well.
The Design Problem
The AI couldn't judge. The moderators couldn't scale. The readers couldn't wait.
Moderators could not keep up with volume. The AI signal could not be trusted to make final calls on its own. And a publisher that removes comments by algorithm is making editorial judgments without editorial accountability. The design had to answer all three.
Perspective API returns a probability, not a judgment. It is context-blind, culturally uneven, and wrong in specific, understandable ways. Knowing exactly what it could not do was the prerequisite for designing around it responsibly.
How Moderators Work
Moderators weren't scoring comments. They were reading rooms.
Before I could design anything, I needed to understand how moderators already made decisions without the tool. I sat with them and watched. That fieldwork was part of 300+ interviews across 150+ newsrooms, alongside a large-scale survey of moderation practices.
They were never evaluating one comment
Moderation was about building a picture: the words, who said them, their history, the full thread. A flagged comment could be a signal for a whole thread going sideways. Single-comment evaluation was a last resort, not a default.
Fast and well were not the same job
Speed got them through the queue. Judgment was what the queue was for. A tool that optimized for one at the expense of the other would make the actual job harder, not easier.
Both tracks pointed at the same principle: the design had to keep speed and judgment from competing with each other.
The Design
Surface, not delegate.
The three constraints pointed in one direction. AI routes attention. Humans keep the call.
The obvious move
Hard-gate on the toxicity score. Above threshold, block. Below, publish.
Moderation load solved. Automation handles the judgment. Ship it.
What I built
Surface, not delegate. AI routes attention. Humans still decide.
The bottleneck was not making decisions. It was finding which comments needed a decision at all.
Decision 01
Have an opinion on what comes first.
Reported comments are already live. They are on the site, visible to readers, causing harm right now. Pending comments have not shipped yet. Unmoderated comments got through and are waiting. These are not equivalent states, and the tool should not treat them as if they are.
The queues are ordered left to right by urgency. Moderators open the tool and know immediately where to start. That judgment is not something they should have to make themselves, every session.
Queue order encodes urgency. Left is live on site. Right is already resolved.
Reported
77
live on sitePending
568
pre-moderation, not live
Unmoderated
2.3k
published, not reviewed
Rejected
actioned
most urgent to least urgent, left to right
What I chose
An opinionated left-to-right queue order. Reported is first because it is live. The tool makes the urgency judgment so moderators do not have to.
What I rejected
Filterable status columns: flexible, customizable, no opinion on what matters most. Moderators could sort however they wanted.
Why
A filter UI is infinitely flexible and gives moderators no guidance on where to start. That judgment call, made from scratch every session, is exactly the kind of friction a well-designed tool should eliminate. Flexibility was not what moderators needed. An opinion was.
Decision 02
Fixed locations for the Approve and Reject buttons.
Moderators make hundreds of decisions per session. Every pixel of mouse movement, every moment of searching for a button, every eye shift away from the comment compounds. At scale, friction is not a minor inconvenience. It is the job.
The reject and approve buttons sit in a fixed panel on the right, large enough to see in peripheral vision while reading. A minimum card height keeps their position consistent regardless of comment length. When a moderator actions one comment and the next card loads, the buttons are in the same place. The mouse does not need to move.
Short or long, the decision buttons never move. The mouse does not have to travel.
Extending with keyboard shortcuts.
For high-confidence decisions, moderators never had to touch the mouse at all. Keyboard shortcuts extended the same principle: fewer interruptions to the reading rhythm, more decisions per session without added cognitive load.
J Reject K Approve F Feature ↵ Next comment
What I chose
Fixed button placement with consistent position enforced by min card height. Large targets visible in peripheral vision. Keyboard shortcuts for zero-mouse decisions.
What I rejected
Inline actions embedded in the card body. Buttons that shift position based on comment length, or appear only on hover.
Why
The physical workflow had to be designed as carefully as the information hierarchy. None of this came from a requirements document. It came from watching moderators work.
Decision 03
A signal to consider, not a verdict to accept.
Perspective returns a probability. The card had to reflect that. If the toxicity signal led the card, moderators would evaluate the comment through the machine's lens before forming their own read. The signal had to be present without being in charge.
Signal labels sit at the bottom of the card. Flagged words are highlighted inline in the comment text, giving context for why the flag fired without leading with the conclusion. The tags use an outline treatment rather than solid fill: visible enough to notice, quiet enough not to decide for you.
Solid fill: verdict energy
Toxic Suspect Words
The machine's assessment arrives as a conclusion. Heavy, decided, hard to look past.
Outline: signal energy
Toxic Suspect Words
The flag is visible. The judgment is still yours.
What I chose
Outline tags at the bottom. Inline word highlighting for context. Visual weight calibrated to signal, not verdict: visible enough to notice, quiet enough not to decide for you.
What I rejected
Solid fill tags at the top of the card. The machine's assessment as the first thing a moderator sees, before reading a single word.
Why
Perspective returns a probability, not a judgment. Every visual choice in the card had to reflect that. A solid tag at the top makes the AI feel like it has already decided. An outline tag at the bottom keeps the human in the role of judge.
Decision 04
The commenter has a different job.
A moderator is reading to judge. A commenter is writing to participate. The design problem wasn't protecting an independent read. It was giving the commenter a moment of reconsideration without silently suppressing speech.
Every word in the warning is deliberate.
💬 ✕
Are you sure?
The language in this comment might violate our community guidelines. You can edit the comment or submit it for moderator review.
✓
Your comment has been submitted and will be reviewed by a moderator.
✕The warning, annotated
"Are you sure? The language in this comment might violate our community guidelines. You can edit the comment or submit it for moderator review."
"Are you sure?"
Framed as reconsideration, not accusation. The commenter is being asked to reflect, not told they did something wrong.
"might violate"
Mirrors the model's uncertainty. A probabilistic score should not speak with more certainty than it has.
"submit it for moderator review"
Makes clear that a human, not the system, makes the final call. The confirmation state repeats this.
What I chose
One chance to reconsider. The nudge fires once, the commenter decides, and a human moderator makes the final call either way.
What I rejected
An infinite re-checking loop: re-run the score after each edit and keep nudging until the comment passes threshold.
Why
An endlessly nudging system becomes a game to beat. Good-faith commenters get trapped in a correction cycle they didn't deserve. One chance to reconsider respects the commenter's judgment the same way the moderator UI respects the moderator's.
The Proof
The numbers measured fast. The moderators measured well.
36%
of warned commenters edited their comment to reduce toxicity
McClatchy controlled study
34%
of nudged users edited after the warning
OpenWeb, 400,000 comments, 50,000 users
96%
reduction in very toxic comments
The Southeast Missourian
OpenWeb started with the delegate-to-AI model, using Perspective scores to auto-reject high-toxicity comments. They changed their mind. The pattern wasn't a Coral pattern. It was a pattern about what works when AI surfaces instead of decides.
The system was surfacing harmful comments moderators would have otherwise missed. Longer, subtler ones that didn't use obvious slurs but could still turn a thread.
The moderators told us they wanted to be fast, but more importantly well. The study numbers measured fast. This measured well.
Signal-not-verdict is now the default for serious trust and safety teams running moderation at scale. Coral was one of the first to put that approach in production, and the work helped shape the pattern the industry converged on. The moderators had been telling us the whole time. Fast, but more importantly well.