What most buyers get wrong about Trust & Safety vendor POCs

The first two pieces in this series covered how T&S vendor pricing gets shaped by investor dynamics, and how to negotiate the contract once you've decided to buy. This piece covers the phase that should happen before either of those: the proof of concept.

Most POCs don't tell buyers what they think they're telling them. Not because the vendors are running scams, but because buyers set the POCs up in ways that guarantee misleading signal. Then the buyer walks away from a working vendor because the numbers looked bad, or picks a losing vendor because the numbers looked good on a test that couldn't have been failed. Both outcomes trace back to the same root cause: how the POC was designed.

This piece is what a good one looks like. What dataset to bring, what to score, how long to run, who to involve, and the counter-conventional posture that most buyers get exactly backwards.

Bring a finite, labeled dataset. Not a big one.

The single most common POC mistake is assuming that a bigger dataset produces a more meaningful result. It doesn't. It produces a slower result with worse signal.

The right dataset for a T&S POC is finite and labeled. Finite so you can actually reason about the results in the time you have. Labeled so you have ground truth to score against.

The reflex to bring "everything from the last three months" comes from a good instinct (representativeness matters) and lands in the wrong place. What you end up with is a dataset too large to spot-check, containing edge cases you don't have ground-truth labels for, and a POC that runs out of time before either party can seriously interpret the results. The vendor tunes against a fog, you evaluate against a fog, and the outcome tells you nothing useful.

Similarly wrong: bringing your hardest cases. Adversarial datasets composed of your worst historical false positives and false negatives will make any classifier look terrible, including one that would perform beautifully in production. You're not evaluating the classifier; you're proving you can construct a hard test.

The right posture: a finite labeled dataset that represents your actual production distribution. A few thousand pieces of content, categorized against your taxonomy, with clear labels. If you can't produce that internally, that's a signal about your operational maturity worth confronting on its own.

The single most common POC mistake is assuming that a bigger dataset produces a more meaningful result. It doesn't.

Score honestly. Don't get hung up on individual cases.

The next mistake is scoring methodology. This is where most buyers, even sophisticated ones, quietly sabotage their own evaluations.

Serious buyers know that a well-performing classifier lands somewhere in the 85% precision and recall range on production data. That's excellent, not embarrassing. A classifier that claims 99% is either overfit, evaluated against a friendly dataset, or the vendor is measuring something narrower than you think they are. If your POC scoring methodology treats anything below 95% as a failure, you're setting a bar no honest vendor can clear.

Some categories are also structurally harder than others. Profanity is a mostly-solved problem; a decent classifier will hit near-ceiling accuracy on obvious cases. Grooming, harassment, and coordinated harm are meaningfully harder, and no classifier on the market handles them cleanly. Weighting these categories equally in your scoring will produce numbers that hide where the real capability is and where the real gaps are.

The other trap: getting stuck on individual cases. You will find false positives that seem obviously wrong to you, and false negatives that seem obviously catchable. Every buyer does. The temptation is to walk into the vendor review meeting armed with a list of "how did you miss this?" examples.

Don't. Individual cases tell you nothing about aggregate performance, and pushing on them wastes the meeting where you should be discussing the actual signal: precision and recall across categories, at your production distribution, with your taxonomy applied. If the aggregate numbers work, the individual cases will resolve. If the aggregate numbers don't work, the individual cases weren't the reason.

Category alignment is where mutual disappointment lives

Vendors have their own taxonomies. You have yours. They almost never map cleanly.

This is the single biggest source of POC frustration that isn't anyone's fault. The vendor's "harassment" category may include what you consider bullying, but exclude what you consider coordinated harm. Their "adult content" may cover explicit imagery but not thematic content that you'd want caught. Their "hate speech" may score high on obvious slurs but low on dog-whistled variants that matter more to your community.

You will not fully resolve category alignment during a POC. What you can do is surface where the mismatches are, decide which ones are dealbreakers, which ones the vendor can accommodate with configuration, and which ones require post-contract customization work.

Buyers who go into POCs treating category alignment as a solved problem walk out disappointed. Buyers who treat it as the primary calibration exercise walk out with a realistic assessment of what the vendor will and won't do for their specific use case. The second group makes better buying decisions.

Two weeks. Not longer.

The right POC duration is one to two weeks. Not one to three months.

Longer POCs sound like they'll produce better data. In practice they produce disengaged buyers. The internal champion who kicked off the POC moves on to something else. The executive who authorized the budget forgets what the vendor's differentiators were supposed to be. The engineering team that was going to run integration testing gets pulled onto a fire. By month two, the POC is a slow-motion demonstration to an empty room.

Two weeks is enough to run a labeled dataset through, review results in aggregate, discuss category alignment, and make a real decision. What matters more than the calendar length is the meeting cadence during it. One or two meetings per week, with the vendor's team present, structured around actual results rather than status updates.

Those meetings are where the real POC value gets generated. Not the raw numbers. The conversation about what the numbers mean, what the vendor is willing to tune, and what your production reality is going to demand.

Longer POCs sound like they'll produce better data. In practice they produce disengaged buyers.

Both product and T&S at the table

The POC should be co-owned by product and T&S. Both stakeholders matter for the eventual outcome and both need to be bought in for the integration to survive first contact with production.

Product owns the surface: what the integration looks like, how the API fits the platform's flow, what the fallback behavior is when the vendor is slow or down. T&S owns the substance: what the categories mean for the community, how policy translates to configuration, what the reviewer workflow looks like when human-in-the-loop is invoked.

Buyers who let one function run the POC alone tend to discover the other function's requirements post-contract. Product-only POCs produce elegant integrations that don't match policy. T&S-only POCs produce policy-perfect configurations that engineering can't ship.

The scoring conversation especially benefits from both perspectives. Product will care about latency, uptime, error handling, and how classifications flow through the platform. T&S will care about category coverage, precision-recall tradeoffs at your specific volume, and how confidence scores map to reviewer workflows. Both matter. Only having one of them in the room during POC review means you're solving for half the problem.

The counter-conventional part: take the vendor's attention

Here's the one that most POC-running buyers get exactly backwards.

During a POC, a good vendor will offer you a lot of attention. Their ML team will be responsive to questions. Their product team will offer to sit in on review meetings. Their sales engineering lead will personally walk through calibration options. Buyers often read this attention as a red flag: "they're overselling," or "this level of hand-holding won't survive contract," or "they're trying to influence the outcome."

Take the attention. Use it. It's not duplicity, and it's not a salesperson trying to trick you.

The vendor's investment during the POC is the mechanism by which they earn the deal, and it's also the mechanism by which the POC produces a better result for you. When the vendor's ML team explains that a specific category needs calibration to align with your taxonomy, they're not spinning. They're telling you where the actual work is. Ignoring that guidance because you're worried about being "influenced" produces worse POCs, not more objective ones.

Similarly, share labeled data with the vendor. Buyers sometimes hold this back on the theory that it would let the vendor tune to your test. That misunderstands what tuning is. Category calibration against your labeled data is the work. It's not cheating; it's the process by which you find out whether the vendor's model can adapt to your policy at all. A vendor that can't calibrate to your labels during a POC isn't going to magically calibrate to them after you sign.

If the vendor offers to interface with their platform during the POC, take that too. Familiarity with the actual product surface is one of the highest-value signals you'll get during evaluation. You'll learn more from twenty minutes clicking around the vendor's dashboard than from any slide deck.

The reframe worth internalizing: vendor attention during a POC isn't a bias you're supposed to control for. It's the substrate that makes the POC useful. Buyers who treat vendors as adversaries during evaluation get worse evaluations. Buyers who treat vendors as collaborators, while maintaining honest skepticism about the aggregate results, get useful ones.

Vendor attention during a POC isn't a bias you're supposed to control for. It's the substrate that makes the POC useful.

What a good POC produces

A well-run POC produces three things.

First, aggregate performance numbers on precision and recall across categories, run against a finite labeled dataset representing your production distribution. Realistic numbers in the 85% to 95% range, with honest acknowledgment of where the harder categories fall short.

Second, a shared understanding of category alignment. Which of the vendor's categories map cleanly to yours, which ones require configuration, and which ones will need post-contract customization work.

Third, a working relationship with the vendor's team. You'll know who to call when something breaks in production. You'll know how their ML team thinks about calibration. You'll know whether their product surface fits your operational reality.

If your POC produces those three things, you have what you need to make a real buying decision. If it doesn't, the problem is almost never that the vendor was hiding something. It's that the POC was designed to obscure what a good result would look like.

The next piece in this series covers the phase that starts once the POC ends: how to structure the integration itself, and what to insist on before you sign. Subscribe below to get it when it lands.