Chat Moderation: Filter Chat SDK Messages & LLM Outputs

AI chatbots built on LLMs can generate harmful, inappropriate, or factually dangerous content. Model providers add safety training, but it's not enough on its own. Jailbreaks bypass training-level safeguards, model behavior varies across providers, and your product's safety requirements are different from the model's default behavior.

You need an independent moderation layer that checks LLM outputs against your policies before they reach the user.

For an implementation overview, see Vettly's AI content moderation API. This guide shows where to check user messages and model outputs in a chat flow.

Why Model Safety Isn't Enough

LLM providers invest heavily in safety training — RLHF, constitutional AI, and system-level filters. But relying solely on model-level safety has problems:

• Jailbreaks: new jailbreak techniques emerge weekly. Prompt injection, roleplay attacks, and encoding tricks can bypass training-level safeguards.
• Model updates: when the model is updated, safety behavior can change without notice. A response that was blocked last month might not be blocked today.
• Your policy ≠ their policy: model providers optimize for general safety. Your product may need stricter rules (e.g., no medical advice, no financial predictions) or different rules (e.g., adult content is allowed in your app).
• No audit trail: model safety filters don't give you a decision ID, a log of what was filtered, or evidence for compliance reporting.

Architecture: Where Moderation Fits

The moderation layer sits between the LLM and the user:

1. User sends a message
2. (Optional) Check the user's input for policy violations
3. Send the input to the LLM
4. Check the LLM's output against your moderation policy
5. If allowed, display the response to the user
6. If blocked, display a fallback message

import { Vettly } from '@vettly/sdk';
import { OpenAI } from 'openai';

const vettly = new Vettly(process.env.VETTLY_API_KEY);
const openai = new OpenAI();

async function chat(userMessage: string) {
// 1. (Optional) Check user input
const inputCheck = await vettly.check({
  content: userMessage,
  policy: 'chatbot-input',
});

if (inputCheck.action === 'block') {
  return { response: "I can't help with that request.", blocked: true };
}

// 2. Get LLM response
const completion = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: userMessage },
  ],
});

const llmResponse = completion.choices[0].message.content;

// 3. Check LLM output
const outputCheck = await vettly.check({
  content: llmResponse,
  policy: 'chatbot-output',
});

if (outputCheck.action === 'block') {
  return {
    response: "I'm not able to provide that information.",
    blocked: true,
    decisionId: outputCheck.decisionId,
  };
}

return {
  response: llmResponse,
  blocked: false,
  decisionId: outputCheck.decisionId,
};
}

Input vs. Output Moderation

Both matter, but they serve different purposes:

Input moderation catches problematic requests early:

• Jailbreak attempts
• Requests for harmful instructions
• PII that shouldn't be sent to the model
• Off-topic queries (if your bot has a narrow scope)

Output moderation catches problematic responses:

• Harmful content that bypassed model safety
• Responses that violate your product policies
• Leaked system prompts or internal instructions
• Content that's technically safe but inappropriate for your audience

You can use different policies for each. Input policies tend to be more permissive (let the model try to answer), while output policies enforce your product's standards.

Handling Streaming Responses

Many chatbots stream responses for better UX. This complicates moderation because you can't check the full response before the user starts seeing it. Two approaches:

Buffer and Check

Buffer the full response, check it, then stream it to the user. This adds latency but is simpler:

async function chatWithBufferedCheck(userMessage: string) {
const stream = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: userMessage }],
  stream: true,
});

// Buffer the full response
let fullResponse = '';
for await (const chunk of stream) {
  fullResponse += chunk.choices[0]?.delta?.content || '';
}

// Check the buffered response
const check = await vettly.check({
  content: fullResponse,
  policy: 'chatbot-output',
});

if (check.action === 'block') {
  return { response: "I can't provide that information.", blocked: true };
}

return { response: fullResponse, blocked: false };
}

Stream with Periodic Checks

Stream tokens to the user but check accumulated text at intervals. If a check fails, stop the stream and retract:

async function* chatWithStreamCheck(userMessage: string) {
const stream = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: userMessage }],
  stream: true,
});

let accumulated = '';
let tokensSinceCheck = 0;

for await (const chunk of stream) {
  const token = chunk.choices[0]?.delta?.content || '';
  accumulated += token;
  tokensSinceCheck++;

  // Check every 50 tokens
  if (tokensSinceCheck >= 50) {
    const check = await vettly.check({
      content: accumulated,
      policy: 'chatbot-output',
    });

    if (check.action === 'block') {
      yield { type: 'retract', decisionId: check.decisionId };
      return;
    }
    tokensSinceCheck = 0;
  }

  yield { type: 'token', content: token };
}
}

Filtering Messages in a Chat SDK

If your chat runs on a hosted SDK (Stream, Sendbird, Twilio Conversations, PubNub) rather than your own backend, the place to moderate is the provider's server-side hook that fires before a message is delivered. Stream calls it a "before message send" webhook; Twilio Conversations uses pre-event webhooks such as onMessageAdd. Client-side checks are easy to bypass, so don't rely on them alone.

The pattern is the same for every provider:

1. Enable the pre-send webhook in the chat provider's dashboard and point it at your server
2. In the handler, send the message text (and any attachment URLs) to your moderation API
3. Allow the message, reject it, or replace its text based on the decision
4. Respond within the provider's timeout - a few seconds at most - and fail open or closed deliberately

import express from 'express';
import { Vettly } from '@vettly/sdk';

const app = express();
const vettly = new Vettly(process.env.VETTLY_API_KEY);

// Point your chat provider's pre-send webhook here.
// Verify the provider's signature before trusting the body.
app.post('/chat/pre-send', express.json(), async (req, res) => {
const text = req.body.message?.text ?? '';

const check = await vettly.check({
  content: text,
  policy: 'community-safe',
});

if (check.action === 'block') {
  // Each provider has its own "reject" shape - e.g. a non-2xx status
  // for Twilio pre-event webhooks, or an error/replaced message for Stream.
  return res.status(403).json({ reason: 'message_blocked' });
}

res.status(200).json(req.body);
});

Check your provider's docs for the exact request and response shape - the moderation call in the middle doesn't change. If the provider also offers a built-in profanity list, keep it as a first pass and use the API for context-aware categories like harassment, self-harm, and sexual content.

Policy Configuration

Your chatbot moderation policy should reflect your product's use case:

• Customer support bot: block profanity, PII exposure, competitive mentions, unauthorized promises
• Education bot: block inappropriate content for the target age group, flag misinformation
• Creative writing bot: more permissive on language, strict on illegal content
• Healthcare bot: block unqualified medical advice, flag symptom-based queries for disclaimer insertion

Logging and Compliance

Every moderation decision should be logged:

await db.chatLogs.create({
sessionId,
userMessage,
llmResponse: check.action === 'block' ? '[BLOCKED]' : llmResponse,
inputCheck: { action: inputCheck.action, decisionId: inputCheck.decisionId },
outputCheck: { action: check.action, decisionId: check.decisionId },
timestamp: new Date(),
});

This gives you an audit trail for:

• Understanding what your chatbot is saying to users
• Investigating incidents where harmful content was generated
• Compliance reporting (DSA, industry regulations)
• Improving your moderation policy based on real-world data