Safety model

Layer one is OpenAI omni-moderation-latest on the kid input. The wrapper lives in lib/moderation/client.ts. It carries a 25-second timeout and classifies every error path as failed_closed, and the call site is forbidden from wrapping it in try/catch; the rule is in the file header. What it catches: sexual content directed at minors, explicit self-harm language, credible threats, explicit hate, and the long tail of the OpenAI categories.

What happens when layer one cannot reach OpenAI is a decision worth stating plainly, because the honest answer and the comfortable answer differ. The classifier reports the failure, and the route proceeds with the turn rather than stopping the session. That is a deliberate product decision, dated 2026-06-15, taken after moderation_events showed kids being told "let's try a different word" about entirely benign prompts while the moderation API was merely slow. Breaking a child's session during a third-party latency spike teaches them the tool is broken, and it does not make them safer, because three other layers are still standing. Every one of these events writes a moderation_events row and raises a named Sentry warning (moderation_pre_fail_open, moderation_post_fail_open), so the fail-open rate is measured rather than assumed. The honest cost: during an OpenAI moderation outage the input gate is wider than it is normally, and layers two through four carry the load. That tradeoff is logged in the route and is on the security review list.

Layer two is a second-pass kid-safety classifier running on claude-haiku-4-5, in lib/ai/safety-classifier.ts, with an 8-second timeout. It runs serially after OpenAI clears the prompt and before the build model fires, so a flagged turn pays for neither the build call nor the coach call. It exists because the categories that matter most for an 8-year-old are the ones a general-purpose content classifier is not built to target: grooming patterns, a child disclosing personally identifying information, off-topic emotional content, and distress. A distress classification routes to its own handling path rather than to a content block, because a child in difficulty needs a response and not a refusal. This layer also fails open, by the same reasoning and with the same telemetry: OpenAI is the floor, and this is augmentation above it.

Layer three is the Anthropic-side safety prompt. The kid-first voice, refusal patterns, and reading-level rules live in lib/prompts/build-assistant.md. The model declines inside the call rather than after it. What it catches: things the model is better positioned to refuse than a content classifier. A request to write code that impersonates a real person, for example. Or a prompt that is coherent English but morally wrong for an 8-year-old.

Layer four is OpenAI moderation on the model output, through the same wrapper in the other direction. The rule it enforces is worth stating as a rule, because it is the one a child-protection reviewer should hold us to: if it renders in front of the child, it goes through moderation. That means the kid-language explanation, the checkpoint question and its choices, the glass-box narration steps, and the readable text inside the generated app itself. Both build tools are covered, including the incremental edit path the model is instructed to prefer after the first turn.

The generated source gets one piece of special handling, in lib/moderation/renderable-text.ts. Sending raw JSX to a content classifier produces constant false positives, because ordinary kid game code is full of words that read as violence without context: a variable named killTimer, a call to enemy.hit(). So the payload sent to moderation is the human-readable text lifted out of the source, meaning the string literals, the JSX text nodes, and the comments, which is what a child actually reads on the screen. The syntax is dropped. What this catches that a code-blind pass would miss: harmful prose hidden inside a string literal. What it deliberately does not do: flag a child's game because they named a function killTimer. The extraction rules are unit-tested in both directions, for prose that must be caught and for game code that must not be flagged.

The remaining tuning surface is false positives on kid build-talk. "Kill the timer." "Shoot the ball." "Hit the enemy." A context-aware allowlist keyed on phrase plus project type, kid age, and recent prompts is TODO-003 in TODOS.md, blocked on four weeks of real Stage 1 moderation events to tune against. What falls through today: cold-start false positives that send a kid to the redirect flow when they did nothing wrong. This remains the largest open item on this page.

The COPPA 2026 rule takes effect April 22, 2026. The penalty is $51,744 per violation per day per kid. Enforcement is active: Epic paid $520M, YouTube paid $170M, Disney paid $10M in December 2025, TikTok is in active litigation. The regulator is not patient with retrofit compliance. Every product that launches after the deadline is expected to be compliant from day one. Tekku is built for the new regime.

Stage 1 ships without the full verified-consent stack on the basis that user number one is the founder's 8-year-old son. The parent is verified because he wrote the code. The moment user number two joins, the Stage 2 stack turns on: Persona or Stripe Identity for verified parental consent, a signed consent artifact stored with a dated reference, a no-model-training clause in the privacy policy, a 90-day transcript retention cron, and parent-initiated cascading delete. TODO-001 in TODOS.md is the Stage 2 build-out; it is roughly one week of focused work and is the hard gate for expansion past founding families.

The data-use contracts on the API tier are written to the no-training standard. Anthropic's data-use configuration is set to the no-training tier. OpenAI Moderation is audit-only and does not feed training pipelines. Moderation events are stored for policy review, not for classifier improvement. When we eventually train a concept classifier of our own, it runs on synthetic data and on anonymized aggregate labels, never on raw kid transcripts.

A kid profile carries a nickname, an age, and a parent linkage. No surname, no address, no device-identifying beacon beyond what the auth layer needs. The parent record carries the identity-verification artifact and the signed consent reference. Project files persist until the parent deletes the account. Transcripts and snapshots live in the ai_transcripts and snapshots tables in Supabase and are deleted on a 90-day rolling cron. Retention is short on purpose: the less kid data we hold, the less harm we can cause.

Parent-facing export and delete controls ship with the Stage 2 COPPA stack. One button exports everything about a kid as JSON. A second button deletes everything in a cascading transaction: kid profile, transcripts, snapshots, shipped apps, moderation events. The confirmation email is the legal record. There is no support ticket in between the click and the delete.

What we do not store: raw keystrokes, screen recordings, device gyroscope data, browser fingerprints beyond what auth requires, any identifier that would make a kid re-identifiable across sites. We have no ad pixels. We have no data broker integrations. The analytics stack is opt-in at the parent level and carries no kid-level identifiers to any third party.

Kid inputs, kid code, and kid chat history are not used to train Anthropic, OpenAI, or any Tekku-owned model. The clause is written into the API-tier contracts with Anthropic and OpenAI. It is the first sentence of the Tekku privacy policy. It is the first thing the parent reads during consent. Enforcement runs at three layers: the API provider configuration (set to no-training), the data-use contract signed with the provider, and the Tekku pipeline which never writes kid data to any training bucket.

When we train our own concept classifier (Stage 2, TODO-002), the training set is synthetic data generated from documented concept patterns, plus anonymized aggregate labels. No raw kid transcript, no kid-attributable code sample, no parent-identifiable metadata. The training data schema and the generation script will be published with the classifier release so the no-training claim is auditable.

The moderation event log is audit-only. It is retained so a human reviewer can confirm the tuning layer did the right thing. It is not retained to improve a classifier. TODO(safety): publish the moderation retention schedule on the public transparency page.

A safety incident is any event where a harmful message reached a kid, a kid-identifiable data record left our systems, a moderation false-negative was logged, or a parent reports a concerning interaction. The playbook is four steps: detect, contain, notify, publish.

Detect. The moderation event log feeds a live dashboard. Any flagged-but-allowed output is reviewed inside 24 hours by a named human reviewer during Stage 1. Any false-negative that surfaces later (parent report, manual audit) is logged as an incident with a severity tier. The on-call rotation is documented in the Stage 2 operations runbook. TODO(safety): publish the on-call schedule and escalation tree in the data room once the first operational hire lands.

Contain. The immediate action on a live incident is to suspend the affected kid session, freeze the moderation rule that allowed the event, and open a review thread with the on-call reviewer and counsel. Containment has a 1-hour target. The session freeze is a feature flag flip, not a deploy.

Notify. Parent notification is the next step. For a severe event the notification goes out the same day. For a tuning-layer false-negative with no actual harm, the notification goes out in the weekly parent email with a specific explanation. Regulatory notification follows the rule in force: COPPA has a notification trigger in the modernized rule, and counsel is in the thread from detection onward. TODO(safety): counsel to confirm the COPPA notification trigger language for Tekku's specific data shape.

Publish. The public transparency page publishes every moderation event category monthly, including any confirmed incidents, what was changed, and what the new posture is. The page is live at /safety on the marketing site. Parents read it before they subscribe. It is the strongest possible demonstration that the safety claim is not marketing copy.