Scaling notes

10 breaks · v1 → v2

Cherry v1 is demo-scale on purpose: one user, one browser, no database. That was enough to prove the core idea, synthesis plus human judgment. Below is each place it breaks once hundreds of submitters and dozens of teams use it, and what v2 changes.

  1. 01Ground truth lives in one person's browser
  2. 02Two PMs correct the same issue in opposite directions
  3. 03Single-tenant, no auth
  4. 04Cost guards reset when the server restarts
  5. 05No observability
  6. 06The eval set is one person's opinion
  7. 07Prompt changes ship to everyone at once
  8. 08Data permissions are chosen per paste
  9. 09Intake is manual
  10. 10Only one person can run it

The breaks, and the fixes

01Ground truth lives in one person's browser

BreakCorrections, the human judgment Cherry learns from, are saved in localStorage: one user, one machine. A colleague's Cherry never sees what yours learned, so judgment can't build up across a team, and trend detection can only compare against your own earlier runs.

FixMove corrections, snapshots and lifecycle states into a shared store (Postgres, plus the warehouse sync in the integration architecture) so they apply across the org. Each correction records who made it, when and why, because a shared source of truth has to be auditable. Sharing corrections raises the next problem: what happens when two people disagree (02).

02Two PMs correct the same issue in opposite directions

Breakv1 has one reviewer, so their word is final. At scale, the Product PM lowers an issue's severity while the Support lead raises it. Last-write-wins throws away one person's judgment, and averaging throws away both.

FixThree rules, applied in order. (1) Check whether it's a real conflict. Usually it isn't: severity for Product and severity for Support answer different questions, which persona views already handle. A correction applies to the corrector's own view first. (2) Ownership settles shared fields. For fields with one correct value (owner, disposition), the reviewer from the team that owns the issue decides. Other teams' corrections attach as stakeholder notes instead of overwriting. (3) Keep real disagreements visible. A genuinely contested call is marked contested, shows both corrections, and goes to a steward's queue. Each ruling becomes a labeled eval case, so resolving conflicts also produces golden data.

03Single-tenant, no auth

BreakAnyone with the URL counts as the reviewer. There are no teams or roles, and no difference between who can view a triage, who can correct it, and who can send a ticket to Slack.

FixSSO, a workspace per team, and three roles: viewers; correctors, whose judgment feeds learning; and senders, who can trigger external actions. The roles line up with the autonomy ladder below, so permissions double as the guardrails.

04Cost guards reset when the server restarts

BreakThe per-IP rate limit, daily cap and 24-hour result cache live in memory on each serverless instance. They reset on cold start and aren't shared between instances. That's acceptable for a portfolio tool and a real gap at scale.

FixPut limits and cache in shared state (Redis or Vercel KV) and add cost controls. The bill is users × runs/day × tokens × price; at 800 users it becomes a budget line someone has to defend. The levers are prompt caching (already in v1), result caching, matching the model to the job (small models enrich and tag, the frontier model synthesizes), and trimming context. Set budget alerts at 50, 80 and 100%. The number to report is cost per triaged issue compared with the hour of human work it replaces.

05No observability

Breakv1 doesn't record model calls. If someone says "Tuesday's triage looked wrong," there's nothing to replay.

FixLog every call with its full input, output, model version, latency and cost, so any run can be replayed. Storage is cheap: a year of text transcripts costs less than one recorded video call. The real limits are policy: retention windows, PII, and zero-retention contracts, which the data-terms flags already enforce.

06The eval set is one person's opinion

BreakOne person wrote and graded the golden files and labeled cases. At scale, that means the evals encode a single person's idea of "correct."

FixGive each labeled set an owner and a review schedule. Feed production corrections in weekly as candidate cases, adjudicated conflicts first (see 02). Recalibrate the LLM judge against human labels every quarter. CI runs two kinds of eval: the contract check (check.mjs: is the output well-formed?) and the accuracy check (classify.mjs: are issues in the right category?). A well-formed wrong answer passes the first and fails the second, so both are needed.

07Prompt changes ship to everyone at once

BreakEditing the system prompt in v1 ships it to every user immediately. With 800 users, each wording change is a bet on the whole user base, and rolling back means digging through git late at night.

FixStore prompts as versioned config rather than code. Roll out in stages: shadow (the new prompt runs silently next to the old one and outputs are compared offline), then 5% of users, 25%, and 100%. Each stage has to clear its metrics first: correction rate, how useful receivers find the output, and eval scores. Rolling back means pointing at the previous version, which takes seconds and no deploy. Don't change the prompt and the model version in the same step: if something breaks you won't know which one caused it, and you'll have to roll back both.

08Data permissions are chosen per paste

Breakv1 asks whoever pastes feedback to declare the data terms. That's fine for a demo, but at scale nobody is going to re-read an NDA before triaging.

FixCapture terms once, when the contract is signed, as structured CRM fields: can this be used for analytics, can it be used for learning, and how long can it be kept? Each feedback record carries its account ID and inherits those flags, and the pipeline enforces them automatically. When flags are missing it defaults to deny, so mistakes err toward caution rather than learning from data it shouldn't. New contracts get a standard data-use clause, and the largest existing accounts get a one-time backfill.

09Intake is manual

BreakWeb search and paste are the only ways in. The richest signal (support tickets, Gong calls, CRM loss reasons, the warehouse) only arrives if someone copies it over by hand, so it goes stale.

FixThe connector architecture: small adapters convert each source into the format paste mode already accepts, so the core doesn't change. Work items are filed in Linear or Jira, and their status syncs back automatically. For warehouse volumes, plain SQL narrows millions of rows to the slice that needs judgment, and the model only reads that slice.

10Only one person can run it

BreakOnly one person knows how Cherry runs, how it fails and how to switch it off. When that person changes roles or goes on vacation, the knowledge goes with them.

FixBefore opening Cherry to more users, set a minimum operability bar: a one-page runbook (what it is, how to tell it's broken, how to turn it off, who takes over), a named backup owner who has run it at least once, a documented kill switch, and a list of known failure modes with how each is detected, how severe it is and how it's contained.

The order of operations

the rule

Never increase scale and autonomy in the same step. If both change at once and something goes wrong, you can't tell whether the new users' inputs or the missing human check caused it, so the only safe move is to roll everything back. Changing one at a time gives every regression a clear cause and keeps every rollback targeted.

  1. Shadow. Cherry runs, people decide, and outputs are only compared. Nothing can go wrong at this stage, and it's where the trust data comes from.
  2. Cherry proposes, a person decides. Cherry drafts everything; a person triggers anything external. (v1 is here: drafts are free, and sending to Slack is a button.)
  3. More users, same autonomy. Add teams and intake sources with the human check still in place, so new edge cases show up while someone is watching.
  4. Autonomy for reversible actions. Once months of review data support it, remove the human check from reversible actions first, one action type at a time. Tagging, clustering and drafting can run on their own; sending, closing and paying still need approval.
  5. Never fully unattended. Sampled human audits, drift alarms and the kill switch stay in place permanently.