Attest

## One-Sentence Concept ATTEST is a quality-assurance and client-reporting workspace for AI agencies and automation firms that run agents for multiple clients: it continuously grades each deployed agent against a per-client golden test set, catches accuracy regressions before the client notices, and turns the results into the monthly evidence report that justifies the maintenance retainer — measurably reducing retainer churn and the unbillable hours spent firefighting. ## Target Customer - **Industry:** AI agencies, automation firms, boutique AI consultancies, and MSPs that build and operate conversational or document-processing agents for clients. - **Company size:** 3–30 people, operating 5–40 live client deployments. Big enough that quality problems arrive faster than one person can track, small enough to have no platform team. - **Buyer:** Founder, managing partner, or head of delivery. They own margin and client retention, and they feel every churned retainer personally. - **Daily end user:** The delivery lead or automation engineer who currently reviews transcripts by hand, plus (read-only) the account manager who sends the client the monthly update. - **Trigger events**, in descending order of reliability: 1. **A client churned or nearly churned after a quality complaint the agency did not see coming.** This is the strongest trigger. The pain is fresh, quantified in lost MRR, and the cause is legible. 2. **A client asked "why am I paying you monthly if the AI does the work?"** and the agency had no artifact to answer with. This is a near-universal objection in the category and there is no good answer without measurement. 3. Crossing roughly 10 live deployments, where manual transcript review stops fitting in the week. 4. A model provider deprecation or version change forcing a re-verification pass across every client at once, with no way to do it except by hand. 5. A prospect's security or procurement review asking how quality is monitored, and the agency having nothing written down. ## Core Problem **What agencies do today.** The build is the cash flow and the retainer is the business — industry pricing guidance puts setup at $2,500–$15,000 and ongoing maintenance at $500–$5,000/month, and the retainer is described as nearly pure margin. But retainer work is not passive. Practitioner guidance is explicit that agencies must review transcripts regularly (especially the first two weeks), tune behaviour when the agent fumbles, and send monthly performance reports — one estimate puts this at 10–15 hours per week at ten clients. That review is almost entirely manual: open the conversation log, read a sample, judge it by eye, fix what looks wrong. There is no baseline, so "worse than last month" is a feeling rather than a measurement. Nobody re-runs last month's questions to check whether an answer that used to be right still is. **The tooling gap.** Real evaluation platforms exist — Langfuse, LangSmith, Braintrust, Arize, Confident AI, W&B Weave — and they are good. They are also built for a different buyer. They are developer tools priced per engineer seat with consumption metering on top, they assume an engineering team that writes evaluators in code, and their "multi-tenant" support generally means _internal teams inside one enterprise_, not _external clients of an agency_ — to the point that an agency should verify whether the licence even permits hosting third-party client data. An eight-person automation firm running twenty client bots does not have someone to own a trace-and-scorer pipeline, and the pricing model punishes exactly their shape: many small tenants, few engineers. **Where the money leaks.** - **Churn, front-loaded.** Roughly 43% of B2B client churn happens in the first 90 days rather than at renewal — the window where quality problems are most likely and least visible. - **Unbillable firefighting.** A quality problem surfaced by the client's complaint costs a scramble, an apology, and hours nobody bills. The same problem caught by a scheduled check costs a quiet fix. - **Retainer erosion.** With no evidence of work performed, the maintenance fee looks like rent. Practitioner guidance is blunt that a visible monthly results report is what keeps a retainer renewing. - **Silent regression.** Prompt edits, knowledge base updates, re-chunking, and provider version changes all degrade answer quality without throwing an error. A retrieval step returns the wrong document while every HTTP status reads 200. - **No re-verification path.** When a provider deprecates a model, the agency must re-check every client deployment by hand or ship the change blind. **Why it is urgent enough to pay for.** The cost of the problem is denominated in churned retainers. One saved $1,500/month client pays for the tool many times over, and the buyer already believes retention is their central business risk. ## Product Concept ATTEST is a workspace organized around one object: **the client deployment, and its test set.** **Setup, once per client (target: under 30 minutes).** The agency connects a deployed agent as an endpoint the tool can send questions to. It then builds a **Checkset** — 30–80 representative questions with an expected answer or an expected behaviour. Three routes in, and the third is the one that matters: type them manually; import a spreadsheet the agency already has; or point ATTEST at the deployment's existing conversation history and have it propose a starting Checkset from real traffic, which the agency edits down. Practitioner guidance suggests 50–100 hand-labelled representative questions is enough to detect regressions, so a Checkset is a realistic afternoon of work, not a project. Each Checkset entry carries an expectation type: an exact answer, a required fact, a required citation, a required refusal ("agent must decline and hand off"), or a forbidden claim. Refusal and forbidden-claim cases matter disproportionately — they are where agents embarrass their agency. **The recurring loop.** On a schedule the agency sets, ATTEST replays every Checkset against every live deployment and scores the results. The dashboard is a grid: clients down one axis, health over time across the other. Green is steady, amber is a drop worth reading, red is a regression that needs action today. When a score drops, ATTEST opens a **Regression** — the question, the previous passing answer, the current failing answer, the diff between them, and a suspected cause drawn from what changed since the last passing run (a knowledge base document was updated, the prompt was edited, the provider version moved). The delivery lead confirms or dismisses it, fixes the agent, and re-runs the Checkset to confirm recovery. Every step is timestamped. **The artifact that closes the loop.** At month end, each client gets a one-page **Quality Report**: checks run, pass rate and its trend, regressions caught and how fast they were fixed, questions the agent now handles that it did not before, and what the agency changed. This is the report the retainer renews against. It is also, notably, the thing the agency cannot produce today at any price. **A normal Tuesday.** Maya runs delivery at a six-person automation firm with fourteen live client agents. She opens ATTEST with coffee. Thirteen clients are green. One — a building-products supplier whose bot answers installer questions from product PDFs — is amber: 71% pass rate, down from 94%. She opens it. Six failures cluster on questions about one product line, and the suspected cause names a document set replaced two days ago. Spot-checking one failure, the agent cites the discontinued 2024 installation guide, which was re-uploaded alongside the new one instead of replacing it. Fifteen minutes to remove the stale document, one re-run, back to 96%. She adds two of the failed questions to the Checkset permanently so that specific mistake can never recur silently. The client never noticed. At month end they receive a report showing a regression caught and fixed in under an hour, which is precisely what they are paying $1,800/month for. Before ATTEST, Maya would have learned about this in six weeks from an angry email about an installer following the wrong instructions on a roof. **Business outcome:** fewer surprise escalations, a defensible retainer, and a quality story the agency can show prospects during a security review. ## MVP Scope **1. The single most important workflow.** Build a Checkset for one client deployment, run it on a schedule, get alerted when the pass rate drops, and see which specific questions broke and what changed. Everything else waits. **2. Minimum user roles.** Two, plus one deliberately deferred. **Agency admin** (adds clients, connects deployments, sees everything, gets billed) and **delivery member** (runs checks, resolves regressions). Client-facing read-only access is v2 — at MVP the Quality Report is a file the agency sends, which lets the agency control the narrative and removes an entire tier of permissions, branding, and support burden from v1. **3. The six highest-priority capabilities.** 1. Connect a deployment as a callable endpoint, with a connection test. 2. Checkset authoring — manual, spreadsheet import, and proposed-from-conversation-history. 3. Scheduled replay with per-question scoring against the declared expectation type. 4. A multi-client health grid with trend, so drops are visible without opening anything. 5. Regressions with before/after diff, suspected cause from a change log, and a resolve-and-re-run action. 6. The exportable monthly Quality Report, agency-branded. **4. Explicitly NOT in version one.** No marketplace, no agent distribution, no listings, no revenue share (that is CONDUIT, and it is the thing being deliberately deferred). No agent building, prompt authoring, or hosting — ATTEST never competes with how the agency builds. No live production traffic interception or tracing; scheduled replay only, which avoids the entire deep-integration problem the observability vendors solved and lets ATTEST work against any agent reachable by an endpoint. No client-facing logins. No SSO/SCIM. No custom code-defined evaluators. No cost or token analytics. No multi-step agent trajectory evaluation — question in, answer out. No white-labelling beyond a logo on the report. No public API. **5. Proof of product-market potential.** Ten paying agencies, together covering 100+ client deployments under scheduled checks. At least 30 regressions caught and resolved before the end client reported them — this is the core claim, and it must be countable. At least 5 agencies sending the Quality Report to clients unprompted, which proves the retainer-justification job is real rather than assumed. 90-day retention above 85%, and at least 3 agencies who can name a specific client they believe they retained because of it. ## AI-Native Opportunity AI does four bounded jobs, each with a human gate. None of them is a chatbot. **Grading a free-text answer against an expectation.** _Input:_ the agent's answer, the expected answer or behaviour, and the expectation type. _Task:_ judge pass, fail, or borderline with a one-line reason, since string matching cannot tell that "you'll need to replace the gasket first" satisfies "must state gasket replacement precedes reassembly. " _Human gate:_ borderline routes to human review rather than auto-failing, and any grade can be overridden — an override becomes a labelled example that tightens future grading on that Checkset. _Low confidence:_ marked borderline, never silently resolved. _Why AI:_ this is the whole reason the loop is affordable. Grading 80 questions across 14 clients by hand every week is the labour that makes manual QA collapse. **Proposing a Checkset from real conversation history.** _Input:_ exported conversation logs from an existing deployment. _Task:_ cluster real questions by intent, pick representative examples, flag the ones where the agent already looks wrong or evasive, and draft expectations. _Human gate:_ the agency edits and approves every entry before it becomes a Checkset; nothing self-activates. _Low confidence:_ proposed with a flag rather than dropped. _Why AI:_ it turns setup from a blank page into an edit pass, which is the difference between activating in an afternoon and never activating at all. This is the single highest-leverage AI use in the product because onboarding friction is what kills tools like this. **Attributing a regression to a cause.** _Input:_ the failure set, and the change log since the last passing run — documents added or removed, prompt edits, provider version changes. _Task:_ name the most likely cause with its evidence. _Human gate:_ presented as a suspicion to confirm or dismiss; dismissals are tracked, and a high dismissal rate is a bug in ATTEST, not in the user. _Low confidence:_ says "cause unclear" and shows the raw change list, which is still useful. **Drafting the monthly report narrative.** _Input:_ the month's runs, regressions, resolutions, and Checkset growth. _Task:_ draft the plain-language summary around real numbers. _Human gate:_ the agency edits before sending, and no number is generated — every figure traces to a logged run. _Why AI:_ it removes the last unbillable hour, which is what makes agencies actually send the report every month instead of the first two. **The honest limit:** an LLM judge is imperfect, and a false failure that wakes someone at 7am costs trust fast. Mitigations are structural: borderline-not-fail, overrides that teach, a visible per-Checkset agreement rate between AI grades and human overrides, and alerts that require a sustained drop rather than one bad run. ## Customer Value and ROI For an agency with 15 client deployments at an average $1,500/month retainer ($22,500 MRR): - **Time.** Manual review runs 10–15 hours/week at ten clients; scheduled grading with human review only on failures and borderlines plausibly takes that to 3–5 hours at fifteen clients. Call it 8 hours/week returned, roughly $1,600/month of delivery capacity at a modest internal rate. - **Churn.** One retained $1,500/month client is $18,000 of annual revenue. Given that 43% of B2B churn lands in the first 90 days, catching quality problems during onboarding is aimed directly at the largest slice of the risk. - **Retainer defensibility.** The "why am I paying monthly?" conversation moves from hand-waving to a document. Even holding one $1,500 retainer that would otherwise have been renegotiated downward pays for the tool for a year. - **Sales.** A quality-monitoring process is an answer to the prospect question agencies currently fumble, and it supports a higher retainer rather than a discount. **Before:** quality is a feeling, problems arrive as client complaints, retainer value is asserted, and provider changes are shipped blind and hoped over. **After:** every deployment has a pass rate with a trend, regressions surface within a day with a suspected cause, retainer value arrives monthly as evidence, and a provider change is a re-run with a visible delta. ## Monetization **Who pays:** the agency. Never the end client — that would recreate CONDUIT's two-sided problem and put ATTEST between the agency and its customer, which is the relationship the agency guards most jealously. **Pricing metric: per monitored client deployment.** This is the crux of the wedge. The incumbents price per engineer seat with consumption metering; the agency's business scales by _client count_, not headcount, and per-seat pricing actively penalizes the shape of their business. Charging per deployment means ATTEST's bill rises exactly as the agency's revenue rises, which makes the expense self-justifying and the pricing conversation short. | Tier | Deployments | Price | Rationale | | -------- | ----------- | ---------- | ------------------------------------------------------------------------------------- | | Solo | up to 3 | $99/month | The independent operator. Priced to be an easy personal decision. | | Studio | up to 15 | $349/month | The core tier — matches the 5–15 deployment firm where manual review has just broken. | | Firm | up to 40 | $799/month | Multiple delivery leads, report volume, higher check frequency. | | Above 40 | — | custom | Rare at this segment; handle by conversation. | **Why the price is justified.** At the Studio tier, $349/month against 15 retainers worth $22,500 MRR is about 1.5% of the revenue being protected. One retained client covers roughly four years of subscription. The internal comparison an agency will actually make is against per-seat evaluation tooling — a developer platform at $39–$50 per seat per month plus consumption metering, needing an engineer who can write evaluators, is both more expensive in practice and unusable by the person who would operate it. **Deliberately excluded from v1:** no usage metering on checks run (unpredictable bills are what agencies hate about their own vendors, and metering would discourage the frequent running that makes the product work), no setup fee, no annual lock-in before the product has earned it. Grading costs money per run, so cap check frequency by tier and monitor gross margin from day one — this is the one place the simple pricing model could bleed. ## Competitive Landscape **Manual review in a spreadsheet.** The real incumbent, and the one to beat. Free, flexible, already in place. It loses on repetition — nobody re-runs last month's questions — so it cannot detect regression, only current badness. ATTEST wins by making the _second_ run free. **Developer evaluation platforms** — Braintrust, LangSmith, Langfuse, Arize, Confident AI, W&B Weave. Genuinely strong products with real funding behind them (Braintrust raised an $80M Series B at an $800M valuation in February 2026; Cisco acquired Galileo in April 2026 and folded it into Splunk). They are built for an engineering team monitoring its own application, priced per engineer seat, and operated in code. Their multi-tenancy generally means internal teams within one enterprise, not external agency clients — an agency may need to check whether the licence even permits hosting third-party client data. **The wedge is not "better evals." It is a different buyer, a different unit of account, and a client-facing artifact none of them produce.** A tool for a platform engineer at a 400-person company is not a tool for a delivery lead at an eight-person automation firm, and the gap is organizational, not technical. **Observability platforms and the open-source stack** (Langfuse self-hosted, Ragas, DeepEval, Phoenix). Powerful and free, and they demand an engineer to assemble and maintain. Running one Langfuse instance per client for isolation is a real option — and it is an infrastructure project, which is precisely the thing a small agency will not staff. **The agent platforms the agency builds on** — Botpress, Voiceflow, Chatbase, Intercom, Salesforce Agentforce. Each offers analytics for agents built on its own platform. Two structural weaknesses: an agency's clients are spread across several platforms, so no single vendor's dashboard covers the portfolio; and a vendor grading its own agent is not evidence a client's procurement team will weigh. ATTEST is deliberately cross-platform and deliberately not the builder — independence is part of the product. **Doing nothing.** Common and rational until the first bad churn. This is why the churned-client trigger is the one to sell into. **The wedge in one line:** the evaluation category is built and priced for engineering teams grading their own product; nobody has built it for the service firm grading twenty of somebody else's, whose deliverable is not a dashboard but a monthly document that keeps a retainer alive. ## Defensibility - **The Checkset is the asset, and the agency builds it.** Every regression caught adds questions. After a year, a client's Checkset encodes hundreds of hours of accumulated knowledge about what that agent must never get wrong. Leaving means abandoning it — or exporting a spreadsheet and losing the history, the trend, and the graded baseline that make it meaningful. Make export easy anyway; the switching cost is real without being a hostage-taking, and the segment punishes lock-in. - **Workflow embedding at a monthly cadence.** The Quality Report becomes part of how the agency's clients experience the retainer. Once a client expects the report, the agency cannot quietly stop sending it. - **Cross-agency benchmarking, later and carefully.** With enough deployments, ATTEST can say "your pass rate is in the top quartile for document-Q&A agents of this size" — valuable to the agency, unavailable to any single-tenant tool, and dependent on aggregate data nobody else holds. Opt-in and anonymized only, and worth nothing until scale, so treat it as year-two upside rather than an early moat. - **Grading calibration per Checkset.** Every human override teaches the judge for that client's domain. A two-year-old Checkset grades that vertical's language better than a fresh one, which is a data advantage that compounds quietly. - **Quality history as a procurement artifact.** An agency with eighteen months of documented monitoring wins security reviews against one without. That history lives in ATTEST. - **The option on CONDUIT.** Hundreds of agencies with graded, versioned, evidence-bearing agents is the supply side of a marketplace, already proven and already measured. If the marketplace thesis is ever worth pursuing, this is the only sane order to attempt it in. **Honest limit on the moat.** In year one there is little defensibility beyond accumulated Checksets and switching friction. A funded evaluation vendor could ship an agency tier and per-deployment pricing. What protects the position is that they are structurally aimed elsewhere: their buyer is an engineering team, their pricing is per seat, their product surface assumes code, and a client-facing branded report is a strange feature for a developer tool. That is a real window, not a permanent one. ## Go-to-Market Starting Point **First segment:** AI automation agencies and boutique AI consultancies with 5–20 live client deployments, English-speaking, 3–15 people. Narrow further at the start to those running document-Q&A or support agents, where questions have checkable answers — the product demos best where correctness is unambiguous. **Where to find them.** The partner and expert directories of the agent-building platforms (Botpress, Voiceflow, Chatbase, Make, n8n) are public lists of firms that build agents for clients, with contact details and often their client verticals. AI automation communities and paid cohort programs where these operators congregate. YouTube and newsletter creators teaching the AI-agency business model, whose audience is the entire target market and who need genuinely useful material. Regional AI meetups. **Content angle: publish the thing nobody else can.** Run Checksets against a set of real deployed agents (with permission) and publish what breaks and how often, with the methodology attached. _"We ran 500 questions against 20 live client chatbots every week for a month. Nine degraded. Here is what changed and how long it took anyone to notice. "_ Concrete, useful, and it makes the case for the category without arguing for the product. It doubles as lead generation because every agency reading it wants to know whether they are one of the nine. A cautionary note on the numbers circulating in this space: claims like "68% of enterprise RAG deployments lose meaningful accuracy within three months" are widely repeated but, on inspection, untraceable (see Sources). Do not build marketing on them. Generate your own figures — that is exactly what makes the content defensible and the category argument credible. **Pilot offer.** Free setup done by you, personally: you build the first client's Checkset with them on a call, connect the deployment, and run it for 30 days at no charge. The concierge onboarding is not generosity — Checkset quality decides whether the product works, and doing the first one yourself teaches you what to automate. Convert to paid at day 30 with the first Quality Report in hand. **Why they take the meeting.** "How would you know if one of your client agents got worse this week — before the client tells you?" Every honest operator in this segment knows the answer is that they would not, and most have a specific painful memory attached to it. ## Main Risks and Validation Questions **Biggest assumption, the one that kills it:** that agencies experience silent quality degradation _often enough to feel it_, and would rather pay to prevent it than continue absorbing it. The failure mode is not that agencies say quality matters — everyone says that. It is that they shrug and say problems are rare enough to handle when clients report them, and that clients renew anyway without a report. If retainers renew fine on relationship alone, ATTEST is a vitamin. **Second assumption, nearly as dangerous:** that Checkset creation actually happens. A tool requiring 50–80 hand-written questions per client before delivering any value has a brutal activation cliff. If agencies sign up and never finish setup, nothing else matters. This is why proposing Checksets from real conversation history is a v1 capability rather than a v2 nicety, and it is the first thing to prototype. **Other risks.** Grading accuracy — false alarms destroy trust faster than missed regressions. Margin — grading cost scales with checks run while pricing is flat. Endpoint access — some agencies build on platforms with no callable endpoint, which simply excludes them from v1 and needs early quantification. Client data sensitivity — ATTEST holds real questions and answers from the agency's clients, so isolation and retention terms matter from day one. And the segment itself is churn-prone: many AI agencies started in the last two years and some will not survive, so expect customer mortality unrelated to your product. **Three interview questions.** Past behaviour only — no hypotheticals about whether they would like this. 1. _"Tell me about the last time a client told you their agent was getting something wrong. How did you find out, how long had it been happening, and what did it cost you to fix?"_ — establishes whether silent degradation is real and remembered, or theoretical. 2. _"Walk me through what you actually did last month to check your live client agents were still working. Show me the spreadsheet or the doc if there is one."_ — reveals the true current workflow, and whether any artifact exists. Asking to see it separates real process from described process. 3. _"When a client questions the monthly fee, what do you send them?"_ — tests whether the retainer-justification job is felt. If they have nothing and it stings, that is the buying trigger. Follow-up worth asking: _"Have you lost a client over quality? What happened?"_ — the answer predicts urgency better than any other question. **Low-cost validation experiment.** No product. Recruit five agencies and offer a free manual "quality audit" of two client deployments each. Write 40 questions per deployment by hand with them, run the questions manually, grade the answers, and deliver a written one-page report. Repeat 30 days later on the same questions. This tests everything that matters: whether degradation appears between the two runs, whether building a Checkset is tolerable or agonizing, whether the report lands with the agency and gets forwarded to their client, and whether any of the five will pay for the second month. Cost is your time plus modest inference spend, and it produces both the content asset above and the first customers. **Strongest positive signal:** an agency forwards a Quality Report to their end client unprompted, then asks whether they can get it for their other twelve deployments. That combination — voluntary external use plus an expansion request — is worth more than any stated purchase intent. **Strongest negative signal:** agencies find the audit interesting, agree the problem is real, and say they will just add a recurring calendar reminder to check manually. That means the pain is real but small, and a tool is not the answer. ## Idea Scorecard | Dimension | Score | Justification | | ------------------------------ | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Pain severity** | 7 | Real, recurring, and tied to churned revenue — but it is prevention, and prevention always scores below a burning fire. The pain is sharpest immediately after a quality-driven churn and fades between incidents. | | **Willingness to pay** | 7 | The ROI arithmetic is easy and the price is small against protected MRR. Held down because small agencies are cash-conscious and habitually cheap on internal tooling, preferring free and self-hosted. | | **Ease of reaching customers** | 9 | The best dimension by a distance. This segment is publicly listed in platform partner directories, clustered in identifiable communities, reachable by cold outreach, and buys fast without procurement. One person can reach hundreds. | | **MVP simplicity** | 8 | Endpoint call, scheduled replay, graded comparison, report. No deep integrations, no live traffic interception, no marketplace, no billing complexity, one paying side. **CONDUIT scored 4 here — this is the entire point of the exercise. ** | | **Market size** | 5 | The honest weak spot. Thousands, not millions, of agencies at this size, with an ACV of $1,200–$9,600. A credible single-founder business in the low seven figures; not a venture-scale outcome without expanding to in-house teams running their own agents — where the same product works and the market is far larger, but the client-report wedge disappears. | | **Competitive intensity** | 6 | The evaluation category is crowded and well funded, but aimed at a different buyer with the wrong pricing metric. The specific position is open. Scored mid because the distance an incumbent must travel to reach it is short if they choose to. | | **Defensibility potential** | 6 | Accumulated Checksets, grading calibration, and workflow embedding are genuine and compound slowly. Cross-agency benchmarking is real but far off. Thin in year one. | | **Founder suitability** | 9 | A solo founder can build, sell, and support this. No two-sided cold start, no merchant-of-record obligation, no regulated billing, no marketplace trust layer. Concierge onboarding is a competitive advantage at small scale rather than a burden. **CONDUIT scored 5 here. ** | **Weighted read:** ATTEST trades market size and pain severity for reachability, buildability, and founder fit. It is the version a solo founder can start on Monday and have paying customers for within a quarter — and it accumulates precisely the asset that would make the larger marketplace idea viable later, if it ever is.