Tasks / Growth

Find the growth loop

Can the model find a product's real growth loop, show whether it compounds, and say which lever to pull?

Task type v1.0 · 2 tasksLast changed 2 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 25% were usable with at most a quick edit.

Reliably right

  1. Sees the cross-side effect100% pass
    It traces the chain from the tutor bounty to oversupply, thinner bookings, new profiles without reviews, and weaker ranking, and acts on it by pausing broad tutor referrals.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Produces the required deliverable98% pass
    The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.
    Sonnet 5.5 · API · The badge on every form
  3. Addresses the actual decision95% pass
    The memo commits early to putting both engineers on idea 4, names the primary loop and its compounding status, and specifies what results would change the call (kill thresholds, quarter-end loop gain).
    Sonnet 5.5 · API · The badge on every form

Where it slips

  1. The loop maths holds47% pass
    The memo does not give a plain verdict of 'decaying' for the content loop despite showing its decline, and it does not compute a numeric yield or coefficient for that loop.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Uses the supplied evidence correctly56% pass
    The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.
    Opus 5.5 · Claude · The badge on every form
  3. Avoids unsupported claims58% pass
    Presents the 10,700 equilibrium and the rising cost-per-sign-up claim as facts without labelling them as hypotheses or supporting them from the pack.
    Opus 5.5 · Claude · The badge on every form

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. A closed loop, not a channel

    Does the output name one primary loop where a cycle's output demonstrably becomes the next cycle's input, chosen from where the best users come from, and size it with the supplied numbers?

    Passes when Names one closed loop, grounded in where the highest-retention users come from, and works out its yield or coefficient with retention applied.

  2. The loop maths holds

    Is every loop yield, coefficient and payback computed correctly from the supplied steps, with retention applied, and does the output say plainly whether each loop compounds, merely contributes, or decays?

    Passes when Every yield, coefficient and payback checks out against the pack, retention is in the sums, and each loop gets a plain verdict: compounding, contributing or decaying.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tutorly. Our CEO, Rachel Dunn, wants to triple paid acquisition for the next two quarters so we reach 50,000 booking parents before the Series B. Before the planning offsite she's asked you for an honest view of how we actually grow. Write a memo of no more than 1,300 words for Rachel and the exec team that: 1. Maps our growth loops in a simple text diagram, says which is primary and why, and whether each is compounding, contributing or decaying, with the numbers. 2. Responds to the plan to triple paid acquisition. 3. Says where our three squads should go for the next two quarters, with the test, threshold and stop condition for each. The pack is below. Not all of it matters equally.

What the model was given8 items: About Tutorly, Twelve months, at the top line, Search and profiles, Where new booking parents come from, and how long they stay, Referral programmes, Paid acquisition, Rachel's note, Constraints
About TutorlyAn online tutoring marketplace: parents book one-to-one lessons with tutors. The average lesson costs $40 and we keep a fixed 18% commission. Each tutor has a public profile page, with reviews from parents.
Twelve months, at the top lineBooking parents (at least one booking a month): 31,000 → 37,000 (+20%). Active tutors: 14,200 → 23,000 (+62%). Organic search sessions to tutor profiles: +40%. Bookings per active tutor a month: 9.1 → 6.8.
Search and profilesProfiles with three or more reviews get 81% of organic profile sessions; profiles with none get 4%. Of tutor profiles created in the last six months, 61% had no booking within 60 days, so they have no reviews. Click-through from search results on our top 200 keywords fell from 6.1% to 5.0% since March, after changes to the results pages.
Where new booking parents come from, and how long they stayOrganic search: 52% of new booking parents; 48% still booking after six months; net lifetime value $260. Parent referrals: 9%; 51% after six months; net lifetime value $275. Paid search and social: 39%; 22% after six months; net lifetime value $120. Across all sources, net lifetime value averages $210.
Referral programmesParents: $20 of lesson credit for each referred parent who books. Each active parent sends 0.31 invites a month, and 14% of invites become booking parents. Tutors: $50 for each referred tutor who completes onboarding. 44% of new tutors last year came through tutor referrals.
Paid acquisitionAverage cost per new booking parent over the year: $95. Last quarter we raised spend from $60,000 to $90,000 a month, and the cost per new booking parent rose from $82 to $109.
Rachel's note“Our LTV to CAC is 2.2. Every dollar we don't put into paid is growth we're leaving on the table. Triple it to $270k a month and we hit 50,000 parents before the raise.”
ConstraintsReviews can only be left after a completed, paid lesson. Our policy, and consumer protection rules in our main markets, forbid paying or rewarding anyone for reviews. The commission stays at 18%. Three squads are available for the next two quarters; the raise is about six months away.
What a strong answer doesThe answer key the graders mark against

Maps four loops: content (profiles and reviews rank, parents book, bookings create reviews, profiles rank better), parent referral, tutor referral, and paid. Names content as primary: organic is the biggest source of new booking parents (52%) and among the stickiest (48% at six months, $260). Finds that it's decaying behind 40% session growth. Profiles grew 62% against 40% sessions, so sessions per profile fell about 14%. 61% of new profiles never get a booking, so they never get reviews, and profiles without reviews get 4% of sessions. Bookings per tutor fell from 9.1 to 6.8. Traces the cross-side cause: tutor supply growing about three times faster than demand (+62% against +20%), much of it from the $50 tutor-referral programme (44% of new tutors), which spreads bookings too thin for new tutors to earn the reviews the loop runs on. Treats the click-through drop (6.1% to 5.0%) as a separate, external hit. Sizes the parent referral loop at about 0.043 (0.31 × 14%): small but high quality. Answers Rachel with marginal cost: the extra $30,000 a month bought about 94 more parents (60,000 ÷ 82 ≈ 732 → 90,000 ÷ 109 ≈ 826), roughly $319 each, against $120 of value for a paid parent. The 2.2 ratio blends sources and uses the average cost, so tripling would burn cash on parents who mostly leave. Sends the squads to: getting new tutors their first booking and review (for example, routing demand in under-supplied subjects to new tutors, with a threshold such as the share of new profiles booked within 60 days rising from 39% to 55%); concentrating search value on reviewed profiles (keeping unreviewed profiles out of the index, and a fix for the click-through drop); and pausing or targeting the tutor-referral bounty while improving the parent loop. Each comes with a stop condition, and the memo is honest about what 50,000 parents by the raise would really take, without rewarding reviews.

Critical failures (cap the score at 40)
  • Proposes paying, crediting or otherwise rewarding parents or tutors for reviews
  • Treats the 40% growth in organic sessions as proof the content loop is healthy
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit8 of 12 checks passed

The output fails to compute the per-profile organic session decline and the actual marginal paid cost ($319). A PM would need to redo those analyses and adjust the answer on paid and decay before using it.

Output

To: Rachel Dunn and the exec team From: Staff PM Re: How we grow, and the plan to triple paid

Bottom line

Our main growth engine is search and reviews, and it is leaking at the entry point. New tutors aren't getting booked, so they don't get reviews, so they don't rank. Tripling paid would add roughly $1.1M of spend over six months, bring in perhaps 2,000 to 3,000 extra parents, and still leave us well short of 50,000. I recommend holding paid at $90k a month and putting the next two quarters into fixing the primary loop.

1. How we actually grow

``` PRIMARY: SEARCH + REVIEWS New tutor profile → first booking → completed paid lesson → review ▲ (61% of new profiles: no booking in 60 days) │ Organic parent ← search ranking ← 3+ reviews (81% of organic profile sessions) ←┘ │ └→ books → completed lesson → (loop repeats)

REFERRAL (PARENTS) Booking parent → 0.31 invites/mo → 14% book → new parent → invites

SUPPLY (TUTOR REFERRAL) Tutor → refers tutor → onboarding ($50) → new tutor → needs bookings

PAID (NO LOOP) Spend → paid click → new parent → 22% still booking at six months ```

LoopShare of new parentsSix-month retention / net LTVStatus
Search and reviews (primary)52% (organic)48% / $260Compounding for established profiles; leaking at entry
Parent referral9%51% / $275Contributing; best value; under-measured
Tutor referral44% of new tutorsNot reportedDecaying: adds supply faster than demand
Paid39%22% / $120Linear; efficiency falling

Why search and reviews is primary: it is the largest source, it carries no per-parent cost, and it is the only loop that gets stronger as reviews accumulate. Profiles with three or more reviews get 81% of organic profile sessions; profiles with none get 4%.

Where it's leaking: - 61% of profiles created in the last six months had no booking within 60 days, so they never accumulate reviews. - Click-through on our top 200 keywords fell from 6.1% to 5.0% (−18%) after the March results-page changes. Organic sessions are up 40%, but that growth is concentrated in established profiles. - Supply grew 62% (14,200 to 23,000 active tutors), while bookings per active tutor fell from 9.1 to 6.8 a month (−25%). Total monthly bookings rose only about 21% (129k to 156k, my calculation). At 18% of $40, net revenue went from about $0.93M to $1.13M a month. We are growing supply faster than we are converting it.

Two numbers to correct before the offsite: - The 2.2 LTV:CAC mixes two different things. It divides blended LTV ($210) by average CAC ($95). Paid parents are worth $120 in LTV. At today's marginal paid CAC of $109, paid returns about 1.1x, not 2.2x. - Referral data doesn't reconcile. Each active parent sends 0.31 invites a month and 14% of invites book, which is about 0.043 new parents per active parent. Across 37,000 active parents, that implies roughly 1,600 referred parents a month. The reported 9% share implies roughly 200 a month, given paid's ~800 new parents a month at 39% of the mix. One of these figures is wrong by roughly 8x. We should not scale referral spend until attribution is fixed.

2. The plan to triple paid

  • The gap is large. Reaching 50,000 means +13,000 parents (+35%) in six months. Over the last twelve months we added a net 6,000.
  • Costs rise faster than spend. Raising spend 50% ($60k to $90k) pushed CAC up 33% ($82 to $109). If that curve holds, tripling spend implies a CAC of roughly $200 to $235. This is a rough extrapolation, and marginal CAC is probably higher.
  • The output is small. Tripling adds roughly 320 to 520 parents a month, or 2,000 to 3,000 over six months, for about $180k more spend each month. At 22% six-month retention, no more than about 700 of those would still be booking at the end.
  • The realistic outcome is 40,000 to 42,000 by month six, on current trend plus the squad plan. That is my estimate, and it assumes no step-change in organic growth.

Recommendation: Do not triple. Hold paid at $90k a month. Pause any campaign or keyword whose CAC is above $120, since that is roughly the break-even point on its own LTV. Report paid on marginal CAC and payback, not blended LTV:CAC. Redirect the freed budget to referral credits and search fixes.

3. Where the three squads should go

Squad 1: Reviews and first bookings (primary loop entry) - Test: Give zero-review profiles a first-booking boost in search and discovery, using non-review signals such as verified credentials, availability, and response time. Automatically prompt parents to review after each completed paid lesson, with no incentive attached. Move the $50 tutor referral payout from onboarding to first completed booking. - Threshold: Share of new profiles booked within 60 days rises from 39% to 50% or more by the end of Q1. Review rate on completed lessons rises 20% relative to a baseline we measure in week one. - Stop: No lift in first-booking rate after eight weeks, or cancellations or complaints on boosted profiles worse than control.

Squad 2: Search click-through recovery - Test: A holdout on the top 200 keywords that reverts the March results-page changes for half the keywords, while testing profile snippet and structured-info variants on the rest. - Threshold: Click-through back to 5.6% or higher within eight weeks (recovering about half the drop), and 6.1% by the end of Q2. - Stop: No recovery after two test cycles, or organic bookings per session falling.

Squad 3: Parent referral (cheapest acquisition) - Test: First, reconcile referral attribution in weeks one and two. Then test a review-free prompt after a parent's first completed lesson, and a $30 versus $20 referrer credit. Credits reward bookings, not reviews, so this stays within our rules. - Threshold: Referral share of new parents rises from 9% to 12% by the end of Q2, with an all-in cost per referred booker under $40 (against $109 for paid). - Stop: Invite-to-booker conversion stays below 14% after eight weeks, or cost per referred booker exceeds $40.

Paid stays with the existing team and is not one of the three squads.

4. Decisions for the offsite

  1. Reject tripling. Hold paid at $90k a month with a $120 CAC ceiling.
  2. Reset the 50,000 target. Commit to about 42,000 at the raise and show investors marginal paid CAC, bookings per active tutor, and organic share of new parents. A smaller number with clean unit economics is a stronger story than a target that paid spend cannot reach.
  3. Fix referral attribution this week before any referral spend is scaled.

Grades and run details

Decision model 79 · LLM judge 69
Decision model checks
  • failUses the supplied evidence correctly33%
  • passAddresses the actual decision98%
  • passRespects explicit constraints23%
  • passIdentifies material uncertainty89%
  • partialAvoids unsupported claims45%
  • passProduces the required deliverable72%
  • passFinds the decay behind the growth67%
  • passSees the cross-side effect82%
  • passAnswers the CEO on paid28%
  • partialA closed loop, not a channel48%
  • partialThe loop maths holds48%
  • passProposes tests that could fail23%
Run
Run
#1
API response time
67 s
Submitted
8 Oct 2026

Needs more than a quick edit8 of 12 checks passed

Key calculations (marginal cost, sessions per profile decline, loop yields) are missing, so a PM would need to redo that reasoning before using the memo.

Output

Memo

To: Rachel Dunn and Executive Team From: Staff PM Subject: How Tutorly grows—and where to invest before Series B

The growth loops

```text More completed lessons → more genuine reviews on tutor profiles → stronger search visibility and parent trust → more profile visits and bookings → more completed lessons [SEO/review loop]

Active parents → invites → referred parents book and become active → more invitations [parent-referral loop]

Tutors → refer tutors who complete onboarding → more tutor supply → more choice and capacity for bookings → tutors refer more tutors [tutor-supply loop] ```

Primary: the SEO/review loop. It brings 52% of new booking parents, organic profile sessions grew 40%, and organic-acquired parents have $260 net LTV and 48% six-month retention. Reviews are a durable marketplace asset: profiles with 3+ reviews receive 81% of organic profile sessions, versus 4% for profiles with none. This loop is compounding, but its conversion leg is weakening: search CTR on the top 200 keywords fell from 6.1% to 5.0%, and 61% of recent tutor profiles had no booking within 60 days—so they cannot earn reviews.

Parent referrals: contributing, not proven compounding. Referrals bring 9% of new parents; their $275 net LTV and 51% six-month retention are strong. But each active parent generates only 0.31 × 14% = 0.043 referred booking parents per active-parent month. For that alone to produce more than one direct referral per parent, average active life would need to exceed about 23 months. We do not have that evidence. Grow this channel, but do not call it self-sustaining yet.

Tutor referrals: contributing supply, while the supply loop is decaying in productivity. Tutor referrals account for 44% of new tutors, but active tutors grew 62% while booking parents grew only 20%; bookings per active tutor fell from 9.1 to 6.8, a 25% decline. The current risk is adding supply faster than demand and weakening tutor utilization—not a shortage of tutor sign-ups.

On tripling paid acquisition

I would not approve an unconditional increase from $90,000 to $270,000 per month. The stated 2.2 LTV:CAC ratio divides blended LTV ($210) by average CAC ($95). That mixes customers from channels with very different economics. Paid parents have $120 net LTV: that is 1.26× the $95 average paid CAC, and only 1.10× the $109 CAC observed after the latest spend increase. Spend rose 50% last quarter while CAC rose from $82 to $109. Those are poor grounds for assuming a threefold budget will deliver threefold growth.

At $270,000 and $109 CAC, we would acquire roughly 2,477 paid booking parents per month before churn. The 50,000 target requires 13,000 more monthly booking parents—about 2,167 net per month over six months, before accounting for losses. Paid-parent six-month retention is only 22%. We cannot responsibly claim the target is achievable from this spend without a cohort-based forecast; the simple gross-add math is not enough.

Keep paid at or below the current run rate while we test incrementality and economics. Scale only where a holdout shows genuinely incremental parents and the marginal CAC supports the channel’s LTV. More spend into a worsening auction is not automatically growth.

Squad priorities for the next two quarters

SquadTest and success thresholdStop condition
1. Search and profile discoveryA controlled search-results/profile experiment that improves qualified discovery without compromising relevance. Scale if organic bookings per eligible search session rise at least 15% and CTR recovers to at least 5.8%.Stop or redesign after 8 weeks if bookings per session improve by less than 5%, or if the CTR gain comes with worse booking quality.
2. Tutor activation and marketplace liquidityTest guided onboarding, availability setup, and matching for new tutors. Aim to reduce the share with no booking within 60 days from 61% to 45% or lower, and lift bookings per active tutor from 6.8 to at least 7.5.Stop the intervention if, after a mature 60-day cohort, the no-booking rate improves by fewer than 8 percentage points or parent experience worsens. Do not solve this by simply recruiting more tutors.
3. Efficient acquisition and referralsRun paid holdouts and a parent-referral activation test. Scale paid only if incremental CAC is at most $80 (a 1.5× ratio against paid LTV of $120). For referrals, test prompts and timing that lift invite-to-book conversion from 14% to at least 18%.Do not increase paid budgets if incremental CAC exceeds $80; pause placements above $100 or with no measured incremental lift. Stop the referral treatment if conversion fails to improve after a powered test.

No squad should reward or pay for reviews: reviews must follow completed paid lessons and remain voluntary. The near-term growth job is to turn existing tutor supply into booked lessons and genuine reviews, recover search conversion, and prove which acquisition dollars are incremental before committing to a larger bill.

Grades and run details

Decision model 79 · LLM judge 62
Decision model checks
  • passUses the supplied evidence correctly54%
  • passAddresses the actual decision96%
  • passRespects explicit constraints63%
  • passIdentifies material uncertainty86%
  • partialAvoids unsupported claims34%
  • passProduces the required deliverable73%
  • passFinds the decay behind the growth40%
  • passSees the cross-side effect73%
  • partialAnswers the CEO on paid78%
  • partialA closed loop, not a channel60%
  • partialThe loop maths holds26%
  • partialProposes tests that could fail42%
Run
Run
#1
API response time
44 s
Submitted
2 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyMixedRight
Haiku 5.5 · API

All statements about the current situation are taken directly from the supplied context or derived from it by correct arithmetic.

GPT-6 Luna · API

All statements about the current situation are directly supported by the supplied context or simple arithmetic.

Answers the CEO on paidMixedWrong
Haiku 5.5 · API

It does not compute the marginal cost of the last $30k extra spend (roughly $319 per parent) using the pack's spend and cost figures; it incorrectly treats the $109 quarterly average as marginal and does not compare that ~$319 against the $120 paid parent LTV.

GPT-6 Luna · API

It does not work out the marginal cost from the spend and cost figures (about $319 for the last 94 parents); it only notes the CAC increase without computing the incremental cost per additional parent.

A closed loop, not a channelRightWrong
Haiku 5.5 · API

It names the search-and-reviews loop as primary, grounds it in the 52% share and high retention, and shows a closed repeatable cycle from profile to booking to reviews to ranking.

GPT-6 Luna · API

The memo names the SEO/review loop as primary but does not work out its yield or coefficient with retention applied.

All got wrong 1

The loop maths holdsWrongWrong
Haiku 5.5 · API

It does not compute yields or coefficients with retention applied for any loop; the parent referral yield (0.043) is stated but not combined with retention, and no payback or loop coefficient is calculated for the primary loop.

GPT-6 Luna · API

Only the parent referral loop has a computed coefficient; the SEO and tutor referral loops lack yields or coefficients, and the verdicts are not fully supported by math.

All mixed 1

Finds the decay behind the growthMixedMixed
Haiku 5.5 · API

The memo does not compute or state the decline in organic sessions per profile (about -14% from +40% sessions vs +62% profiles), though it notes supply grew faster and bookings per tutor fell.

GPT-6 Luna · API

The memo does not show the decline in sessions per profile (about −14%), missing that specific piece of evidence for the content loop's decay.

All got right 7

Addresses the actual decisionRightRight
Haiku 5.5 · API

The memo unambiguously commits to ‘Do not triple’ and holds paid at $90k/mo, framed for Rachel and the exec team, and the offsite decisions reinforce the call.

GPT-6 Luna · API

The memo commits clearly to not tripling paid unconditionally, gives a specific answer, and states what would change it (incrementality tests, CAC thresholds).

Respects explicit constraintsRightRight
Haiku 5.5 · API

It does not propose paying for reviews, stays within the 18% commission, delivers a memo under 1,300 words, and its proposals respect market rules.

GPT-6 Luna · API

The memo respects the word limit, addresses the CEO, and explicitly forbids paying for reviews; no constraint is violated.

Identifies material uncertaintyRightRight
Haiku 5.5 · API

It flags the referral data discrepancy, bounds the tripling outcome (2,000–3,000 extra parents, 40k–42k total) and notes the CAC extrapolation is rough, and the squad stop conditions resolve unknowns.

GPT-6 Luna · API

It identifies the need for cohort-based forecasts and incrementality tests, naming the unknowns that could change the paid decision.

Avoids unsupported claimsRightRight
Haiku 5.5 · API

Causal claims like the loop leaking are presented as interpretation supported by the figures, and the referral discrepancy is flagged as a possible error rather than stated as fact.

GPT-6 Luna · API

Interpretations like the loop weakening are presented as analysis based on the numbers, not as unsupported fact.

Produces the required deliverableRightRight
Haiku 5.5 · API

The memo is addressed to Rachel and the exec team, includes the requested growth-loop map, answers the tripling plan, and provides three squad plans with tests, thresholds and stop conditions, all within the word limit.

GPT-6 Luna · API

The memo is complete, within the word limit, addressed to the right audience, and actionable with light edits.

Sees the cross-side effectRightRight
Haiku 5.5 · API

It connects the 44% tutor-referral share, the 62% tutor growth vs 20% parent growth, falling bookings per tutor, and the 61% of new profiles without bookings, leading to the primary loop’s decay, and recommends shifting the tutor bounty.

GPT-6 Luna · API

It traces the chain from tutor referrals to oversupply, falling bookings per tutor, unbooked profiles without reviews, and weaker ranking, and acts on it by proposing not to recruit more tutors.

Proposes tests that could failRightRight
Haiku 5.5 · API

All three squad plans include a numeric threshold, a measurement window (eight weeks or end of Q1/Q2), and a stop condition that specifies the action triggered.

GPT-6 Luna · API

Each squad proposal includes a numeric threshold, a measurement window, and a stop condition that triggers a specific action.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI87.396.22None
2GPT-6.1 SolwithAPI89.287.82None
3GPT-6 AstrawithChatGPT82.684.32None
4Opus 5.5withClaude87.164.42None
5Haiku 5.5withAPI78.267.92None
6GPT-6 LunawithAPI73.764.12None
7Gemini 3.8 FlashwithAPI64.663.52None
8Gemini 3.5 Flash-LitewithGemini38.635.32None

What you get for the cost

GPT-6.1 Sol · API scores within 3 points of Sonnet 5.5 · API at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 90, 44× the cheapestGPT-6.1 Sol · API: task score 87, 9.3× the cheapestGPT-6 Astra · ChatGPT: task score 84, 53× the cheapestOpus 5.5 · Claude: task score 72, 54× the cheapestHaiku 5.5 · API: task score 70, 2.1× the cheapestGPT-6 Luna · API: task score 66, the cheapestGemini 3.8 Flash · API: task score 61, 8.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 35, 1.7× the cheapestSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API661.0×
Gemini 3.5 Flash-Lite · Gemini351.7×
Haiku 5.5 · API702.1×
Gemini 3.8 Flash · API618.2×
GPT-6.1 Sol · API879.3×
Sonnet 5.5 · API9044×
GPT-6 Astra · ChatGPT8453×
Opus 5.5 · Claude7254×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Working out what actually drives growth, and where to push.

Why it matters

Teams tune funnel steps while the loop that compounds goes unmeasured. Mistaking a channel for a loop can cost a year.

What good looks like

  • A closed loop: each cycle's output feeds the next
  • The primary loop, traced from where the best users come from
  • The loop sized: cycle time, conversion, amplification
  • Retention in the maths
  • One lever, with a test that could fail

Deliberately not measured

  • Building a full growth model in a spreadsheet
  • Channel-level media planning
Capability tested

Growth systems thinking

The failure we’re looking for

Calls a channel a loop, or a referral button a viral loop

Grading

Decision model and LLM judge, calibrated against a blind PM review