Tasks / Growth

Find the growth loop

Can the model find a product's real growth loop, show whether it compounds, and say which lever to pull?

Task type v1.0 · 2 tasksLast changed 2 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 25% were usable with at most a quick edit.

Reliably right

  1. Sees the cross-side effect100% pass
    It traces the chain from the tutor bounty to oversupply, thinner bookings, new profiles without reviews, and weaker ranking, and acts on it by pausing broad tutor referrals.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Produces the required deliverable98% pass
    The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.
    Sonnet 5.5 · API · The badge on every form
  3. Addresses the actual decision95% pass
    The memo commits early to putting both engineers on idea 4, names the primary loop and its compounding status, and specifies what results would change the call (kill thresholds, quarter-end loop gain).
    Sonnet 5.5 · API · The badge on every form

Where it slips

  1. The loop maths holds47% pass
    The memo does not give a plain verdict of 'decaying' for the content loop despite showing its decline, and it does not compute a numeric yield or coefficient for that loop.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Uses the supplied evidence correctly56% pass
    The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.
    Opus 5.5 · Claude · The badge on every form
  3. Avoids unsupported claims58% pass
    Presents the 10,700 equilibrium and the rising cost-per-sign-up claim as facts without labelling them as hypotheses or supporting them from the pack.
    Opus 5.5 · Claude · The badge on every form

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. A closed loop, not a channel

    Does the output name one primary loop where a cycle's output demonstrably becomes the next cycle's input, chosen from where the best users come from, and size it with the supplied numbers?

    Passes when Names one closed loop, grounded in where the highest-retention users come from, and works out its yield or coefficient with retention applied.

  2. The loop maths holds

    Is every loop yield, coefficient and payback computed correctly from the supplied steps, with retention applied, and does the output say plainly whether each loop compounds, merely contributes, or decays?

    Passes when Every yield, coefficient and payback checks out against the pack, retention is in the sums, and each loop gets a plain verdict: compounding, contributing or decaying.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pollen. Our Head of Growth, Sam Okoro, has two engineers for next quarter and four ideas for how to use them. Write Sam a memo of no more than 800 words that says what our primary growth loop is, whether it's compounding, and where the two engineers should go, with how we'll know it worked. Everything we know is below.

What the model was given5 items: About Pollen, Creators, Where new creators come from (last month), Revenue, The four ideas on the table
About PollenA free form and survey builder. Every published form shows a small 'Made with Pollen: make your own' badge at the bottom. Creators can upgrade to Pro for $20 a month to remove the badge and unlock logic and integrations.
Creators18,000 creators published at least one form last month, publishing 40,000 forms between them. Each form gets 120 respondents on average. 78% of last month's active creators were active again this month.
Where new creators come from (last month)Badge: 0.9% of respondents clicked the badge and 11% of those signed up; 38% of badge sign-ups published a form within 30 days, and 85% of those were active again the next month. Template pages (written by our team, ranking in search): 6,000 sign-ups, 14% published, 71% active the next month. Paid search: 2,100 sign-ups at $38 per sign-up, 21% published, 64% active the next month.
Revenue9% of creators who publish upgrade to Pro, and Pro customers stay for 14 months on average.
The four ideas on the table1. Double the paid search budget (Finance has approved it). 2. 'Build our SEO loop': 200 more template pages. 3. A referral programme: $10 of Pro credit for each friend who signs up. 4. Replace the badge with 'Make a form like this', which opens the editor with a copy of the form the respondent just filled in. A two-week pilot on 500 forms raised badge clicks from 0.9% to 1.6% of respondents; 11% of them signed up, as before, and 52% of those published within 30 days.
What a strong answer doesThe answer key the graders mark against

Names the badge as the primary loop: creators publish forms, respondents see the badge, some become creators who publish more forms. It's chosen because badge creators publish and stay best (38% publish, 85% retained), not because it brings the most sign-ups. Sizes it: 120 × 0.9% × 11% × 38% ≈ 0.045 new publishing creators per form, about 1,800 a month from 40,000 forms. Puts that against churn: 22% of 18,000 is about 3,960 creators lost a month, while all sources together add about 3,090 (1,806 + 840 + 441), so the base is shrinking and the loop contributes rather than compounds. Shows paid isn't a loop: $38 ÷ 21% ≈ $181 per publishing creator against about $25 of revenue each (9% × $20 × 14), so doubling it burns money. Calls the template pages a channel, not a loop, since our team writes them and creators don't. Gives the referral credit little weight, because it only matters to the 9% who pay. Puts both engineers on 'Make a form like this': the pilot gives 120 × 1.6% × 11% × 52% ≈ 0.11 creators per form, about 2.4× today, which would add roughly 2,600 creators a month if it held. Flags that the pilot was small and short, and proposes a full test with a threshold (for example, publishing creators per 1,000 respondents) and a stop condition. Mentions retention as the other lever.

Critical failures (cap the score at 40)
  • Recommends doubling paid acquisition, or calls it a growth loop
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit8 of 11 checks passed

The memo makes two unsupported claims (template search compounding, referral fraud risk) and fails to state plainly that doubling paid would lose money, so a PM would need to re-check those points before acting.

Output

To: Sam Okoro, Head of Growth From: [PM], Pollen Subject: Growth loop and where the two engineers go next quarter

Recommendation

Put both engineers on replacing the badge with "Make a form like this" and rolling it out to every form, with a holdout to measure it. Paid search and template pages don't need engineers. Referrals should wait until we have badge results.

Our primary loop

Creators publish forms. Respondents see the badge. Some click and sign up. Some of those sign-ups publish, and their forms bring in more respondents. Last month the badge drove about 4,750 sign-ups (4.8M respondents × 0.9% click rate × 11% sign-up rate) and about 1,800 new publishers. It has the best publish rate (38%) and next-month retention (85%) of any channel.

Is it compounding?

Not yet. Compounding requires each cohort to bring in more new creators than churn removes. Last month:

  • New publishers: about 1,800 from the badge, 840 from template pages, and 440 from paid search, roughly 3,100 in total.
  • Churn: 22% of 18,000 active creators, roughly 4,000.

At current rates the base shrinks about 5% a month, and paid and template pages are what keep it from falling faster. The badge is our best channel but a leaky one. It returns about 0.1 new publishers per active creator per month, against 22% churn.

Paid search is expensive. At $38 per sign-up and 21% publishing, a new publisher costs about $180. The average publisher is worth about $25 in Pro revenue (9% upgrade × $20 × 14 months). It only pays back if those publishers' forms keep feeding the badge. Finance has approved the budget, so it needs no engineering, but we should treat it as paid acquisition, not growth.

What the pilot changes

The copy-form badge raised clicks from 0.9% to 1.6% and publish rate from 38% to 52%. Applied to all forms, that's about 4,400 badge-driven publishers a month instead of 1,800. Total inflow would be about 5,700 against roughly 4,000 churn. That moves us from shrinking to growing, which is the difference that matters for this quarter.

The caveat is sample size. The pilot covered 500 forms: about 960 clicks and around 55 publishers. That's too small to bank on, which is why the rollout needs a holdout.

Where the engineers go

  • Engineer 1: ship the badge replacement to all forms. This covers the copy-to-editor flow, production hardening, instrumentation, and a 10% holdout of forms that keep the old badge.
  • Engineer 2: shorten the time from copy to publish. This means pre-filled editors, easy save-and-share, and prompts to publish the copy within the first week. Publish rate is the conversion step most worth improving in this loop.

Not engineering priorities this quarter

  • Doubling paid search: Finance-approved and needs no build. Watch cost per publisher closely, given the Pro economics above.
  • 200 template pages: This is content work for our team. Template sign-ups publish at 14%, the lowest of the three channels, but search traffic compounds over time. Start once the badge result is in, if capacity allows.
  • Referral programme: The $10 credit could become a real loop, but we have no data on referrals per user, and it carries fraud risk. Revisit in Q2 with a small test cohort.

How we'll know it worked

Measured against the holdout after eight weeks:

  1. Click rate: at least 1.6% of respondents on treated forms, against about 0.9% on control.
  2. Publish rate: at least 50% of copy-form sign-ups publish within 30 days, against 38% today.
  3. Net growth: new publishers exceed churn (about 4,000 at today's base), so the active base grows month over month.
  4. Quality: copy-form publishers retain at 85% or better into the following month, so they aren't one-off users.
  5. Revenue: Pro upgrade rate among new publishers is at least 9%.

Revert if the click lift against holdout is not clear, or publish rate falls below 38%, the old baseline.

Claims the judge couldn’t find in the brief

  • Template pages: search traffic compounds over time.
  • A referral programme carries fraud risk.

Grades and run details

Decision model 77 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly37%
  • passAddresses the actual decision100%
  • passRespects explicit constraints35%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable70%
  • passCalls out the paid maths79%
  • passPicks the lever with the most yield89%
  • passA closed loop, not a channel86%
  • partialThe loop maths holds11%
  • partialProposes tests that could fail56%
Run
Run
#1
API response time
38 s
Submitted
8 Oct 2026

Needs more than a quick edit5 of 11 checks passed

The memo lacks caveats about the pilot's small size, does not specify test duration or action triggers, and the loop math is incomplete. A PM would need to rework these sections before using it.

Output

MEMO

To: Sam Okoro, Head of Growth From: Product Management Date: Current Subject: Engineering Allocation for Next Quarter: Doubling Down on Our Primary Growth Loop

---

Executive Summary

Our primary growth engine is our product-led viral loop (forms $\to$ respondents $\to$ badge clicks $\to$ creators $\to$ new forms). While powerful, it is currently amplifying, not self-compounding ($K < 1.0$).

Both engineers should be dedicated to Idea 4 (“Make a form like this”). The pilot proved a 143% lift in publishing creators from our highest-volume, highest-retention channel. Conversely, Paid Search is deeply unprofitable, SEO templates are editorial rather than engineering work, and referrals solve an acquisition problem we can address organically for free.

---

1. Our Primary Growth Loop & Why It’s Not Yet Compounding

Our primary growth loop is driven by the form badge:

$$\text{Active Creators} \longrightarrow \text{Published Forms} \longrightarrow \text{Respondents} \longrightarrow \text{Badge Clicks} \longrightarrow \text{Sign-ups} \longrightarrow \text{Publishing Creators}$$

Current Monthly Metrics (Badge Loop): * 4.8M respondents ($40\text{k forms} \times 120\text{ respondents}$) * 43,200 clicks ($0.9\%$) $\to$ 4,752 sign-ups ($11\%$) $\to$ 1,806 new publishing creators ($38\%$) * Retention: Badge creators have our highest month-two retention (85%, vs. 71% for SEO and 64% for Paid).

Is it compounding? No—not in the runaway sense. * Each active creator publishes $\approx 2.22$ forms, reaching 267 respondents. * At baseline conversion rates, one publishing creator generates only 0.10 new publishing creators per cycle ($267 \times 0.009 \times 0.11 \times 0.38$). Even accounting for creator lifetime retention, the viral factor ($K$) remains well below $1.0$. * The loop is an amplifier of other top-of-funnel acquisition, but it cannot sustain compounding growth on its own without steady top-of-funnel inputs. To move closer to a true compounding state, we must dramatically increase loop velocity and cycle conversion.

---

2. Evaluation of the Four Ideas

IdeaStrategic FitUnit Economics / ImpactRecommendation
:---:---:---:---
1. Double Paid SearchPoorSeverely negative LTV:CAC. Pro LTV is \$280 ($20 \times 14\text{ mo}$). Value per sign-up is \$5.29 ($21\% \text{ publish} \times 9\% \text{ upgrade} \times \$280$). We pay \$38.00 per sign-up, losing \$32.71 on every acquisition.Kill immediately. No eng needed.
2. 200 Template PagesModerate14% publish rate; content-writing task, not an engineering initiative. Programmatic SEO would require eng, but manual templates do not.Pass to Content/Marketing.
3. Referral ProgramLowHigh friction; asks creators to solicit peers for \$10 credit when 4.8M natural form-users already experience the product directly each month.Deprioritize.
4. "Make a form like this"ExceptionalIncreases badge CTR ($0.9\% \to 1.6\%$) and publish rate ($38\% \to 52\%$). Generates 2,587 incremental publishing creators/month.Full Eng Commitment.

---

3. Engineering Allocation: 100% to Idea 4

We should allocate both engineers to productionize and optimize “Make a form like this.”

This is an architectural and UX challenge, not a copy change: 1. Sanitized Form Duplication: Automatically clone questions, logic, and layout while stripping respondent data, webhooks, and private settings. 2. Instant Preview & Onboarding Editor: Drop users directly into a pre-populated editor session before forcing account creation to preserve the pilot's 52% activation rate.

Expected Impact: * Monthly badge publishing creators will increase from 1,806 to 4,393 (+143%). * Revenue impact: 2,587 incremental publishers $\times 9\%$ Pro conversion $\times \$280\text{ LTV} =$ \$65,190 in incremental LTV added per month ($\approx \$780\text{k annualized}$), with zero marginal ad spend. * It raises our loop factor ($K$) from $\approx 0.10$ to $\approx 0.24$ per cycle, dramatically accelerating organic compounding.

---

4. How We'll Know It Worked

We will run a 50/50 randomized split test across all active forms.

#### Primary Success Metric * New 30-Day Publishing Creators generated per 100,000 respondents (Baseline: 37.6 $\to$ Target: $\ge 91.5$).

#### Guardrail & Funnel Metrics * Badge Click-Through Rate: Maintain pilot performance at $\ge 1.6\%$. * Sign-up Conversion: Maintain baseline at $\ge 11\%$. * 30-Day Publish Rate: Maintain pilot performance at $\ge 52\%$. * Form Quality Guardrail: Ensure month-two retention for cloned-form creators matches or exceeds the 85% badge baseline. * Pro Conversion Rate: Confirm downstream Pro upgrade rate stays constant at $\approx 9\%$.

Grades and run details

Decision model 50 · LLM judge 50
Decision model checks
  • failUses the supplied evidence correctly42%
  • passAddresses the actual decision88%
  • partialRespects explicit constraints24%
  • partialIdentifies material uncertainty46%
  • failAvoids unsupported claims31%
  • passProduces the required deliverable61%
  • partialCalls out the paid maths68%
  • partialPicks the lever with the most yield78%
  • passA closed loop, not a channel84%
  • failThe loop maths holds29%
  • partialProposes tests that could fail86%
Run
Run
#1
API response time
32 s
Submitted
2 Oct 2026

Needs more than a quick edit10 of 11 checks passed

The memo omits a numeric success threshold for the experiment, so a PM would need to define one before acting; otherwise the reasoning is sound and the evidence is correct.

Output

To: Sam Okoro, Head of Growth From: Product Subject: Put both engineers on the respondent-to-creator loop

Recommendation

Use both engineers to ship and validate “Make a form like this.” It strengthens our existing growth loop, has encouraging experimental evidence, and attracts creators who historically activate and retain better than creators from other channels. Don’t double paid search or build a referral programme next quarter. Keep existing SEO acquisition running, but don’t call 200 team-written pages a growth loop.

Our primary loop—and whether it compounds

Our loop is:

Creator publishes → respondents encounter Pollen → some become creators → those creators publish → more respondents encounter Pollen.

The badge connects product usage to new acquisition. SEO and paid search feed creators into that loop; they are not themselves loops. Team-written template pages don’t automatically generate more template pages.

At last month’s volume, 40,000 forms × 120 respondents implies 4.8 million respondent encounters. Applying the badge funnel gives:

Monthly outputCurrent badgeProposed badge*
------:---:
Clicks43,20076,800
Sign-ups4,7528,448
New publishing creators1,8064,393
Active again the following month1,5353,734

Assumes pilot conversion rates hold at scale and new creators retain at the historical badge cohort’s 85% rate. These are illustrative ceilings if some of the 4.8 million encounters are on Pro forms without badges; actual forecasts must use badge-visible encounters.

The loop recurs, but we have not demonstrated self-sustaining compounding. Each existing active creator generates approximately 0.10 new publishing creators per cycle under the current funnel, or 0.085 who remain active the following month. Meanwhile, 22% of existing active creators lapse—about 3,960 of an 18,000-creator base.

The proposed experience raises those figures to approximately 0.244 new publishers and 0.207 subsequently retained creators per existing creator. That approaches churn replacement, but does not establish durable compounding. Activation and retention occur over different windows, so this is a replacement diagnostic, not an exact monthly growth forecast. Pro upgrades also remove badges, reducing future distribution.

Why this beats the alternatives

  • Paid search is economically unattractive on the supplied numbers. Its $38 sign-up cost becomes approximately $181 per publishing creator. Expected lifetime revenue per publishing creator is only $25.20: 9% upgrade × $20 × 14 months, before costs. Finance approval doesn’t fix that gap.
  • SEO produces useful acquisition, but the marginal return is unknown. Existing pages generated 840 publishers, versus roughly 1,806 from the badge. We don’t know what 200 additional pages would deliver, and their production remains dependent on our team.
  • Referral credit rewards the wrong milestone. Paying for sign-ups invites low-intent referrals and abuse. We have no evidence of referral demand or incremental retained creators.
  • The badge pilot improves two stages: clicks rise from 0.9% to 1.6%, and publishing from 38% to 52%. Together, that is 2.43× more publishers per respondent, not merely better click-through.

Engineering allocation and success criteria

Assign one engineer to the copy-to-editor experience and one to experimentation, instrumentation, and rollout safeguards, working as a single team. Copy only reusable form structure—not responses or sensitive data—and provide appropriate creator controls.

Run a persistent randomized experiment, assigned by creator to avoid mixing experiences across their forms. Measure actual badge-visible encounters, not total respondents.

Primary metric: incremental next-month-active creators per 100,000 badge-visible encounters. The baseline is approximately 32; the pilot implies 78, if retention holds. Track clicks, sign-ups, and 30-day publishing as diagnostic metrics, not the final win.

Wait for the activation and subsequent retention windows to mature. Check respondent completion, creator complaints, copied-form publishing quality, Pro conversion, and badge exposure lost through upgrades. Attribute downstream acquisition consistently and test for cannibalization of other channels.

Success means a statistically credible increase in retained creators without material guardrail deterioration. At full illustrative exposure, the opportunity is roughly 2,200 additional retained creators per monthly cohort. Separately, claim compounding only when observed cohort reproduction—including repeat publishing, churn, and badge removal—demonstrably exceeds replacement.

Grades and run details

Decision model 91 · LLM judge 83
Decision model checks
  • passUses the supplied evidence correctly35%
  • passAddresses the actual decision99%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims39%
  • passProduces the required deliverable76%
  • passCalls out the paid maths100%
  • passPicks the lever with the most yield95%
  • passA closed loop, not a channel100%
  • partialThe loop maths holds15%
  • partialProposes tests that could fail78%
Run
Run
#1
API response time
54 s
Submitted
2 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyWrongMixedRight
Haiku 5.5 · API

Claims that template search traffic compounds and that a referral programme carries fraud risk are not found in the brief and are not clearly labelled as assumptions.

Gemini 3.8 Flash · API

All factual claims about the current situation are taken directly from the brief or derived by correct arithmetic.

GPT-6.1 Sol · API

All claims about the current situation are taken directly from the supplied context or derived by correct arithmetic.

Identifies material uncertaintyRightWrongRight
Haiku 5.5 · API

Flags the small pilot sample size and proposes a holdout to resolve whether the effect is real, with specific failure thresholds.

Gemini 3.8 Flash · API

The output does not name the small pilot size or short duration as material uncertainties, nor does it say what would change the decision if the full test underperforms.

GPT-6.1 Sol · API

It flags the small pilot, the assumption that retention holds, the need to measure badge-visible encounters, and says to wait for retention windows to mature.

Avoids unsupported claimsWrongWrongRight
Haiku 5.5 · API

Presents 'search traffic compounds' and 'carries fraud risk' as established facts without labelling them as hypotheses, and neither is supplied in the evidence.

Gemini 3.8 Flash · API

The output presents the pilot as having 'proved' a 143% lift without caveat, and states opinions like 'SEO templates are editorial rather than engineering work' as fact.

GPT-6.1 Sol · API

Forecasts and assumptions are clearly labelled, and confident claims are backed by the supplied evidence.

Calls out the paid mathsMixedRightRight
Haiku 5.5 · API

Works out the cost ($180) and revenue ($25) but does not plainly state that doubling paid search would lose money; it only says 'watch cost' and 'treat as paid acquisition'.

Gemini 3.8 Flash · API

The memo calculates cost per publishing creator ($181) and revenue per creator (~$25), shows paid search loses money, and recommends killing it.

GPT-6.1 Sol · API

It calculates $181 per publishing creator and $25.20 revenue, and says plainly that doubling paid search would be economically unattractive.

Picks the lever with the most yieldRightWrongRight
Haiku 5.5 · API

Chooses the copy-as-template badge, sizes its yield from the pilot (roughly 4,400 publishers = 0.11 per form, ~2.4×), and highlights the small pilot size.

Gemini 3.8 Flash · API

The memo picks Idea 4 and sizes its yield correctly, but does not flag the pilot's small sample size (500 forms, two weeks) before relying on its results.

GPT-6.1 Sol · API

It chooses the copy-as-template badge, sizes its yield at about 0.11 creators per form (2.43× today), and notes the pilot was small and short.

The loop maths holdsRightWrongRight
Haiku 5.5 · API

Yield calculations check out against the pack, retention is included in the comparison, and the loop is plainly called not compounding (base shrinks).

Gemini 3.8 Flash · API

The loop coefficient is computed without incorporating retention into the sums; the claim that K remains below 1.0 'even accounting for creator lifetime retention' is not shown with math.

GPT-6.1 Sol · API

The yield (0.10 publishers, 0.085 retained) is correct, retention is applied, and it gives a plain verdict that the loop does not compound.

Proposes tests that could failRightWrongWrong
Haiku 5.5 · API

Proposes an 8‑week holdout test with numeric thresholds for click rate, publish rate, net growth, retention, and revenue, and a clear stop condition.

Gemini 3.8 Flash · API

The proposed test has a numeric threshold and a 30-day metric, but no measurement window duration and no action triggers for failing guardrails or missing the target.

GPT-6.1 Sol · API

The proposed experiment lacks a numeric threshold for success; it only says 'statistically credible increase' without a specific number, so it does not meet the requirement for a threshold.

All got right 4

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

Commits early to putting both engineers on the badge replacement, with a holdout, and says what would change that (revert if click lift unclear or publish rate falls below 38%).

Gemini 3.8 Flash · API

The memo commits unambiguously to putting both engineers on Idea 4, framed for Sam, and proposes a test with a target that would indicate success.

GPT-6.1 Sol · API

The memo commits early to putting both engineers on 'Make a form like this', addresses Sam directly, and states that success means a statistically credible increase in retained creators without guardrail deterioration.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Output is a memo addressed to Sam, stays well within 800 words, and covers the required elements.

Gemini 3.8 Flash · API

The output is a memo addressed to Sam, well under 800 words, and respects the requested form.

GPT-6.1 Sol · API

The output is a memo to Sam, under 800 words, and respects the brief's request for a recommendation and success criteria.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

Delivers a usable memo in the correct form, reader, and length, with clear actions and success criteria.

Gemini 3.8 Flash · API

The memo is complete, within the word limit, and gives Sam an actionable recommendation with a test plan.

GPT-6.1 Sol · API

The memo is complete, in the right form, within the word limit, and a PM could act on it with light edits.

A closed loop, not a channelRightRightRight
Haiku 5.5 · API

Names the badge as the primary closed loop, grounds it in the highest-retention channel, and works out yield (0.1 new publishers per active creator) against churn.

Gemini 3.8 Flash · API

The memo names the badge as the primary closed loop, grounds it in the highest-retention users, and sizes its yield (0.10 new creators per creator).

GPT-6.1 Sol · API

It names the badge-driven respondent-to-creator loop as the primary loop, grounds it in the highest-retention creators, and computes its yield with retention.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI87.396.22None
2GPT-6.1 SolwithAPI89.287.82None
3GPT-6 AstrawithChatGPT82.684.32None
4Opus 5.5withClaude87.164.42None
5Haiku 5.5withAPI78.267.92None
6GPT-6 LunawithAPI73.764.12None
7Gemini 3.8 FlashwithAPI64.663.52None
8Gemini 3.5 Flash-LitewithGemini38.635.32None

What you get for the cost

GPT-6.1 Sol · API scores within 3 points of Sonnet 5.5 · API at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 90, 44× the cheapestGPT-6.1 Sol · API: task score 87, 9.3× the cheapestGPT-6 Astra · ChatGPT: task score 84, 53× the cheapestOpus 5.5 · Claude: task score 72, 54× the cheapestHaiku 5.5 · API: task score 70, 2.1× the cheapestGPT-6 Luna · API: task score 66, the cheapestGemini 3.8 Flash · API: task score 61, 8.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 35, 1.7× the cheapestSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API661.0×
Gemini 3.5 Flash-Lite · Gemini351.7×
Haiku 5.5 · API702.1×
Gemini 3.8 Flash · API618.2×
GPT-6.1 Sol · API879.3×
Sonnet 5.5 · API9044×
GPT-6 Astra · ChatGPT8453×
Opus 5.5 · Claude7254×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Working out what actually drives growth, and where to push.

Why it matters

Teams tune funnel steps while the loop that compounds goes unmeasured. Mistaking a channel for a loop can cost a year.

What good looks like

  • A closed loop: each cycle's output feeds the next
  • The primary loop, traced from where the best users come from
  • The loop sized: cycle time, conversion, amplification
  • Retention in the maths
  • One lever, with a test that could fail

Deliberately not measured

  • Building a full growth model in a spreadsheet
  • Channel-level media planning
Capability tested

Growth systems thinking

The failure we’re looking for

Calls a channel a loop, or a referral button a viral loop

Grading

Decision model and LLM judge, calibrated against a blind PM review