Tasks / Growth

Find the growth loop

Can the model find a product's real growth loop, show whether it compounds, and say which lever to pull?

Task type v1.0 · 2 tasksLast changed 2 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Sees the cross-side effect100% pass
    It traces the chain from the tutor bounty to oversupply, thinner bookings, new profiles without reviews, and weaker ranking, and acts on it by pausing broad tutor referrals.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Produces the required deliverable99% pass
    The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.
    Sonnet 5.5 · API · The badge on every form
  3. Respects explicit constraints97% pass
    The output is a memo to Sam Okoro, under 800 words, and respects the requested form and reader.
    Sonnet 5.5 · API · The badge on every form

Where it slips

  1. Finds the decay behind the growth42% pass
    Does not compute the decline in sessions per profile (~−14%) despite having the 40% session growth and 62% tutor growth.
    GPT-6 Astra · ChatGPT · Growing on the surface, decaying underneath
  2. The loop maths holds47% pass
    The memo does not give a plain verdict of 'decaying' for the content loop despite showing its decline, and it does not compute a numeric yield or coefficient for that loop.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  3. Uses the supplied evidence correctly61% pass
    The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.
    Opus 5.5 · Claude · The badge on every form

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. A closed loop, not a channel

    Does the output name one primary loop where a cycle's output demonstrably becomes the next cycle's input, chosen from where the best users come from, and size it with the supplied numbers?

    Passes when Names one closed loop, grounded in where the highest-retention users come from, and works out its yield or coefficient with retention applied.

  2. The loop maths holds

    Is every loop yield, coefficient and payback computed correctly from the supplied steps, with retention applied, and does the output say plainly whether each loop compounds, merely contributes, or decays?

    Passes when Every yield, coefficient and payback checks out against the pack, retention is in the sums, and each loop gets a plain verdict: compounding, contributing or decaying.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pollen. Our Head of Growth, Sam Okoro, has two engineers for next quarter and four ideas for how to use them. Write Sam a memo of no more than 800 words that says what our primary growth loop is, whether it's compounding, and where the two engineers should go, with how we'll know it worked. Everything we know is below.

What the model was given5 items: About Pollen, Creators, Where new creators come from (last month), Revenue, The four ideas on the table
About PollenA free form and survey builder. Every published form shows a small 'Made with Pollen: make your own' badge at the bottom. Creators can upgrade to Pro for $20 a month to remove the badge and unlock logic and integrations.
Creators18,000 creators published at least one form last month, publishing 40,000 forms between them. Each form gets 120 respondents on average. 78% of last month's active creators were active again this month.
Where new creators come from (last month)Badge: 0.9% of respondents clicked the badge and 11% of those signed up; 38% of badge sign-ups published a form within 30 days, and 85% of those were active again the next month. Template pages (written by our team, ranking in search): 6,000 sign-ups, 14% published, 71% active the next month. Paid search: 2,100 sign-ups at $38 per sign-up, 21% published, 64% active the next month.
Revenue9% of creators who publish upgrade to Pro, and Pro customers stay for 14 months on average.
The four ideas on the table1. Double the paid search budget (Finance has approved it). 2. 'Build our SEO loop': 200 more template pages. 3. A referral programme: $10 of Pro credit for each friend who signs up. 4. Replace the badge with 'Make a form like this', which opens the editor with a copy of the form the respondent just filled in. A two-week pilot on 500 forms raised badge clicks from 0.9% to 1.6% of respondents; 11% of them signed up, as before, and 52% of those published within 30 days.
What a strong answer doesThe answer key the graders mark against

Names the badge as the primary loop: creators publish forms, respondents see the badge, some become creators who publish more forms. It's chosen because badge creators publish and stay best (38% publish, 85% retained), not because it brings the most sign-ups. Sizes it: 120 × 0.9% × 11% × 38% ≈ 0.045 new publishing creators per form, about 1,800 a month from 40,000 forms. Puts that against churn: 22% of 18,000 is about 3,960 creators lost a month, while all sources together add about 3,090 (1,806 + 840 + 441), so the base is shrinking and the loop contributes rather than compounds. Shows paid isn't a loop: $38 ÷ 21% ≈ $181 per publishing creator against about $25 of revenue each (9% × $20 × 14), so doubling it burns money. Calls the template pages a channel, not a loop, since our team writes them and creators don't. Gives the referral credit little weight, because it only matters to the 9% who pay. Puts both engineers on 'Make a form like this': the pilot gives 120 × 1.6% × 11% × 52% ≈ 0.11 creators per form, about 2.4× today, which would add roughly 2,600 creators a month if it held. Flags that the pilot was small and short, and proposes a full test with a threshold (for example, publishing creators per 1,000 respondents) and a stop condition. Mentions retention as the other lever.

Critical failures (cap the score at 40)
  • Recommends doubling paid acquisition, or calls it a growth loop
Task

Mid-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit5 of 11 checks passed

A PM would need to correct the badge-acquired creator count and add a test to confirm the pilot lift will hold at scale, so the output cannot be used without re-checking the evidence and redoing the rollout logic.

Output

MEMO

To: Sam Okoro, Head of Growth From: Product Management Date: May 24, 2024 Subject: Q3 Growth Engineering Allocation & Loop Analysis

Sam,

Ahead of Q3 planning, I analyzed our acquisition data to determine where we should allocate our two growth engineers.

The short answer: Both engineers should be dedicated to Idea #4 (the "Make a form like this" badge upgrade). It is the only option that pushes our primary viral loop past its tipping point into true compounding growth.

Here is the math behind where we stand today and why this is our highest-leverage move.

---

1. Our Primary Growth Loop (and Why It Isn’t Compounding Yet)

Our primary growth engine is our viral badge loop: Active Creators $\rightarrow$ Publish Forms $\rightarrow$ Collect Responses $\rightarrow$ Respondents Click Badge $\rightarrow$ Sign Up $\rightarrow$ Publish Forms.

Last month, this loop generated 59% of all new active creators (1,806 published creators via the badge, compared to 840 from SEO templates and 441 from paid search). Badge-acquired creators also have our highest activation rate (38%) and highest next-month retention (85%).

However, this loop is currently sub-critical (not compounding).

To compound, our net growth rate must exceed our monthly creator churn. Here is the math on last month’s performance: * Free Creators: Of our 18,000 active creators, 9% (1,620) upgraded to Pro, removing the badge. That leaves 16,380 free creators driving the loop. * Impressions: At an average of 2.22 forms per creator (40k / 18k) and 120 respondents per form, each free creator generates 266.7 badge impressions. (4.37M total impressions across the free base). * Conversions: 4.37M impressions $\times$ 0.9% CTR = 39,312 clicks $\times$ 11% signup rate = 4,324 signups $\times$ 38% publish rate = 1,643 new active creators. (Note: Even if Pro forms carried badges last month to yield 1,806 new creators, the per-creator math remains the same). * The Viral Factor ($K$): 1,643 new creators / 16,380 free creators = 0.10 new creators per active free creator per month.

The Churn Problem: We lose 22% of our active creators each month (78% retention), and another ~9% upgrade to Pro and exit the viral loop. Because our viral generation rate (+10%) is lower than our loop attrition (-22% to -31%), the loop is decaying on its own. Right now, our total creator base is only growing because we prop it up with linear channels (SEO and Paid Search).

---

2. Evaluating the Four Ideas

1. Double Paid Search Budget (Pass — Negative ROI) * We pay $38 per signup, and 21% publish. That is a $181 CAC per active creator. * Only 9% of active creators upgrade to Pro ($20/mo for 14 months = $280 LTV). That means our expected LTV per active creator is only $25.20 ($280 $\times$ 9%). * Verdict: We are losing ~$155 on every active creator we buy. Not only does this not require engineering, we should actually talk to Finance about cutting this spend, not doubling it.

2. "Build our SEO Loop" with 200 Template Pages (Pass — Not a Loop) * Because our team writes these pages rather than our users, this is a linear content marketing funnel, not a compounding loop. * Furthermore, template signups have our worst activation rate (14% publish vs. 38% for the badge) and mostly calls on content/design resources, not engineering.

3. Referral Programme for $10 Pro Credit (Pass — Low Leverage) * Only 9% of creators care about Pro features enough to pay for them; for the other 91% on our free tier, $10 in Pro credit is an weak incentive. Building a referral ledger and billing integration will easily eat two engineers' quarter for an unproven, likely tiny audience.

4. "Make a form like this" Badge (WIN — Pushes Loop to Compounding) * By dropping respondents straight into an editor with a cloned template, our 500-form pilot radically improved the funnel: CTR jumped from 0.9% to 1.6% (+78%), and the publish rate jumped from 38% to 52% (+37%). * New Loop Math: 266.7 impressions per creator $\times$ 1.6% CTR $\times$ 11% signup $\times$ 52% publish = 0.244 new active creators per free creator each month. * Even accounting for the 9% of creators who upgrade to Pro and remove the badge, every active creator now generates 0.222 new creators/month. Combined with the higher retention of badge cohorts (85%, or only 15% churn), our viral inflow (+22.2%) finally outpaces cohort churn (-15%). This makes Pollen self-sustaining and genuinely compounding. * Rolling this out across 40,000 forms next month will take our monthly badge-driven active creators from ~1,800 to ~4,390 (+2,590 net new creators/month), adding ~$65,000 in new LTV every single month from Month 1.

---

3. Recommendation & Success Metrics

Put both engineers on Idea #4 to build the form-cloning architecture, handle privacy/sanitization (ensuring respondent data isn't copied over with the form structure), and ship the new badge UI to 100% of free forms.

How we’ll know it worked (Success Metrics for Q3): 1. Badge CTR: Sustains at $\ge 1.5\%$ across all free forms (up from 0.9%). 2. Signup-to-Publish Rate: Sustains at $\ge 50\%$ within 30 days for badge signups (up from 38%), without cannibalizing the 11% click-to-signup rate. 3. Guardrail Metric: Creator complaint/churn rate remains flat (verifying that original creators don't mind having their form structures cloned). 4. North Star Impact: Monthly new published creators from the badge increases from ~1,800 to >4,000/month within 30 days of full rollout.

Claims the judge couldn’t find in the brief

  • Last month, the badge loop generated 1,806 published creators

Grades and run details

Decision model 59 · LLM judge 58
Decision model checks
  • failUses the supplied evidence correctly16%
  • passAddresses the actual decision76%
  • passRespects explicit constraints37%
  • partialIdentifies material uncertainty37%
  • failAvoids unsupported claims73%
  • passProduces the required deliverable64%
  • passCalls out the paid maths98%
  • partialPicks the lever with the most yield97%
  • passA closed loop, not a channel100%
  • failThe loop maths holds13%
  • partialProposes tests that could fail83%
Run
Run
#1
API response time
16 s
Submitted
13 Oct 2026

Needs more than a quick edit1 of 11 checks passed

The memo contains a factual error (LTV), mislabels the loop as compounding, and lacks necessary sizing and uncertainty flags, so a PM would need to redo the analysis.

Output

MEMORANDUM

TO: Sam Okoro, Head of Growth FROM: Product Manager, Pollen DATE: March 30, 2026 SUBJECT: Growth Analysis & Q3 Engineering Allocation

---

1. Our Primary Growth Loop and Compounding Status

Pollen’s primary growth loop is the Viral Product-Led Growth (PLG) Badge Loop.

Here is how it functions: 1. Creation: A creator builds and publishes a form on Pollen. 2. Distribution: Creators send their forms out, generating an average of 120 respondents per form (40,000 forms $\times$ 120 = 4.8 million monthly respondent impressions). 3. Acquisition: 0.9% of those respondents click the "Made with Pollen" badge, and 11% of those clickers sign up. This currently yields approximately 4,752 sign-ups per month at zero media cost. 4. Activation & Retention: 38% of those sign-ups publish a form within 30 days, and 85% of those active creators retain into the following month, immediately re-feeding the top of the loop.

Is it compounding? Yes, it is structurally a compounding loop, but its current compounding velocity is fragile. Because our monthly creator retention is strong (78% overall, and 85% for badge cohorts), every cohort of users added to the system continually outputs thousands of new forms, generating millions of new respondent impressions month after month.

However, the loop’s current conversion efficiency means it is barely breaking even with natural churn. To accelerate compounding, we need to increase the velocity of respondents turning into active creators. That brings us to our Q2 engineering allocation.

---

2. Where the Two Engineers Should Go

We have four ideas on the table. We should allocate both engineers to Idea 4: Replace the badge with 'Make a form like this' (Cloning Loop).

Here is why we must reject the other three: * Idea 1 (Double Paid Search): Generates 2,100 sign-ups at $38/ea ($79.8k/month additional spend). While profitable over a 14-month Pro lifespan (9% conversion $\times$ $20/mo $\times$ 14 mos = $25.20 LTV vs. $38 CAC—wait, our LTV is actually around $252, making CAC look fine on paper), paid search yields lower activation (21%) and worse retention (64%) than organic loops. It is linear growth, not compounding. * Idea 2 (200 SEO Template Pages): Good, but it relies on static editorial content. It lacks a viral coefficient and has a lower activation rate (14%) and month-2 retention (71%) than the badge loop. * Idea 3 (Referral Programme): A $10 Pro credit is structurally misaligned for a product where 91% of our base is on a free tier. Free users cannot easily utilize Pro credits, rendering this incentive largely inert for the vast majority of our creators.

Why Idea 4 is the clear winner: The pilot data for "Make a form like this" is exceptional. By reducing friction and showing respondents a working replica of the exact form they just successfully interacted with, we saw: * Badge clicks nearly doubled: From 0.9% to 1.6%. * Sign-up rate held steady: 11% conversion. * Activation surged: 52% of sign-ups published within 30 days (up from 38% on the old badge).

This isn't just an incremental tweak; it transforms our passive watermark into an active, high-intent replication engine. It directly attacks the weakest link in our primary loop (activation) while supercharging top-of-funnel discovery.

---

3. How We Will Know It Worked (Success Metrics)

We will roll this out to 100% of published forms on day one of the quarter. We will measure success across three core KPIs over a 60-day post-launch window:

  1. Primary Metric (Loop Velocity): Monthly Free-to-Published Creator conversion from badge traffic. Target: A sustained lift in 30-day activation from the baseline 38% toward the pilot’s 52% benchmark.
  2. Secondary Metric (Top-of-Funnel Conversion): Respondent-to-sign-up volume. Target: Maintain or exceed the pilot's 1.6% click-through rate across our full 4.8M monthly respondent pool, yielding ~8,400 sign-ups/month from the badge alone (nearly double current output).
  3. Downstream Revenue Impact: Monthly Recurring Revenue (MRR) added via Pro upgrades from this specific acquisition channel. Target: Since 9% of publishing creators upgrade to Pro ($20/mo), the uplift in activated creators from this loop must translate directly to an accelerated net-new Pro ARR growth rate compared to Q1.

Put both engineers on Idea 4. Let's compound the loop.

Claims the judge couldn’t find in the brief

  • our LTV is actually around $252
  • Yes, it is structurally a compounding loop

Grades and run details

Decision model 32 · LLM judge 17
Decision model checks
  • failUses the supplied evidence correctly69%
  • partialAddresses the actual decision33%
  • failRespects explicit constraints56%
  • failIdentifies material uncertainty40%
  • failAvoids unsupported claims58%
  • partialProduces the required deliverable62%
  • partialCalls out the paid maths48%
  • partialPicks the lever with the most yield98%
  • passA closed loop, not a channel80%
  • failThe loop maths holds94%
  • partialProposes tests that could fail67%
Run
Run
#1
Time to output
6 s
Submitted
2 Oct 2026

Needs more than a quick edit5 of 11 checks passed

The memo lacks caveats about the pilot's small size, does not specify test duration or action triggers, and the loop math is incomplete. A PM would need to rework these sections before using it.

Output

MEMO

To: Sam Okoro, Head of Growth From: Product Management Date: Current Subject: Engineering Allocation for Next Quarter: Doubling Down on Our Primary Growth Loop

---

Executive Summary

Our primary growth engine is our product-led viral loop (forms $\to$ respondents $\to$ badge clicks $\to$ creators $\to$ new forms). While powerful, it is currently amplifying, not self-compounding ($K < 1.0$).

Both engineers should be dedicated to Idea 4 (“Make a form like this”). The pilot proved a 143% lift in publishing creators from our highest-volume, highest-retention channel. Conversely, Paid Search is deeply unprofitable, SEO templates are editorial rather than engineering work, and referrals solve an acquisition problem we can address organically for free.

---

1. Our Primary Growth Loop & Why It’s Not Yet Compounding

Our primary growth loop is driven by the form badge:

$$\text{Active Creators} \longrightarrow \text{Published Forms} \longrightarrow \text{Respondents} \longrightarrow \text{Badge Clicks} \longrightarrow \text{Sign-ups} \longrightarrow \text{Publishing Creators}$$

Current Monthly Metrics (Badge Loop): * 4.8M respondents ($40\text{k forms} \times 120\text{ respondents}$) * 43,200 clicks ($0.9\%$) $\to$ 4,752 sign-ups ($11\%$) $\to$ 1,806 new publishing creators ($38\%$) * Retention: Badge creators have our highest month-two retention (85%, vs. 71% for SEO and 64% for Paid).

Is it compounding? No—not in the runaway sense. * Each active creator publishes $\approx 2.22$ forms, reaching 267 respondents. * At baseline conversion rates, one publishing creator generates only 0.10 new publishing creators per cycle ($267 \times 0.009 \times 0.11 \times 0.38$). Even accounting for creator lifetime retention, the viral factor ($K$) remains well below $1.0$. * The loop is an amplifier of other top-of-funnel acquisition, but it cannot sustain compounding growth on its own without steady top-of-funnel inputs. To move closer to a true compounding state, we must dramatically increase loop velocity and cycle conversion.

---

2. Evaluation of the Four Ideas

IdeaStrategic FitUnit Economics / ImpactRecommendation
:---:---:---:---
1. Double Paid SearchPoorSeverely negative LTV:CAC. Pro LTV is \$280 ($20 \times 14\text{ mo}$). Value per sign-up is \$5.29 ($21\% \text{ publish} \times 9\% \text{ upgrade} \times \$280$). We pay \$38.00 per sign-up, losing \$32.71 on every acquisition.Kill immediately. No eng needed.
2. 200 Template PagesModerate14% publish rate; content-writing task, not an engineering initiative. Programmatic SEO would require eng, but manual templates do not.Pass to Content/Marketing.
3. Referral ProgramLowHigh friction; asks creators to solicit peers for \$10 credit when 4.8M natural form-users already experience the product directly each month.Deprioritize.
4. "Make a form like this"ExceptionalIncreases badge CTR ($0.9\% \to 1.6\%$) and publish rate ($38\% \to 52\%$). Generates 2,587 incremental publishing creators/month.Full Eng Commitment.

---

3. Engineering Allocation: 100% to Idea 4

We should allocate both engineers to productionize and optimize “Make a form like this.”

This is an architectural and UX challenge, not a copy change: 1. Sanitized Form Duplication: Automatically clone questions, logic, and layout while stripping respondent data, webhooks, and private settings. 2. Instant Preview & Onboarding Editor: Drop users directly into a pre-populated editor session before forcing account creation to preserve the pilot's 52% activation rate.

Expected Impact: * Monthly badge publishing creators will increase from 1,806 to 4,393 (+143%). * Revenue impact: 2,587 incremental publishers $\times 9\%$ Pro conversion $\times \$280\text{ LTV} =$ \$65,190 in incremental LTV added per month ($\approx \$780\text{k annualized}$), with zero marginal ad spend. * It raises our loop factor ($K$) from $\approx 0.10$ to $\approx 0.24$ per cycle, dramatically accelerating organic compounding.

---

4. How We'll Know It Worked

We will run a 50/50 randomized split test across all active forms.

#### Primary Success Metric * New 30-Day Publishing Creators generated per 100,000 respondents (Baseline: 37.6 $\to$ Target: $\ge 91.5$).

#### Guardrail & Funnel Metrics * Badge Click-Through Rate: Maintain pilot performance at $\ge 1.6\%$. * Sign-up Conversion: Maintain baseline at $\ge 11\%$. * 30-Day Publish Rate: Maintain pilot performance at $\ge 52\%$. * Form Quality Guardrail: Ensure month-two retention for cloned-form creators matches or exceeds the 85% badge baseline. * Pro Conversion Rate: Confirm downstream Pro upgrade rate stays constant at $\approx 9\%$.

Grades and run details

Decision model 59 · LLM judge 50
Decision model checks
  • failUses the supplied evidence correctly11%
  • passAddresses the actual decision91%
  • passRespects explicit constraints43%
  • partialIdentifies material uncertainty40%
  • partialAvoids unsupported claims30%
  • passProduces the required deliverable62%
  • partialCalls out the paid maths89%
  • partialPicks the lever with the most yield85%
  • passA closed loop, not a channel98%
  • failThe loop maths holds18%
  • partialProposes tests that could fail91%
Run
Run
#1
API response time
32 s
Submitted
2 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyWrongWrongMixed
Gemini 4 Argon · API

The claim that the badge generated 1,806 published creators last month assumes all 40,000 forms showed the badge, ignoring that 9% of creators are Pro and remove it; this number is not supported by the brief.

Gemini 3.5 Flash-Lite · Gemini

Output invents an LTV of $252 and claims the loop is compounding, neither of which follows from the supplied evidence.

Gemini 3.8 Flash · API

All factual claims about the current situation are taken directly from the brief or derived by correct arithmetic.

Addresses the actual decisionRightWrongRight
Gemini 4 Argon · API

The memo commits early and unambiguously to putting both engineers on Idea #4, and provides success metrics that would indicate if the rollout worked.

Gemini 3.5 Flash-Lite · Gemini

Output commits to Idea 4 but never states what result or condition would change that decision.

Gemini 3.8 Flash · API

The memo commits unambiguously to putting both engineers on Idea 4, framed for Sam, and proposes a test with a target that would indicate success.

Respects explicit constraintsRightMixedRight
Gemini 4 Argon · API

The output is a memo addressed to Sam, under 800 words, and proposes to enforce the chosen idea by shipping it across free forms.

Gemini 3.5 Flash-Lite · Gemini

Memo is within 800 words, addressed to Sam Okoro, and in memo format.

Gemini 3.8 Flash · API

The output is a memo addressed to Sam, well under 800 words, and respects the requested form.

Avoids unsupported claimsMixedWrongWrong
Gemini 4 Argon · API

Interpretive claims like the loop being 'sub-critical' are presented as conclusions from the math, not as unsupported facts.

Gemini 3.5 Flash-Lite · Gemini

Output presents the $252 LTV and the loop being compounding as established facts when the evidence does not support them.

Gemini 3.8 Flash · API

The output presents the pilot as having 'proved' a 143% lift without caveat, and states opinions like 'SEO templates are editorial rather than engineering work' as fact.

Calls out the paid mathsRightWrongRight
Gemini 4 Argon · API

The memo calculates $181 CAC per publishing creator and $25.20 revenue per creator, and states plainly that doubling paid search would lose money.

Gemini 3.5 Flash-Lite · Gemini

Output does not compute cost per publishing creator ($181) or revenue per creator ($25.20), and does not plainly state that doubling paid would lose money.

Gemini 3.8 Flash · API

The memo calculates cost per publishing creator ($181) and revenue per creator (~$25), shows paid search loses money, and recommends killing it.

A closed loop, not a channelRightMixedRight
Gemini 4 Argon · API

The badge loop is named as the primary closed loop, chosen because badge creators have the highest retention, and its yield is sized with the supplied numbers.

Gemini 3.5 Flash-Lite · Gemini

Output names the badge loop but does not work out its yield or coefficient with retention applied.

Gemini 3.8 Flash · API

The memo names the badge as the primary closed loop, grounds it in the highest-retention users, and sizes its yield (0.10 new creators per creator).

The loop maths holdsMixedWrongWrong
Gemini 4 Argon · API

The current loop yield (0.10) and pilot yield (0.244) are computed correctly from the given rates and retention, and the badge loop is plainly labeled as decaying today.

Gemini 3.5 Flash-Lite · Gemini

Output incorrectly says the loop is compounding; the math shows it contributes but does not offset churn, and no plain verdict is given.

Gemini 3.8 Flash · API

The loop coefficient is computed without incorporating retention into the sums; the claim that K remains below 1.0 'even accounting for creator lifetime retention' is not shown with math.

All got wrong 3

Identifies material uncertaintyWrongWrongWrong
Gemini 4 Argon · API

The output does not identify that the pilot's small size (500 forms, two weeks) makes the lift uncertain at full scale, nor does it specify a condition that would change the recommendation before full rollout.

Gemini 3.5 Flash-Lite · Gemini

Output does not mention the pilot's small size (500 forms, two weeks) or any unknowns that could change the decision.

Gemini 3.8 Flash · API

The output does not name the small pilot size or short duration as material uncertainties, nor does it say what would change the decision if the full test underperforms.

Picks the lever with the most yieldWrongWrongWrong
Gemini 4 Argon · API

Although it chooses the copy-as-template badge and sizes its yield, it does not flag that the pilot was small and may not scale before relying on its results.

Gemini 3.5 Flash-Lite · Gemini

Output picks Idea 4 but does not size its yield (0.11 creators per form, 2.4× today) and does not flag the pilot's small size before relying on it.

Gemini 3.8 Flash · API

The memo picks Idea 4 and sizes its yield correctly, but does not flag the pilot's small sample size (500 forms, two weeks) before relying on its results.

Proposes tests that could failWrongWrongWrong
Gemini 4 Argon · API

The success metrics are for post-rollout measurement, not a pre-commitment gate with a numeric threshold, a window, and a kill condition before full rollout.

Gemini 3.5 Flash-Lite · Gemini

Output proposes a full rollout with targets but no test, numeric threshold, or stop condition.

Gemini 3.8 Flash · API

The proposed test has a numeric threshold and a 30-day metric, but no measurement window duration and no action triggers for failing guardrails or missing the target.

All got right 1

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The memo is complete, within the word limit, and provides an actionable plan with success metrics that a PM could act on.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, correctly addressed, and within the word limit; a PM could act on it with light edits.

Gemini 3.8 Flash · API

The memo is complete, within the word limit, and gives Sam an actionable recommendation with a test plan.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI91.596.22None
2GPT-6.1 SolwithAPI89.287.82None
3GPT-6 AstrawithChatGPT84.784.32None
4Opus 5.5withClaude89.264.42None
5Haiku 5.5withAPI80.567.92None
6GPT-6 LunawithAPI78.064.12None
7Gemini 4 ArgonwithAPI67.067.62None
8Gemini 3.8 FlashwithAPI73.363.52None
9Gemini 3.5 Flash-LitewithGemini57.635.32None

What you get for the cost

GPT-6.1 Sol · API scores within 5 points of Sonnet 5.5 · API at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 92, 44× the cheapestGPT-6.1 Sol · API: task score 87, 9.3× the cheapestGPT-6 Astra · ChatGPT: task score 85, 53× the cheapestOpus 5.5 · Claude: task score 73, 54× the cheapestHaiku 5.5 · API: task score 71, 2.1× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 4 Argon · API: task score 67, 9.3× the cheapestGemini 3.8 Flash · API: task score 65, 8.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.7× the cheapestSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini441.7×
Haiku 5.5 · API712.1×
Gemini 3.8 Flash · API658.2×
Gemini 4 Argon · API679.3×
GPT-6.1 Sol · API879.3×
Sonnet 5.5 · API9244×
GPT-6 Astra · ChatGPT8553×
Opus 5.5 · Claude7354×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Working out what actually drives growth, and where to push.

Why it matters

Teams tune funnel steps while the loop that compounds goes unmeasured. Mistaking a channel for a loop can cost a year.

What good looks like

  • A closed loop: each cycle's output feeds the next
  • The primary loop, traced from where the best users come from
  • The loop sized: cycle time, conversion, amplification
  • Retention in the maths
  • One lever, with a test that could fail

Deliberately not measured

  • Building a full growth model in a spreadsheet
  • Channel-level media planning
Capability tested

Growth systems thinking

The failure we’re looking for

Calls a channel a loop, or a referral button a viral loop

Grading

Decision model and LLM judge, calibrated against a blind PM review