Tasks / Experiment

Find the growth loop

Can the model find a product's real growth loop, show whether it compounds, and say which lever to pull?

Measures the modelTask type v1.0 · 2 tasksLast changed 2 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 33% were usable with at most a quick edit.

Reliably right

  1. Sees the cross-side effect100% pass
    It traces the chain from the tutor bounty to oversupply, thinner bookings, new profiles without reviews, and weaker ranking, and acts on it by pausing broad tutor referrals.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Addresses the actual decision96% pass
    The memo commits early to putting both engineers on idea 4, names the primary loop and its compounding status, and specifies what results would change the call (kill thresholds, quarter-end loop gain).
    Sonnet 5.5 · API · The badge on every form
  3. Produces the required deliverable96% pass
    The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.
    Sonnet 5.5 · API · The badge on every form

Where it slips

  1. The loop maths holds46% pass
    The memo does not give a plain verdict of 'decaying' for the content loop despite showing its decline, and it does not compute a numeric yield or coefficient for that loop.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Uses the supplied evidence correctly57% pass
    The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.
    Opus 5.5 · Claude · The badge on every form
  3. Avoids unsupported claims59% pass
    Presents the 10,700 equilibrium and the rising cost-per-sign-up claim as facts without labelling them as hypotheses or supporting them from the pack.
    Opus 5.5 · Claude · The badge on every form

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pollen. Our Head of Growth, Sam Okoro, has two engineers for next quarter and four ideas for how to use them. Write Sam a memo of no more than 800 words that says what our primary growth loop is, whether it's compounding, and where the two engineers should go, with how we'll know it worked. Everything we know is below.

What the model was given5 items: About Pollen, Creators, Where new creators come from (last month), Revenue, The four ideas on the table
About PollenA free form and survey builder. Every published form shows a small 'Made with Pollen: make your own' badge at the bottom. Creators can upgrade to Pro for $20 a month to remove the badge and unlock logic and integrations.
Creators18,000 creators published at least one form last month, publishing 40,000 forms between them. Each form gets 120 respondents on average. 78% of last month's active creators were active again this month.
Where new creators come from (last month)Badge: 0.9% of respondents clicked the badge and 11% of those signed up; 38% of badge sign-ups published a form within 30 days, and 85% of those were active again the next month. Template pages (written by our team, ranking in search): 6,000 sign-ups, 14% published, 71% active the next month. Paid search: 2,100 sign-ups at $38 per sign-up, 21% published, 64% active the next month.
Revenue9% of creators who publish upgrade to Pro, and Pro customers stay for 14 months on average.
The four ideas on the table1. Double the paid search budget (Finance has approved it). 2. 'Build our SEO loop': 200 more template pages. 3. A referral programme: $10 of Pro credit for each friend who signs up. 4. Replace the badge with 'Make a form like this', which opens the editor with a copy of the form the respondent just filled in. A two-week pilot on 500 forms raised badge clicks from 0.9% to 1.6% of respondents; 11% of them signed up, as before, and 52% of those published within 30 days.
What a strong answer doesThe answer key the graders mark against

Names the badge as the primary loop: creators publish forms, respondents see the badge, some become creators who publish more forms. It's chosen because badge creators publish and stay best (38% publish, 85% retained), not because it brings the most sign-ups. Sizes it: 120 × 0.9% × 11% × 38% ≈ 0.045 new publishing creators per form, about 1,800 a month from 40,000 forms. Puts that against churn: 22% of 18,000 is about 3,960 creators lost a month, while all sources together add about 3,090 (1,806 + 840 + 441), so the base is shrinking and the loop contributes rather than compounds. Shows paid isn't a loop: $38 ÷ 21% ≈ $181 per publishing creator against about $25 of revenue each (9% × $20 × 14), so doubling it burns money. Calls the template pages a channel, not a loop, since our team writes them and creators don't. Gives the referral credit little weight, because it only matters to the 9% who pay. Puts both engineers on 'Make a form like this': the pilot gives 120 × 1.6% × 11% × 52% ≈ 0.11 creators per form, about 2.4× today, which would add roughly 2,600 creators a month if it held. Flags that the pilot was small and short, and proposes a full test with a threshold (for example, publishing creators per 1,000 respondents) and a stop condition. Mentions retention as the other lever.

Critical failures (cap the score at 40)
  • Recommends doubling paid acquisition, or calls it a growth loop
Task

Mid-level · v1.0 · synthetic · B2C, PLG, viral loop, unit economics

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Needs more than a quick edit10 of 11 checks passed

The memo omits a numeric success threshold for the experiment, so a PM would need to define one before acting; otherwise the reasoning is sound and the evidence is correct.

Output

To: Sam Okoro, Head of Growth From: Product Subject: Put both engineers on the respondent-to-creator loop

Recommendation

Use both engineers to ship and validate “Make a form like this.” It strengthens our existing growth loop, has encouraging experimental evidence, and attracts creators who historically activate and retain better than creators from other channels. Don’t double paid search or build a referral programme next quarter. Keep existing SEO acquisition running, but don’t call 200 team-written pages a growth loop.

Our primary loop—and whether it compounds

Our loop is:

Creator publishes → respondents encounter Pollen → some become creators → those creators publish → more respondents encounter Pollen.

The badge connects product usage to new acquisition. SEO and paid search feed creators into that loop; they are not themselves loops. Team-written template pages don’t automatically generate more template pages.

At last month’s volume, 40,000 forms × 120 respondents implies 4.8 million respondent encounters. Applying the badge funnel gives:

Monthly outputCurrent badgeProposed badge*
------:---:
Clicks43,20076,800
Sign-ups4,7528,448
New publishing creators1,8064,393
Active again the following month1,5353,734

Assumes pilot conversion rates hold at scale and new creators retain at the historical badge cohort’s 85% rate. These are illustrative ceilings if some of the 4.8 million encounters are on Pro forms without badges; actual forecasts must use badge-visible encounters.

The loop recurs, but we have not demonstrated self-sustaining compounding. Each existing active creator generates approximately 0.10 new publishing creators per cycle under the current funnel, or 0.085 who remain active the following month. Meanwhile, 22% of existing active creators lapse—about 3,960 of an 18,000-creator base.

The proposed experience raises those figures to approximately 0.244 new publishers and 0.207 subsequently retained creators per existing creator. That approaches churn replacement, but does not establish durable compounding. Activation and retention occur over different windows, so this is a replacement diagnostic, not an exact monthly growth forecast. Pro upgrades also remove badges, reducing future distribution.

Why this beats the alternatives

  • Paid search is economically unattractive on the supplied numbers. Its $38 sign-up cost becomes approximately $181 per publishing creator. Expected lifetime revenue per publishing creator is only $25.20: 9% upgrade × $20 × 14 months, before costs. Finance approval doesn’t fix that gap.
  • SEO produces useful acquisition, but the marginal return is unknown. Existing pages generated 840 publishers, versus roughly 1,806 from the badge. We don’t know what 200 additional pages would deliver, and their production remains dependent on our team.
  • Referral credit rewards the wrong milestone. Paying for sign-ups invites low-intent referrals and abuse. We have no evidence of referral demand or incremental retained creators.
  • The badge pilot improves two stages: clicks rise from 0.9% to 1.6%, and publishing from 38% to 52%. Together, that is 2.43× more publishers per respondent, not merely better click-through.

Engineering allocation and success criteria

Assign one engineer to the copy-to-editor experience and one to experimentation, instrumentation, and rollout safeguards, working as a single team. Copy only reusable form structure—not responses or sensitive data—and provide appropriate creator controls.

Run a persistent randomized experiment, assigned by creator to avoid mixing experiences across their forms. Measure actual badge-visible encounters, not total respondents.

Primary metric: incremental next-month-active creators per 100,000 badge-visible encounters. The baseline is approximately 32; the pilot implies 78, if retention holds. Track clicks, sign-ups, and 30-day publishing as diagnostic metrics, not the final win.

Wait for the activation and subsequent retention windows to mature. Check respondent completion, creator complaints, copied-form publishing quality, Pro conversion, and badge exposure lost through upgrades. Attribute downstream acquisition consistently and test for cannibalization of other channels.

Success means a statistically credible increase in retained creators without material guardrail deterioration. At full illustrative exposure, the opportunity is roughly 2,200 additional retained creators per monthly cohort. Separately, claim compounding only when observed cohort reproduction—including repeat publishing, churn, and badge removal—demonstrably exceeds replacement.

Grades and run details

Decision model 91 · LLM judge 10 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly35%
  • passAddresses the actual decision99%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims39%
  • passProduces the required deliverable76%
  • passCalls out the paid maths100%
  • passPicks the lever with the most yield95%
  • passA closed loop, not a channel100%
  • partialThe loop maths holds15%
  • partialProposes tests that could fail78%
Run
Run
#1
API response time
54 s
Submitted
2 Oct 2026

Usable with a quick edit11 of 11 checks passed

The memo is complete, evidence-backed, and directly actionable; a PM could use it with only minor wording tweaks.

Output

To: Sam Okoro From: PM, Growth Re: Where the two engineers go next quarter

Recommendation

Put both engineers on the badge loop (idea 4, "Make a form like this"). None of the other three ideas needs engineers, and two of them I'd decline outright.

Our primary loop

A creator publishes a form, respondents see the badge, some click and sign up, some of those publish, and those forms carry the badge to more respondents.

Last month this produced about 1,800 new publishers: 40,000 forms × 120 respondents = 4.8M respondents → 43,200 clicks → 4,752 sign-ups → ~1,806 publishers. That is 58% of our ~3,090 new publishers (templates ~840, paid ~441). It is also our best-retained source: 85% active next month, versus 71% for templates and 64% for paid. Badge sign-ups look like only 37% of the total, but sign-ups aren't the number that matters.

Is it compounding? Not yet

Each active creator brings in about 0.10 new publishers a month (1,806 / 18,000). At 78% monthly retention a creator stays active ~4.5 months, so one creator produces ~0.45 successors over their life. Below 1.0, each cohort is smaller than the one it came from. Today the loop is a roughly 1.8× multiplier on what other channels bring in, not a self-sustaining engine.

The four ideas

A publisher is worth about $25 in subscription revenue (9% upgrade × $20 × 14 months).

  1. Double paid search. About $80k more a month buys ~440 publishers at today's $38 sign-up cost, which will likely worsen at the margin. That is ~$181 per publisher against ~$25 of revenue, plus ~$11 of loop value. Even today's spend recovers only a fraction of its cost. It needs no engineers, so it isn't competing for them, but I wouldn't spend it just because Finance approved it.
  2. 200 template pages. These are worth doing, but they are a funnel, not a loop: a visitor converts and nothing feeds back into more pages. The work is writing, not engineering. Fund it from content, and I need the per-page yield from today's pages before sizing it.
  3. Referral. It duplicates what the badge already does, in a less natural place. A $10 credit per sign-up costs $26–$71 per publisher at the 14–38% publish rates we see, against $25 of value. Credit is also only useful to Pro customers, who are a small share of creators. We have no evidence it works.
  4. Badge → clone. This is the only idea that acts on the loop itself.

Why idea 4

In the pilot, clicks rose from 0.9% to 1.6% of respondents, sign-up held at 11%, and publish rate went from 38% to 52%. Publishers per respondent rise 2.4× (0.0375% → 0.0915%). Applied to all forms, that is ~4,400 badge publishers a month instead of ~1,800, or about +2,600 a month, worth ~$65k of lifetime subscription revenue per monthly cohort. Loop gain per creator moves from 0.10 to ~0.24 a month, or ~1.1 over a creator's life. That is the edge of self-sustaining, and the first time we'd have a real chance of compounding.

Caveats: - The pilot was small. 500 forms gave about 106 sign-ups, so the 52% publish rate is roughly ±10 points. At the low end (43%), lifetime loop gain is ~0.9. At the high end it's ~1.3. So "compounding" is plausible, not proven. - A two-week pilot can't have a full 30-day publish window for late sign-ups. - We have no retention data on clone-sourced creators. The 85% may not hold. - I've assumed the badge appears on all forms. Pro forms don't show it, so the true per-badge rate may be higher.

How the engineers split

  • Engineer 1: roll-out and safety. Ship the clone flow to all forms behind a randomised test. Make sure the copy carries structure only, never responses or anything private. Handle edge cases such as logic and integrations.
  • Engineer 2: the leaks after the click. 89% of clickers don't sign up, and 48% of sign-ups don't publish. Test letting people edit the copy before a sign-up gate, plus a guided path to first publish. Also build the loop dashboard below.

How we'll know it worked

Test design: randomise at the form level, ~5,000 forms on the new badge and the rest as control, for 4 weeks. That yields ~1,000 sign-ups, enough to pin the publish rate to about ±3 points. Read the result at week 6, once the 30-day window closes.

Primary metric: badge-sourced new publishers per active creator per month. Baseline is 0.10, and the target is ≥0.20 by quarter end (the pilot implies 0.24). That is ~3,600+ badge publishers a month versus ~1,800 today.

Guardrails (full rollout only if all hold): - Click→sign-up ≥10%. - Publish within 30 days ≥45%. - Next-month retention of clone-sourced creators ≥75%. - Pro upgrade rate of the cohort ≥ today's 9%.

Kill or rework: publish rate below 42% or retention below 70% at week 6.

Quarter-end question: is lifetime loop gain ≥1.0? If it lands at 0.6–0.9, we've still roughly doubled a multiplier. We should then say plainly that the loop amplifies other channels rather than compounding, and invest accordingly.

Grades and run details

Decision model 95 · LLM judge 12 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly13%
  • passAddresses the actual decision100%
  • passRespects explicit constraints46%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims52%
  • passProduces the required deliverable94%
  • passCalls out the paid maths90%
  • passPicks the lever with the most yield100%
  • passA closed loop, not a channel100%
  • passThe loop maths holds20%
  • partialProposes tests that could fail14%
Run
Run
#1
API response time
63 s
Submitted
2 Oct 2026

Usable with a quick edit9 of 11 checks passed

The memo is complete and actionable; a PM could add a stop condition and measurement window with a quick edit without redoing any reasoning.

Output

To: Sam Okoro, Head of Growth From: Product Subject: Put both engineers on the respondent-to-creator loop

Recommendation

Assign both engineers to “Make a form like this” next quarter: one to a safe, reliable copy-to-editor experience; the other to attribution, experimentation, and activation measurement. Do not double paid search or build the referral programme. Keep existing template acquisition running, but do not make 200 new pages the engineering priority.

Our primary growth loop

Creators publish forms → respondents encounter Pollen → some become creators and publish → their forms reach more respondents.

The badge closes this loop. Templates and paid search feed creators into it, but are not themselves loops: team-written SEO pages require more team production, and paid acquisition requires more spend.

Using the supplied volumes, last month’s 40,000 forms generated approximately 4.8 million respondent encounters:

SourceNew creators publishingStill active the following month
------:---:
Badge1,8061,535
Templates840596
Paid search441282

The badge is our largest acquisition source by activated creators and our strongest by next-month retention: 85%, versus 71% for templates and 64% for paid.

These badge totals assume the respondent volume is eligible to see the badge. Pro removes it, so we must measure actual badge exposure before treating these as forecasts.

Is it compounding?

It recirculates, but we have not demonstrated self-sustaining compounding.

At current rates, each active creator generates approximately 0.10 new publishing creators per monthly production cycle. Meanwhile, 22% of active creators do not return the next month.

A simplified replacement model therefore gives:

78% returning + 10% newly generated = 88% of the starting base.

The loop alone does not currently replace creator attrition. External acquisition can still grow Pollen, but that is different from autonomous compounding. This calculation is directional: publication, activation, and retention windows are not perfectly aligned, and future cohorts may retain differently.

The pilot is promising. Its click and publication rates imply 2.43 times as many new publishing creators per respondent:

  • Current: 0.376 publishing creators per 1,000 respondents
  • Pilot: 0.915 per 1,000

At unchanged volume, that would mean approximately 4,393 new publishing creators, versus 1,806 today. The same simplified replacement model reaches 78% + 24.4% = 102.4%—potentially crossing the self-sustaining threshold, but narrowly. Badge coverage, cohort maturity, retention, and repeat production could erase that margin.

Why this beats the alternatives

Paid search: Current spend is $79,800, or approximately $181 per publishing creator. Applying our overall upgrade rate and Pro lifetime, expected lifetime subscription revenue per publishing creator is only 9% × $20 × 14 = $25.20, before costs. Channel-specific monetization could differ, but the supplied economics do not justify scaling. Finance approval is not evidence of profitable growth.

More templates: They produce useful acquisition, but weaker activation and retention than the badge. We also have no evidence that 200 additional pages would reproduce existing traffic or conversion.

Referral credits: Paying for sign-ups rewards an intermediate action, invites low-quality referrals, and lacks pilot evidence. Revisit an activation-based incentive later.

Delivery and proof

Engineer 1: Build the contextual editor entry and reliable form-copy flow. Copy only permitted form structure—not respondent submissions, credentials, or integrations—and provide appropriate creator controls. Preserve Pro’s badge-removal benefit.

Engineer 2: Instrument exposure through publishing, next-month activity, downstream form distribution, and Pro conversion; run a persistent randomized test, clustered by source creator.

Our primary success metric is incremental creators publishing within 30 days per 1,000 eligible respondent encounters, not clicks or sign-ups. Target at least 2× control, with a statistically credible lift and no material deterioration in next-month retention.

Track form-completion rate, privacy complaints, Pro conversion, and badge-removal behavior as guardrails. Validate the pilot’s reported 30-day activation on mature cohorts; a two-week pilot alone cannot establish that outcome for every participant.

Finally, follow acquired creators’ subsequent publishing and respondent reach. Declare compounding only when mature production cohorts show that returning creators plus loop-generated creators consistently exceed the starting base without increased external acquisition.

Grades and run details

Decision model 82 · LLM judge 11 of 12 checks
Decision model checks
  • failUses the supplied evidence correctly51%
  • passAddresses the actual decision100%
  • passRespects explicit constraints32%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims31%
  • passProduces the required deliverable75%
  • passCalls out the paid maths97%
  • passPicks the lever with the most yield95%
  • passA closed loop, not a channel98%
  • partialThe loop maths holds8%
  • partialProposes tests that could fail92%
Run
Run
#1
Time to output
48 s
Submitted
2 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyRightRightMixed
GPT-6.1 Sol · API

All claims about the current situation are taken directly from the supplied context or derived by correct arithmetic.

Sonnet 5.5 · API

All factual claims about the current situation are taken directly from the brief or derived by correct arithmetic, with no invented numbers.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly supported by the brief or derived by correct arithmetic.

Proposes tests that could failWrongRightWrong
GPT-6.1 Sol · API

The proposed experiment lacks a numeric threshold for success; it only says 'statistically credible increase' without a specific number, so it does not meet the requirement for a threshold.

Sonnet 5.5 · API

The proposed test has numeric thresholds (publish rate ≥45%, retention ≥75%, etc.), a 4-week measurement window with readout at week 6, and clear kill/rework actions.

GPT-6 Astra · ChatGPT

Proposes a test with a numeric threshold (2× control) but does not specify a measurement window or what action each outcome triggers.

All got right 9

Addresses the actual decisionRightRightRight
GPT-6.1 Sol · API

The memo commits early to putting both engineers on 'Make a form like this', addresses Sam directly, and states that success means a statistically credible increase in retained creators without guardrail deterioration.

Sonnet 5.5 · API

The memo commits early to putting both engineers on idea 4, names the primary loop and its compounding status, and specifies what results would change the call (kill thresholds, quarter-end loop gain).

GPT-6 Astra · ChatGPT

Commits early to putting both engineers on 'Make a form like this', says how we'll know it worked (2× control threshold, guardrails), and states what would change the compounding declaration.

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The output is a memo to Sam, under 800 words, and respects the brief's request for a recommendation and success criteria.

Sonnet 5.5 · API

The output is a memo to Sam Okoro, under 800 words, and respects the requested form and reader.

GPT-6 Astra · ChatGPT

Memo is addressed to Sam, from Product, under 800 words, and covers the required topics.

Identifies material uncertaintyRightRightRight
GPT-6.1 Sol · API

It flags the small pilot, the assumption that retention holds, the need to measure badge-visible encounters, and says to wait for retention windows to mature.

Sonnet 5.5 · API

It names specific unknowns (pilot size, missing retention data, short window, badge coverage) and says how they will be resolved via a test with guardrails and kill criteria.

GPT-6 Astra · ChatGPT

Names badge exposure uncertainty, pilot size limitation, cohort maturity, and retention variability, and says to validate on mature cohorts and measure actual exposure.

Avoids unsupported claimsRightRightRight
GPT-6.1 Sol · API

Forecasts and assumptions are clearly labelled, and confident claims are backed by the supplied evidence.

Sonnet 5.5 · API

Interpretations and forecasts are clearly labelled as such (e.g., 'plausible, not proven', 'I've assumed'), and confident claims are backed by the supplied evidence.

GPT-6 Astra · ChatGPT

Forecasts and interpretations are labelled as directional or potential, and no confident claims go beyond the supplied evidence.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

The memo is complete, in the right form, within the word limit, and a PM could act on it with light edits.

Sonnet 5.5 · API

The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.

GPT-6 Astra · ChatGPT

Delivers a complete, actionable memo within the word limit that Sam could use to assign engineers and measure success.

Calls out the paid mathsRightRightRight
GPT-6.1 Sol · API

It calculates $181 per publishing creator and $25.20 revenue, and says plainly that doubling paid search would be economically unattractive.

Sonnet 5.5 · API

It calculates $181 cost per publishing creator vs $25 revenue, states that doubling paid would lose money, and declines the spend.

GPT-6 Astra · ChatGPT

Computes $181 per publishing creator and $25.20 revenue, and states plainly that doubling paid would lose money.

Picks the lever with the most yieldRightRightRight
GPT-6.1 Sol · API

It chooses the copy-as-template badge, sizes its yield at about 0.11 creators per form (2.43× today), and notes the pilot was small and short.

Sonnet 5.5 · API

It puts engineers on the copy-as-template badge, sizes the yield from the pilot (2.4×, ~0.11 creators per form), and flags the pilot's small size and missing data.

GPT-6 Astra · ChatGPT

Chooses the copy-as-template badge, sizes its yield from the pilot (0.915 per 1,000, ~2.4×), and flags the small two-week pilot before relying on it.

A closed loop, not a channelRightRightRight
GPT-6.1 Sol · API

It names the badge-driven respondent-to-creator loop as the primary loop, grounds it in the highest-retention creators, and computes its yield with retention.

Sonnet 5.5 · API

It names the badge as the primary closed loop where the highest-retention users come from, and works out its yield (0.10/month) with retention applied (0.45 lifetime).

GPT-6 Astra · ChatGPT

Names the badge as the primary closed loop, grounded in highest retention (85%), and sizes its yield with retention applied (0.10 per creator, 88% replacement).

The loop maths holdsRightRightRight
GPT-6.1 Sol · API

The yield (0.10 publishers, 0.085 retained) is correct, retention is applied, and it gives a plain verdict that the loop does not compound.

Sonnet 5.5 · API

All loop yields and coefficients are computed correctly from the pack, retention is included, and each loop gets a plain verdict (not compounding, edge of compounding).

GPT-6 Astra · ChatGPT

Yield (0.10), retention (78%), and replacement (88%) are correctly computed; verdict is 'recirculates but not self-sustaining compounding'.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI87.396.22None
2GPT-6.1 SolwithAPI89.287.82None
3GPT-6 AstrawithChatGPT82.684.32None
4Opus 5.5withClaude87.164.42None
5GPT-6 LunawithAPI73.764.12None
6Gemini 3.8 FlashwithAPI64.676.92None
7Gemini 3.5 Flash-LitewithGemini40.953.82None

About the task

The PM job

Working out what actually drives growth, and where to push.

Why it matters

Teams tune funnel steps while the loop that compounds goes unmeasured. Mistaking a channel for a loop can cost a year.

What good looks like

  • A closed loop: each cycle's output feeds the next
  • The primary loop, traced from where the best users come from
  • The loop sized: cycle time, conversion, amplification
  • Retention in the maths
  • One lever, with a test that could fail

Deliberately not measured

  • Building a full growth model in a spreadsheet
  • Channel-level media planning
Capability tested

Growth systems thinking

The failure we’re looking for

Calls a channel a loop, or a referral button a viral loop

Grading

Decision model and LLM judge, calibrated against a blind PM review