Tasks / Experiment

Find the growth loop

Can the model find a product's real growth loop, show whether it compounds, and say which lever to pull?

Measures the modelTask type v1.0 · 2 tasksLast changed 2 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 33% were usable with at most a quick edit.

Reliably right

  1. Sees the cross-side effect100% pass
    It traces the chain from the tutor bounty to oversupply, thinner bookings, new profiles without reviews, and weaker ranking, and acts on it by pausing broad tutor referrals.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Addresses the actual decision96% pass
    The memo commits early to putting both engineers on idea 4, names the primary loop and its compounding status, and specifies what results would change the call (kill thresholds, quarter-end loop gain).
    Sonnet 5.5 · API · The badge on every form
  3. Produces the required deliverable96% pass
    The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.
    Sonnet 5.5 · API · The badge on every form

Where it slips

  1. The loop maths holds46% pass
    The memo does not give a plain verdict of 'decaying' for the content loop despite showing its decline, and it does not compute a numeric yield or coefficient for that loop.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Uses the supplied evidence correctly57% pass
    The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.
    Opus 5.5 · Claude · The badge on every form
  3. Avoids unsupported claims59% pass
    Presents the 10,700 equilibrium and the rising cost-per-sign-up claim as facts without labelling them as hypotheses or supporting them from the pack.
    Opus 5.5 · Claude · The badge on every form

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pollen. Our Head of Growth, Sam Okoro, has two engineers for next quarter and four ideas for how to use them. Write Sam a memo of no more than 800 words that says what our primary growth loop is, whether it's compounding, and where the two engineers should go, with how we'll know it worked. Everything we know is below.

What the model was given5 items: About Pollen, Creators, Where new creators come from (last month), Revenue, The four ideas on the table
About PollenA free form and survey builder. Every published form shows a small 'Made with Pollen: make your own' badge at the bottom. Creators can upgrade to Pro for $20 a month to remove the badge and unlock logic and integrations.
Creators18,000 creators published at least one form last month, publishing 40,000 forms between them. Each form gets 120 respondents on average. 78% of last month's active creators were active again this month.
Where new creators come from (last month)Badge: 0.9% of respondents clicked the badge and 11% of those signed up; 38% of badge sign-ups published a form within 30 days, and 85% of those were active again the next month. Template pages (written by our team, ranking in search): 6,000 sign-ups, 14% published, 71% active the next month. Paid search: 2,100 sign-ups at $38 per sign-up, 21% published, 64% active the next month.
Revenue9% of creators who publish upgrade to Pro, and Pro customers stay for 14 months on average.
The four ideas on the table1. Double the paid search budget (Finance has approved it). 2. 'Build our SEO loop': 200 more template pages. 3. A referral programme: $10 of Pro credit for each friend who signs up. 4. Replace the badge with 'Make a form like this', which opens the editor with a copy of the form the respondent just filled in. A two-week pilot on 500 forms raised badge clicks from 0.9% to 1.6% of respondents; 11% of them signed up, as before, and 52% of those published within 30 days.
What a strong answer doesThe answer key the graders mark against

Names the badge as the primary loop: creators publish forms, respondents see the badge, some become creators who publish more forms. It's chosen because badge creators publish and stay best (38% publish, 85% retained), not because it brings the most sign-ups. Sizes it: 120 × 0.9% × 11% × 38% ≈ 0.045 new publishing creators per form, about 1,800 a month from 40,000 forms. Puts that against churn: 22% of 18,000 is about 3,960 creators lost a month, while all sources together add about 3,090 (1,806 + 840 + 441), so the base is shrinking and the loop contributes rather than compounds. Shows paid isn't a loop: $38 ÷ 21% ≈ $181 per publishing creator against about $25 of revenue each (9% × $20 × 14), so doubling it burns money. Calls the template pages a channel, not a loop, since our team writes them and creators don't. Gives the referral credit little weight, because it only matters to the 9% who pay. Puts both engineers on 'Make a form like this': the pilot gives 120 × 1.6% × 11% × 52% ≈ 0.11 creators per form, about 2.4× today, which would add roughly 2,600 creators a month if it held. Flags that the pilot was small and short, and proposes a full test with a threshold (for example, publishing creators per 1,000 respondents) and a stop condition. Mentions retention as the other lever.

Critical failures (cap the score at 40)
  • Recommends doubling paid acquisition, or calls it a growth loop
Task

Mid-level · v1.0 · synthetic · B2C, PLG, viral loop, unit economics

The outputs

7 setups answered this brief. Pick up to three to read side by side.

1 of 11 checks passed

Output

MEMORANDUM

TO: Sam Okoro, Head of Growth FROM: Product Manager, Pollen DATE: March 30, 2026 SUBJECT: Growth Analysis & Q3 Engineering Allocation

---

1. Our Primary Growth Loop and Compounding Status

Pollen’s primary growth loop is the Viral Product-Led Growth (PLG) Badge Loop.

Here is how it functions: 1. Creation: A creator builds and publishes a form on Pollen. 2. Distribution: Creators send their forms out, generating an average of 120 respondents per form (40,000 forms $\times$ 120 = 4.8 million monthly respondent impressions). 3. Acquisition: 0.9% of those respondents click the "Made with Pollen" badge, and 11% of those clickers sign up. This currently yields approximately 4,752 sign-ups per month at zero media cost. 4. Activation & Retention: 38% of those sign-ups publish a form within 30 days, and 85% of those active creators retain into the following month, immediately re-feeding the top of the loop.

Is it compounding? Yes, it is structurally a compounding loop, but its current compounding velocity is fragile. Because our monthly creator retention is strong (78% overall, and 85% for badge cohorts), every cohort of users added to the system continually outputs thousands of new forms, generating millions of new respondent impressions month after month.

However, the loop’s current conversion efficiency means it is barely breaking even with natural churn. To accelerate compounding, we need to increase the velocity of respondents turning into active creators. That brings us to our Q2 engineering allocation.

---

2. Where the Two Engineers Should Go

We have four ideas on the table. We should allocate both engineers to Idea 4: Replace the badge with 'Make a form like this' (Cloning Loop).

Here is why we must reject the other three: * Idea 1 (Double Paid Search): Generates 2,100 sign-ups at $38/ea ($79.8k/month additional spend). While profitable over a 14-month Pro lifespan (9% conversion $\times$ $20/mo $\times$ 14 mos = $25.20 LTV vs. $38 CAC—wait, our LTV is actually around $252, making CAC look fine on paper), paid search yields lower activation (21%) and worse retention (64%) than organic loops. It is linear growth, not compounding. * Idea 2 (200 SEO Template Pages): Good, but it relies on static editorial content. It lacks a viral coefficient and has a lower activation rate (14%) and month-2 retention (71%) than the badge loop. * Idea 3 (Referral Programme): A $10 Pro credit is structurally misaligned for a product where 91% of our base is on a free tier. Free users cannot easily utilize Pro credits, rendering this incentive largely inert for the vast majority of our creators.

Why Idea 4 is the clear winner: The pilot data for "Make a form like this" is exceptional. By reducing friction and showing respondents a working replica of the exact form they just successfully interacted with, we saw: * Badge clicks nearly doubled: From 0.9% to 1.6%. * Sign-up rate held steady: 11% conversion. * Activation surged: 52% of sign-ups published within 30 days (up from 38% on the old badge).

This isn't just an incremental tweak; it transforms our passive watermark into an active, high-intent replication engine. It directly attacks the weakest link in our primary loop (activation) while supercharging top-of-funnel discovery.

---

3. How We Will Know It Worked (Success Metrics)

We will roll this out to 100% of published forms on day one of the quarter. We will measure success across three core KPIs over a 60-day post-launch window:

  1. Primary Metric (Loop Velocity): Monthly Free-to-Published Creator conversion from badge traffic. Target: A sustained lift in 30-day activation from the baseline 38% toward the pilot’s 52% benchmark.
  2. Secondary Metric (Top-of-Funnel Conversion): Respondent-to-sign-up volume. Target: Maintain or exceed the pilot's 1.6% click-through rate across our full 4.8M monthly respondent pool, yielding ~8,400 sign-ups/month from the badge alone (nearly double current output).
  3. Downstream Revenue Impact: Monthly Recurring Revenue (MRR) added via Pro upgrades from this specific acquisition channel. Target: Since 9% of publishing creators upgrade to Pro ($20/mo), the uplift in activated creators from this loop must translate directly to an accelerated net-new Pro ARR growth rate compared to Q1.

Put both engineers on Idea 4. Let's compound the loop.

Grades and run details

Decision model 32
Decision model checks
  • failUses the supplied evidence correctly80%
  • partialAddresses the actual decision36%
  • failRespects explicit constraints81%
  • failIdentifies material uncertainty44%
  • failAvoids unsupported claims95%
  • partialProduces the required deliverable70%
  • partialCalls out the paid maths54%
  • partialPicks the lever with the most yield99%
  • passA closed loop, not a channel25%
  • failThe loop maths holds95%
  • partialProposes tests that could fail53%
Run
Run
#1
Time to output
6 s
Submitted
2 Oct 2026

Needs more than a quick edit6 of 11 checks passed

The test lacks a numeric threshold and measurement window, so a PM would need to add those before acting. The paid math is per signup rather than per publishing creator, but still shows the loss.

Output

To: Sam Okoro, Head of Growth Subject: Next-quarter engineering allocation

Recommendation: Put both engineers on the respondent-to-creator loop: ship and rigorously measure “Make a form like this.” Don’t double paid search or build the referral programme this quarter. Keep producing SEO templates through the content team; the available evidence doesn’t show that engineering is the constraint there.

Our primary loop is: a respondent sees a Pollen badge → visits or copies the form → signs up → publishes a form → brings in more respondents. It’s our clearest product-led acquisition loop because creators’ forms distribute the invitation to make a form.

It is not currently compounding. An average active creator publishes 2.22 forms (40,000 / 18,000), producing about 267 respondent exposures per month. Applying the current badge funnel gives roughly 0.085 next-month-active creators per active creator (267 × 0.9% × 11% × 38% × 85%). That’s well below the 1.0 needed for the loop to reproduce itself.

The pilot is promising: using its 1.6% click rate and 52% publish rate, the same calculation would imply about 0.21 next-month-active creators per creator—roughly 2.4× the current rate, if those new publishers retain at the existing 85% rate. That retention is an assumption, not a pilot result. Even so, the loop would still be subcritical: it could help offset churn, but it would not independently drive exponential growth.

Engineering plan - Engineer 1: Build the copy-to-editor flow, preserving the original form’s structure and making it easy to publish a customized copy. - Engineer 2: Instrument the full funnel and run a randomized rollout against the current badge. Track respondent exposure, click, signup, publish within 30 days, and next-month activity by variant.

How we’ll know it worked: Make next-month-active creators per 1,000 respondent exposures the primary outcome, with publishing and next-month activity as required funnel checks. Compare treatment with control, and don’t assume the pilot’s 52% publish rate will hold at scale. Expand only if the lift is sustained and new creators’ next-month activity doesn’t materially underperform the current badge cohort.

Why not the other ideas: Paid search costs $38 per signup. At its observed 21% publish rate, 9% upgrade rate, and $280 average Pro revenue lifetime, that implies about $5.29 in lifetime revenue per signup before costs—poor grounds for doubling spend. SEO already brings 6,000 signups, but only 14% publish; test additional pages with the content team before assigning engineering. The referral offer has no conversion or retention evidence, and paying credit per signup risks rewarding low-intent users.

Grades and run details

Decision model 68 · LLM judge 8 of 12 checks
Decision model checks
  • failUses the supplied evidence correctly53%
  • passAddresses the actual decision99%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty81%
  • partialAvoids unsupported claims25%
  • passProduces the required deliverable57%
  • partialCalls out the paid maths86%
  • passPicks the lever with the most yield88%
  • passA closed loop, not a channel94%
  • failThe loop maths holds18%
  • partialProposes tests that could fail79%
Run
Run
#1
API response time
38 s
Submitted
2 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The memo contains an unsupported equilibrium number (10,700) and an unsupported claim about rising cost per sign-up, so a PM would need to re-check those parts before using it.

Output

To: Sam Okoro, Head of Growth From: [PM], Growth Re: Our growth loop, and where the two engineers should go next quarter

Summary: Our primary loop is the badge. Today it does not compound, and the active creator base is shrinking. The "Make a form like this" pilot is the only one of the four ideas that could make it compound. Both engineers should ship it properly, with a holdout. We should not double paid search.

1. The loop

Creators publish forms → respondents see the badge → some respondents become creators → they publish forms.

Last month the loop worked like this: - 40,000 forms × 120 respondents = 4.8M respondents - 0.9% clicked the badge = 43,200 clicks - 11% of clicks signed up = 4,752 sign-ups - 38% of sign-ups published = 1,806 new publishing creators

The badge is our largest source of creators. Templates produced 840 publishers and paid search 441. Badge creators also retain best: 85% are active the next month, against 71% for templates and 64% for paid.

2. Is it compounding? Not yet.

Each active creator produces about 267 respondents a month, which yields 0.10 new publishers per creator per month. With 78% monthly retention, a creator stays about 4.5 months. Over that lifetime, each creator brings in about 0.46 new creators.

A loop compounds only when this number exceeds 1. At 0.46, the badge amplifies our other channels by about 1.8×, but it cannot sustain itself.

The overall picture is worse: - We keep 14,040 of last month's 18,000 active creators. - We add about 3,100 new publishers. - That gives about 17,100 active creators, a 5% monthly decline. - At current rates the base settles at roughly 10,700 creators.

This assumes the 78% retention figure covers the whole base, which is how I read the data.

3. The four ideas

Double paid search: no. - Each sign-up costs $38. Only 21% publish, so each publishing creator costs about $181. - Each publisher is worth about $25 in revenue (9% upgrade × $20 × 14 months). - Adding the loop's downstream creators raises that to only about $46. - Doubling spend would add about 440 publishers a month for an extra $80k a month, and cost per sign-up usually rises with spend. - This idea doesn't need engineers. I'd ask Finance to hold the budget rather than spend it because it's approved.

200 more template pages: not now. - This is not a loop, because new creators don't create more pages. Our team writes them, so output grows only as fast as we write. - We don't know how many pages produce today's 6,000 sign-ups, so we can't estimate the yield. - Templates publish at only 14%. - This is mainly content work, not engineering work.

Referral credit: no. - It rewards sign-ups, not publishing, which invites low-quality and fake sign-ups. - $10 of Pro credit means nothing to the 91% of creators who never upgrade.

"Make a form like this": yes. The pilot changed every stage of the loop:

Current badgePilot
Click rate0.9%1.6%
Sign-up rate11%11%
Publish rate38%52%
New publishers per 1,000 respondents0.380.92

At network scale, that would mean about 4,400 badge publishers a month instead of 1,800. Each creator's lifetime yield would rise from 0.46 to about 1.1, so the loop would start to compound. Next month's active base would be about 19,700 instead of 17,100.

The margin is thin. The loop reaches the break-even of 1.0 at 0.83 publishers per 1,000 respondents. The pilot beat that by only about 10%. We also need to treat the pilot result with caution: - It covered 500 forms, which may not be representative. - It ran for two weeks, so check whether the 30-day publish window had fully elapsed. - We have no next-month retention data for these creators. People who copy a form may be less committed than people who build one.

4. What the engineers build

1. Staged rollout with a 20% holdout of forms on the old badge, randomised by form and run for at least 8 weeks so we see 30-day publishing and next-month retention. 2. Creator controls and safety: - Copies take the form structure only, never responses. - Creators can turn off copying for sensitive forms. 3. Instrumentation for each stage: click, sign-up, publish, next-month activity, and Pro upgrade. Results should be split by form type so we learn which forms spread best.

5. How we'll know it worked

Primary metric: new publishing creators per 1,000 respondents, treatment vs holdout. - Ship to 100% if it stays at or above 0.83, the level needed for the loop to compound. - Below about 0.6, the change is still positive but the loop won't compound. We would keep it and look for other levers.

Quality checks: - Next-month retention of copy sign-ups should be at least 70%. If it falls much lower, the extra volume won't last. - The upgrade rate of these creators to Pro should be close to 9%.

Guardrails: - Form completion rate is unchanged. - Creator complaints and copy opt-outs are low. - Pro upgrade rate across the base is unchanged.

Business outcome: the active creator base grows month on month by the end of the quarter, instead of falling about 5%.

Bottom line: paid search and templates rent growth one sign-up at a time. The copy-this-form change is our one realistic chance to make growth compound, and a proper rollout will tell us within a quarter whether it does.

Claims the judge couldn’t find in the brief

  • At current rates the base settles at roughly 10,700 creators.
  • Cost per sign-up usually rises with spend.

Grades and run details

Decision model 91 · LLM judge 9 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly27%
  • passAddresses the actual decision100%
  • passRespects explicit constraints33%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims44%
  • passProduces the required deliverable88%
  • passCalls out the paid maths99%
  • passPicks the lever with the most yield98%
  • passA closed loop, not a channel100%
  • passThe loop maths holds11%
  • partialProposes tests that could fail52%
Run
Run
#1
Time to output
55 s
Submitted
2 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 10

Uses the supplied evidence correctlyWrongMixedMixed
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

All factual claims about the current situation are directly from the supplied context or correct arithmetic.

Opus 5.5 · Claude

The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.

Addresses the actual decisionMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Commits early to putting both engineers on 'Make a form like this', framed for Sam, and says to expand only if lift is sustained and next-month activity doesn't underperform.

Opus 5.5 · Claude

Commits early to putting both engineers on 'Make a form like this' with a holdout, says not to double paid search, and specifies what metric would change the call.

Respects explicit constraintsWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Memo format, addressed to Sam, under 800 words, and respects the brief's request.

Opus 5.5 · Claude

Memo is addressed to Sam Okoro, stays within ~800 words, and proposes enforcement mechanisms (holdout, controls).

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Identifies that retention from the pilot is an assumption, that the 52% publish rate may not hold at scale, and proposes a randomized rollout to resolve.

Opus 5.5 · Claude

Flags pilot's small size, two-week window, missing retention data, and representativeness, and resolves them with an 8-week holdout test and thresholds.

Avoids unsupported claimsWrongRightWrong
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Assumptions are labelled (retention assumption, pilot result not guaranteed), and judgments are grounded in arithmetic.

Opus 5.5 · Claude

Presents the 10,700 equilibrium and the rising cost-per-sign-up claim as facts without labelling them as hypotheses or supporting them from the pack.

Produces the required deliverableMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Complete memo with recommendation, rationale, and measurement plan, usable as is.

Opus 5.5 · Claude

Delivers a complete memo to Sam with a clear recommendation, how to measure success, and is within the word limit.

Calls out the paid mathsMixedWrongRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Computes revenue per signup ($5.29) rather than cost per publishing creator ($181) and revenue per creator ($25), missing the explicit unit economics the criterion requires.

Opus 5.5 · Claude

Calculates $181 cost per publishing creator and $25 revenue, states doubling would lose money, and recommends against it.

Picks the lever with the most yieldMixedMixedRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Chooses the copy-as-template badge and sizes its yield (0.21 per creator, 2.4×), but does not flag the pilot's small size (500 forms, two weeks) explicitly.

Opus 5.5 · Claude

Chooses the copy-as-template badge, sizes its yield from the pilot (0.92 per 1,000 respondents, ~4,400 publishers, ~2.4× current), and notes the pilot's small size.

The loop maths holdsWrongMixedRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Correctly computes current coefficient (0.085) and pilot coefficient (0.21), applies retention, and gives plain verdicts (not compounding, subcritical).

Opus 5.5 · Claude

Correctly computes yield per creator (0.10/month, 0.46 lifetime), retention (4.5 months), amplification (1.8×), and break-even (0.83), and says the loop does not compound.

Proposes tests that could failMixedWrongRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Proposes a randomized rollout but gives no numeric threshold for 'sustained lift' or 'materially underperform', and no measurement window.

Opus 5.5 · Claude

Proposes a holdout test with a numeric threshold (0.83 to ship, below 0.6 to keep but seek other levers), an 8-week window, and quality checks with thresholds.

All got right 1

A closed loop, not a channelRightRightRight
Gemini 3.5 Flash-Lite · Gemini

No reason given.

GPT-6 Luna · API

Names the badge as the primary closed loop, uses the highest-retention cohort's numbers (38% publish, 85% retention), and computes its coefficient with retention.

Opus 5.5 · Claude

Names the badge as the primary closed loop, grounds it in highest-retention creators, and sizes its yield with retention applied.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI87.396.22None
2GPT-6.1 SolwithAPI89.287.82None
3GPT-6 AstrawithChatGPT82.684.32None
4Opus 5.5withClaude87.164.42None
5GPT-6 LunawithAPI73.764.12None
6Gemini 3.8 FlashwithAPI64.676.92None
7Gemini 3.5 Flash-LitewithGemini40.953.82None

About the task

The PM job

Working out what actually drives growth, and where to push.

Why it matters

Teams tune funnel steps while the loop that compounds goes unmeasured. Mistaking a channel for a loop can cost a year.

What good looks like

  • A closed loop: each cycle's output feeds the next
  • The primary loop, traced from where the best users come from
  • The loop sized: cycle time, conversion, amplification
  • Retention in the maths
  • One lever, with a test that could fail

Deliberately not measured

  • Building a full growth model in a spreadsheet
  • Channel-level media planning
Capability tested

Growth systems thinking

The failure we’re looking for

Calls a channel a loop, or a referral button a viral loop

Grading

Decision model and LLM judge, calibrated against a blind PM review