Tasks / Experiment

Find the growth loop

Can the model find a product's real growth loop, show whether it compounds, and say which lever to pull?

Measures the modelTask type v1.0 · 2 tasksLast changed 2 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 33% were usable with at most a quick edit.

Reliably right

  1. Sees the cross-side effect100% pass
    It traces the chain from the tutor bounty to oversupply, thinner bookings, new profiles without reviews, and weaker ranking, and acts on it by pausing broad tutor referrals.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Addresses the actual decision96% pass
    The memo commits early to putting both engineers on idea 4, names the primary loop and its compounding status, and specifies what results would change the call (kill thresholds, quarter-end loop gain).
    Sonnet 5.5 · API · The badge on every form
  3. Produces the required deliverable96% pass
    The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.
    Sonnet 5.5 · API · The badge on every form

Where it slips

  1. The loop maths holds46% pass
    The memo does not give a plain verdict of 'decaying' for the content loop despite showing its decline, and it does not compute a numeric yield or coefficient for that loop.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Uses the supplied evidence correctly57% pass
    The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.
    Opus 5.5 · Claude · The badge on every form
  3. Avoids unsupported claims59% pass
    Presents the 10,700 equilibrium and the rising cost-per-sign-up claim as facts without labelling them as hypotheses or supporting them from the pack.
    Opus 5.5 · Claude · The badge on every form

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pollen. Our Head of Growth, Sam Okoro, has two engineers for next quarter and four ideas for how to use them. Write Sam a memo of no more than 800 words that says what our primary growth loop is, whether it's compounding, and where the two engineers should go, with how we'll know it worked. Everything we know is below.

What the model was given5 items: About Pollen, Creators, Where new creators come from (last month), Revenue, The four ideas on the table
About PollenA free form and survey builder. Every published form shows a small 'Made with Pollen: make your own' badge at the bottom. Creators can upgrade to Pro for $20 a month to remove the badge and unlock logic and integrations.
Creators18,000 creators published at least one form last month, publishing 40,000 forms between them. Each form gets 120 respondents on average. 78% of last month's active creators were active again this month.
Where new creators come from (last month)Badge: 0.9% of respondents clicked the badge and 11% of those signed up; 38% of badge sign-ups published a form within 30 days, and 85% of those were active again the next month. Template pages (written by our team, ranking in search): 6,000 sign-ups, 14% published, 71% active the next month. Paid search: 2,100 sign-ups at $38 per sign-up, 21% published, 64% active the next month.
Revenue9% of creators who publish upgrade to Pro, and Pro customers stay for 14 months on average.
The four ideas on the table1. Double the paid search budget (Finance has approved it). 2. 'Build our SEO loop': 200 more template pages. 3. A referral programme: $10 of Pro credit for each friend who signs up. 4. Replace the badge with 'Make a form like this', which opens the editor with a copy of the form the respondent just filled in. A two-week pilot on 500 forms raised badge clicks from 0.9% to 1.6% of respondents; 11% of them signed up, as before, and 52% of those published within 30 days.
What a strong answer doesThe answer key the graders mark against

Names the badge as the primary loop: creators publish forms, respondents see the badge, some become creators who publish more forms. It's chosen because badge creators publish and stay best (38% publish, 85% retained), not because it brings the most sign-ups. Sizes it: 120 × 0.9% × 11% × 38% ≈ 0.045 new publishing creators per form, about 1,800 a month from 40,000 forms. Puts that against churn: 22% of 18,000 is about 3,960 creators lost a month, while all sources together add about 3,090 (1,806 + 840 + 441), so the base is shrinking and the loop contributes rather than compounds. Shows paid isn't a loop: $38 ÷ 21% ≈ $181 per publishing creator against about $25 of revenue each (9% × $20 × 14), so doubling it burns money. Calls the template pages a channel, not a loop, since our team writes them and creators don't. Gives the referral credit little weight, because it only matters to the 9% who pay. Puts both engineers on 'Make a form like this': the pilot gives 120 × 1.6% × 11% × 52% ≈ 0.11 creators per form, about 2.4× today, which would add roughly 2,600 creators a month if it held. Flags that the pilot was small and short, and proposes a full test with a threshold (for example, publishing creators per 1,000 respondents) and a stop condition. Mentions retention as the other lever.

Critical failures (cap the score at 40)
  • Recommends doubling paid acquisition, or calls it a growth loop
Task

Mid-level · v1.0 · synthetic · B2C, PLG, viral loop, unit economics

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Usable with a quick edit9 of 11 checks passed

The memo is complete and actionable; a PM could add a stop condition and measurement window with a quick edit without redoing any reasoning.

Output

To: Sam Okoro, Head of Growth From: Product Subject: Put both engineers on the respondent-to-creator loop

Recommendation

Assign both engineers to “Make a form like this” next quarter: one to a safe, reliable copy-to-editor experience; the other to attribution, experimentation, and activation measurement. Do not double paid search or build the referral programme. Keep existing template acquisition running, but do not make 200 new pages the engineering priority.

Our primary growth loop

Creators publish forms → respondents encounter Pollen → some become creators and publish → their forms reach more respondents.

The badge closes this loop. Templates and paid search feed creators into it, but are not themselves loops: team-written SEO pages require more team production, and paid acquisition requires more spend.

Using the supplied volumes, last month’s 40,000 forms generated approximately 4.8 million respondent encounters:

SourceNew creators publishingStill active the following month
------:---:
Badge1,8061,535
Templates840596
Paid search441282

The badge is our largest acquisition source by activated creators and our strongest by next-month retention: 85%, versus 71% for templates and 64% for paid.

These badge totals assume the respondent volume is eligible to see the badge. Pro removes it, so we must measure actual badge exposure before treating these as forecasts.

Is it compounding?

It recirculates, but we have not demonstrated self-sustaining compounding.

At current rates, each active creator generates approximately 0.10 new publishing creators per monthly production cycle. Meanwhile, 22% of active creators do not return the next month.

A simplified replacement model therefore gives:

78% returning + 10% newly generated = 88% of the starting base.

The loop alone does not currently replace creator attrition. External acquisition can still grow Pollen, but that is different from autonomous compounding. This calculation is directional: publication, activation, and retention windows are not perfectly aligned, and future cohorts may retain differently.

The pilot is promising. Its click and publication rates imply 2.43 times as many new publishing creators per respondent:

  • Current: 0.376 publishing creators per 1,000 respondents
  • Pilot: 0.915 per 1,000

At unchanged volume, that would mean approximately 4,393 new publishing creators, versus 1,806 today. The same simplified replacement model reaches 78% + 24.4% = 102.4%—potentially crossing the self-sustaining threshold, but narrowly. Badge coverage, cohort maturity, retention, and repeat production could erase that margin.

Why this beats the alternatives

Paid search: Current spend is $79,800, or approximately $181 per publishing creator. Applying our overall upgrade rate and Pro lifetime, expected lifetime subscription revenue per publishing creator is only 9% × $20 × 14 = $25.20, before costs. Channel-specific monetization could differ, but the supplied economics do not justify scaling. Finance approval is not evidence of profitable growth.

More templates: They produce useful acquisition, but weaker activation and retention than the badge. We also have no evidence that 200 additional pages would reproduce existing traffic or conversion.

Referral credits: Paying for sign-ups rewards an intermediate action, invites low-quality referrals, and lacks pilot evidence. Revisit an activation-based incentive later.

Delivery and proof

Engineer 1: Build the contextual editor entry and reliable form-copy flow. Copy only permitted form structure—not respondent submissions, credentials, or integrations—and provide appropriate creator controls. Preserve Pro’s badge-removal benefit.

Engineer 2: Instrument exposure through publishing, next-month activity, downstream form distribution, and Pro conversion; run a persistent randomized test, clustered by source creator.

Our primary success metric is incremental creators publishing within 30 days per 1,000 eligible respondent encounters, not clicks or sign-ups. Target at least 2× control, with a statistically credible lift and no material deterioration in next-month retention.

Track form-completion rate, privacy complaints, Pro conversion, and badge-removal behavior as guardrails. Validate the pilot’s reported 30-day activation on mature cohorts; a two-week pilot alone cannot establish that outcome for every participant.

Finally, follow acquired creators’ subsequent publishing and respondent reach. Declare compounding only when mature production cohorts show that returning creators plus loop-generated creators consistently exceed the starting base without increased external acquisition.

Grades and run details

Decision model 82 · LLM judge 11 of 12 checks
Decision model checks
  • failUses the supplied evidence correctly51%
  • passAddresses the actual decision100%
  • passRespects explicit constraints32%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims31%
  • passProduces the required deliverable75%
  • passCalls out the paid maths97%
  • passPicks the lever with the most yield95%
  • passA closed loop, not a channel98%
  • partialThe loop maths holds8%
  • partialProposes tests that could fail92%
Run
Run
#1
Time to output
48 s
Submitted
2 Oct 2026

Needs more than a quick edit6 of 11 checks passed

The test lacks a numeric threshold and measurement window, so a PM would need to add those before acting. The paid math is per signup rather than per publishing creator, but still shows the loss.

Output

To: Sam Okoro, Head of Growth Subject: Next-quarter engineering allocation

Recommendation: Put both engineers on the respondent-to-creator loop: ship and rigorously measure “Make a form like this.” Don’t double paid search or build the referral programme this quarter. Keep producing SEO templates through the content team; the available evidence doesn’t show that engineering is the constraint there.

Our primary loop is: a respondent sees a Pollen badge → visits or copies the form → signs up → publishes a form → brings in more respondents. It’s our clearest product-led acquisition loop because creators’ forms distribute the invitation to make a form.

It is not currently compounding. An average active creator publishes 2.22 forms (40,000 / 18,000), producing about 267 respondent exposures per month. Applying the current badge funnel gives roughly 0.085 next-month-active creators per active creator (267 × 0.9% × 11% × 38% × 85%). That’s well below the 1.0 needed for the loop to reproduce itself.

The pilot is promising: using its 1.6% click rate and 52% publish rate, the same calculation would imply about 0.21 next-month-active creators per creator—roughly 2.4× the current rate, if those new publishers retain at the existing 85% rate. That retention is an assumption, not a pilot result. Even so, the loop would still be subcritical: it could help offset churn, but it would not independently drive exponential growth.

Engineering plan - Engineer 1: Build the copy-to-editor flow, preserving the original form’s structure and making it easy to publish a customized copy. - Engineer 2: Instrument the full funnel and run a randomized rollout against the current badge. Track respondent exposure, click, signup, publish within 30 days, and next-month activity by variant.

How we’ll know it worked: Make next-month-active creators per 1,000 respondent exposures the primary outcome, with publishing and next-month activity as required funnel checks. Compare treatment with control, and don’t assume the pilot’s 52% publish rate will hold at scale. Expand only if the lift is sustained and new creators’ next-month activity doesn’t materially underperform the current badge cohort.

Why not the other ideas: Paid search costs $38 per signup. At its observed 21% publish rate, 9% upgrade rate, and $280 average Pro revenue lifetime, that implies about $5.29 in lifetime revenue per signup before costs—poor grounds for doubling spend. SEO already brings 6,000 signups, but only 14% publish; test additional pages with the content team before assigning engineering. The referral offer has no conversion or retention evidence, and paying credit per signup risks rewarding low-intent users.

Grades and run details

Decision model 68 · LLM judge 8 of 12 checks
Decision model checks
  • failUses the supplied evidence correctly53%
  • passAddresses the actual decision99%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty81%
  • partialAvoids unsupported claims25%
  • passProduces the required deliverable57%
  • partialCalls out the paid maths86%
  • passPicks the lever with the most yield88%
  • passA closed loop, not a channel94%
  • failThe loop maths holds18%
  • partialProposes tests that could fail79%
Run
Run
#1
API response time
38 s
Submitted
2 Oct 2026

3 of 11 checks passed

Output

MEMO

To: Sam Okoro, Head of Growth From: Product Management Date: Current Subject: Engineering Allocation for Next Quarter: Doubling Down on Our Primary Growth Loop

---

Executive Summary

Our primary growth engine is our product-led viral loop (forms $\to$ respondents $\to$ badge clicks $\to$ creators $\to$ new forms). While powerful, it is currently amplifying, not self-compounding ($K < 1.0$).

Both engineers should be dedicated to Idea 4 (“Make a form like this”). The pilot proved a 143% lift in publishing creators from our highest-volume, highest-retention channel. Conversely, Paid Search is deeply unprofitable, SEO templates are editorial rather than engineering work, and referrals solve an acquisition problem we can address organically for free.

---

1. Our Primary Growth Loop & Why It’s Not Yet Compounding

Our primary growth loop is driven by the form badge:

$$\text{Active Creators} \longrightarrow \text{Published Forms} \longrightarrow \text{Respondents} \longrightarrow \text{Badge Clicks} \longrightarrow \text{Sign-ups} \longrightarrow \text{Publishing Creators}$$

Current Monthly Metrics (Badge Loop): * 4.8M respondents ($40\text{k forms} \times 120\text{ respondents}$) * 43,200 clicks ($0.9\%$) $\to$ 4,752 sign-ups ($11\%$) $\to$ 1,806 new publishing creators ($38\%$) * Retention: Badge creators have our highest month-two retention (85%, vs. 71% for SEO and 64% for Paid).

Is it compounding? No—not in the runaway sense. * Each active creator publishes $\approx 2.22$ forms, reaching 267 respondents. * At baseline conversion rates, one publishing creator generates only 0.10 new publishing creators per cycle ($267 \times 0.009 \times 0.11 \times 0.38$). Even accounting for creator lifetime retention, the viral factor ($K$) remains well below $1.0$. * The loop is an amplifier of other top-of-funnel acquisition, but it cannot sustain compounding growth on its own without steady top-of-funnel inputs. To move closer to a true compounding state, we must dramatically increase loop velocity and cycle conversion.

---

2. Evaluation of the Four Ideas

IdeaStrategic FitUnit Economics / ImpactRecommendation
:---:---:---:---
1. Double Paid SearchPoorSeverely negative LTV:CAC. Pro LTV is \$280 ($20 \times 14\text{ mo}$). Value per sign-up is \$5.29 ($21\% \text{ publish} \times 9\% \text{ upgrade} \times \$280$). We pay \$38.00 per sign-up, losing \$32.71 on every acquisition.Kill immediately. No eng needed.
2. 200 Template PagesModerate14% publish rate; content-writing task, not an engineering initiative. Programmatic SEO would require eng, but manual templates do not.Pass to Content/Marketing.
3. Referral ProgramLowHigh friction; asks creators to solicit peers for \$10 credit when 4.8M natural form-users already experience the product directly each month.Deprioritize.
4. "Make a form like this"ExceptionalIncreases badge CTR ($0.9\% \to 1.6\%$) and publish rate ($38\% \to 52\%$). Generates 2,587 incremental publishing creators/month.Full Eng Commitment.

---

3. Engineering Allocation: 100% to Idea 4

We should allocate both engineers to productionize and optimize “Make a form like this.”

This is an architectural and UX challenge, not a copy change: 1. Sanitized Form Duplication: Automatically clone questions, logic, and layout while stripping respondent data, webhooks, and private settings. 2. Instant Preview & Onboarding Editor: Drop users directly into a pre-populated editor session before forcing account creation to preserve the pilot's 52% activation rate.

Expected Impact: * Monthly badge publishing creators will increase from 1,806 to 4,393 (+143%). * Revenue impact: 2,587 incremental publishers $\times 9\%$ Pro conversion $\times \$280\text{ LTV} =$ \$65,190 in incremental LTV added per month ($\approx \$780\text{k annualized}$), with zero marginal ad spend. * It raises our loop factor ($K$) from $\approx 0.10$ to $\approx 0.24$ per cycle, dramatically accelerating organic compounding.

---

4. How We'll Know It Worked

We will run a 50/50 randomized split test across all active forms.

#### Primary Success Metric * New 30-Day Publishing Creators generated per 100,000 respondents (Baseline: 37.6 $\to$ Target: $\ge 91.5$).

#### Guardrail & Funnel Metrics * Badge Click-Through Rate: Maintain pilot performance at $\ge 1.6\%$. * Sign-up Conversion: Maintain baseline at $\ge 11\%$. * 30-Day Publish Rate: Maintain pilot performance at $\ge 52\%$. * Form Quality Guardrail: Ensure month-two retention for cloned-form creators matches or exceeds the 85% badge baseline. * Pro Conversion Rate: Confirm downstream Pro upgrade rate stays constant at $\approx 9\%$.

Grades and run details

Decision model 50
Decision model checks
  • failUses the supplied evidence correctly57%
  • passAddresses the actual decision87%
  • partialRespects explicit constraints22%
  • partialIdentifies material uncertainty39%
  • failAvoids unsupported claims35%
  • passProduces the required deliverable58%
  • partialCalls out the paid maths68%
  • partialPicks the lever with the most yield82%
  • passA closed loop, not a channel85%
  • failThe loop maths holds34%
  • partialProposes tests that could fail87%
Run
Run
#1
API response time
32 s
Submitted
2 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyMixedMixedWrong
GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly supported by the brief or derived by correct arithmetic.

GPT-6 Luna · API

All factual claims about the current situation are directly from the supplied context or correct arithmetic.

Gemini 3.8 Flash · API

No reason given.

Respects explicit constraintsRightRightMixed
GPT-6 Astra · ChatGPT

Memo is addressed to Sam, from Product, under 800 words, and covers the required topics.

GPT-6 Luna · API

Memo format, addressed to Sam, under 800 words, and respects the brief's request.

Gemini 3.8 Flash · API

No reason given.

Identifies material uncertaintyRightRightMixed
GPT-6 Astra · ChatGPT

Names badge exposure uncertainty, pilot size limitation, cohort maturity, and retention variability, and says to validate on mature cohorts and measure actual exposure.

GPT-6 Luna · API

Identifies that retention from the pilot is an assumption, that the 52% publish rate may not hold at scale, and proposes a randomized rollout to resolve.

Gemini 3.8 Flash · API

No reason given.

Avoids unsupported claimsRightRightWrong
GPT-6 Astra · ChatGPT

Forecasts and interpretations are labelled as directional or potential, and no confident claims go beyond the supplied evidence.

GPT-6 Luna · API

Assumptions are labelled (retention assumption, pilot result not guaranteed), and judgments are grounded in arithmetic.

Gemini 3.8 Flash · API

No reason given.

Calls out the paid mathsRightWrongMixed
GPT-6 Astra · ChatGPT

Computes $181 per publishing creator and $25.20 revenue, and states plainly that doubling paid would lose money.

GPT-6 Luna · API

Computes revenue per signup ($5.29) rather than cost per publishing creator ($181) and revenue per creator ($25), missing the explicit unit economics the criterion requires.

Gemini 3.8 Flash · API

No reason given.

Picks the lever with the most yieldRightMixedMixed
GPT-6 Astra · ChatGPT

Chooses the copy-as-template badge, sizes its yield from the pilot (0.915 per 1,000, ~2.4×), and flags the small two-week pilot before relying on it.

GPT-6 Luna · API

Chooses the copy-as-template badge and sizes its yield (0.21 per creator, 2.4×), but does not flag the pilot's small size (500 forms, two weeks) explicitly.

Gemini 3.8 Flash · API

No reason given.

The loop maths holdsRightMixedWrong
GPT-6 Astra · ChatGPT

Yield (0.10), retention (78%), and replacement (88%) are correctly computed; verdict is 'recirculates but not self-sustaining compounding'.

GPT-6 Luna · API

Correctly computes current coefficient (0.085) and pilot coefficient (0.21), applies retention, and gives plain verdicts (not compounding, subcritical).

Gemini 3.8 Flash · API

No reason given.

Proposes tests that could failWrongWrongMixed
GPT-6 Astra · ChatGPT

Proposes a test with a numeric threshold (2× control) but does not specify a measurement window or what action each outcome triggers.

GPT-6 Luna · API

Proposes a randomized rollout but gives no numeric threshold for 'sustained lift' or 'materially underperform', and no measurement window.

Gemini 3.8 Flash · API

No reason given.

All got right 3

Addresses the actual decisionRightRightRight
GPT-6 Astra · ChatGPT

Commits early to putting both engineers on 'Make a form like this', says how we'll know it worked (2× control threshold, guardrails), and states what would change the compounding declaration.

GPT-6 Luna · API

Commits early to putting both engineers on 'Make a form like this', framed for Sam, and says to expand only if lift is sustained and next-month activity doesn't underperform.

Gemini 3.8 Flash · API

No reason given.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

Delivers a complete, actionable memo within the word limit that Sam could use to assign engineers and measure success.

GPT-6 Luna · API

Complete memo with recommendation, rationale, and measurement plan, usable as is.

Gemini 3.8 Flash · API

No reason given.

A closed loop, not a channelRightRightRight
GPT-6 Astra · ChatGPT

Names the badge as the primary closed loop, grounded in highest retention (85%), and sizes its yield with retention applied (0.10 per creator, 88% replacement).

GPT-6 Luna · API

Names the badge as the primary closed loop, uses the highest-retention cohort's numbers (38% publish, 85% retention), and computes its coefficient with retention.

Gemini 3.8 Flash · API

No reason given.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI87.396.22None
2GPT-6.1 SolwithAPI89.287.82None
3GPT-6 AstrawithChatGPT82.684.32None
4Opus 5.5withClaude87.164.42None
5GPT-6 LunawithAPI73.764.12None
6Gemini 3.8 FlashwithAPI64.676.92None
7Gemini 3.5 Flash-LitewithGemini40.953.82None

About the task

The PM job

Working out what actually drives growth, and where to push.

Why it matters

Teams tune funnel steps while the loop that compounds goes unmeasured. Mistaking a channel for a loop can cost a year.

What good looks like

  • A closed loop: each cycle's output feeds the next
  • The primary loop, traced from where the best users come from
  • The loop sized: cycle time, conversion, amplification
  • Retention in the maths
  • One lever, with a test that could fail

Deliberately not measured

  • Building a full growth model in a spreadsheet
  • Channel-level media planning
Capability tested

Growth systems thinking

The failure we’re looking for

Calls a channel a loop, or a referral button a viral loop

Grading

Decision model and LLM judge, calibrated against a blind PM review