Tasks / Growth

Find the growth loop

Can the model find a product's real growth loop, show whether it compounds, and say which lever to pull?

Task type v1.0 · 2 tasksLast changed 2 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Sees the cross-side effect100% pass
    It traces the chain from the tutor bounty to oversupply, thinner bookings, new profiles without reviews, and weaker ranking, and acts on it by pausing broad tutor referrals.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  2. Produces the required deliverable99% pass
    The memo answers all parts of the brief (primary loop, compounding, engineer allocation, success measurement) in a usable form for the Head of Growth.
    Sonnet 5.5 · API · The badge on every form
  3. Respects explicit constraints97% pass
    The output is a memo to Sam Okoro, under 800 words, and respects the requested form and reader.
    Sonnet 5.5 · API · The badge on every form

Where it slips

  1. Finds the decay behind the growth42% pass
    Does not compute the decline in sessions per profile (~−14%) despite having the 40% session growth and 62% tutor growth.
    GPT-6 Astra · ChatGPT · Growing on the surface, decaying underneath
  2. The loop maths holds47% pass
    The memo does not give a plain verdict of 'decaying' for the content loop despite showing its decline, and it does not compute a numeric yield or coefficient for that loop.
    GPT-6.1 Sol · API · Growing on the surface, decaying underneath
  3. Uses the supplied evidence correctly61% pass
    The claim that the base settles at 10,700 creators is unsupported by the pack's arithmetic, and the claim that cost per sign-up usually rises with spend is not in the supplied evidence.
    Opus 5.5 · Claude · The badge on every form

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. A closed loop, not a channel

    Does the output name one primary loop where a cycle's output demonstrably becomes the next cycle's input, chosen from where the best users come from, and size it with the supplied numbers?

    Passes when Names one closed loop, grounded in where the highest-retention users come from, and works out its yield or coefficient with retention applied.

  2. The loop maths holds

    Is every loop yield, coefficient and payback computed correctly from the supplied steps, with retention applied, and does the output say plainly whether each loop compounds, merely contributes, or decays?

    Passes when Every yield, coefficient and payback checks out against the pack, retention is in the sums, and each loop gets a plain verdict: compounding, contributing or decaying.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tutorly. Our CEO, Rachel Dunn, wants to triple paid acquisition for the next two quarters so we reach 50,000 booking parents before the Series B. Before the planning offsite she's asked you for an honest view of how we actually grow. Write a memo of no more than 1,300 words for Rachel and the exec team that: 1. Maps our growth loops in a simple text diagram, says which is primary and why, and whether each is compounding, contributing or decaying, with the numbers. 2. Responds to the plan to triple paid acquisition. 3. Says where our three squads should go for the next two quarters, with the test, threshold and stop condition for each. The pack is below. Not all of it matters equally.

What the model was given8 items: About Tutorly, Twelve months, at the top line, Search and profiles, Where new booking parents come from, and how long they stay, Referral programmes, Paid acquisition, Rachel's note, Constraints
About TutorlyAn online tutoring marketplace: parents book one-to-one lessons with tutors. The average lesson costs $40 and we keep a fixed 18% commission. Each tutor has a public profile page, with reviews from parents.
Twelve months, at the top lineBooking parents (at least one booking a month): 31,000 → 37,000 (+20%). Active tutors: 14,200 → 23,000 (+62%). Organic search sessions to tutor profiles: +40%. Bookings per active tutor a month: 9.1 → 6.8.
Search and profilesProfiles with three or more reviews get 81% of organic profile sessions; profiles with none get 4%. Of tutor profiles created in the last six months, 61% had no booking within 60 days, so they have no reviews. Click-through from search results on our top 200 keywords fell from 6.1% to 5.0% since March, after changes to the results pages.
Where new booking parents come from, and how long they stayOrganic search: 52% of new booking parents; 48% still booking after six months; net lifetime value $260. Parent referrals: 9%; 51% after six months; net lifetime value $275. Paid search and social: 39%; 22% after six months; net lifetime value $120. Across all sources, net lifetime value averages $210.
Referral programmesParents: $20 of lesson credit for each referred parent who books. Each active parent sends 0.31 invites a month, and 14% of invites become booking parents. Tutors: $50 for each referred tutor who completes onboarding. 44% of new tutors last year came through tutor referrals.
Paid acquisitionAverage cost per new booking parent over the year: $95. Last quarter we raised spend from $60,000 to $90,000 a month, and the cost per new booking parent rose from $82 to $109.
Rachel's note“Our LTV to CAC is 2.2. Every dollar we don't put into paid is growth we're leaving on the table. Triple it to $270k a month and we hit 50,000 parents before the raise.”
ConstraintsReviews can only be left after a completed, paid lesson. Our policy, and consumer protection rules in our main markets, forbid paying or rewarding anyone for reviews. The commission stays at 18%. Three squads are available for the next two quarters; the raise is about six months away.
What a strong answer doesThe answer key the graders mark against

Maps four loops: content (profiles and reviews rank, parents book, bookings create reviews, profiles rank better), parent referral, tutor referral, and paid. Names content as primary: organic is the biggest source of new booking parents (52%) and among the stickiest (48% at six months, $260). Finds that it's decaying behind 40% session growth. Profiles grew 62% against 40% sessions, so sessions per profile fell about 14%. 61% of new profiles never get a booking, so they never get reviews, and profiles without reviews get 4% of sessions. Bookings per tutor fell from 9.1 to 6.8. Traces the cross-side cause: tutor supply growing about three times faster than demand (+62% against +20%), much of it from the $50 tutor-referral programme (44% of new tutors), which spreads bookings too thin for new tutors to earn the reviews the loop runs on. Treats the click-through drop (6.1% to 5.0%) as a separate, external hit. Sizes the parent referral loop at about 0.043 (0.31 × 14%): small but high quality. Answers Rachel with marginal cost: the extra $30,000 a month bought about 94 more parents (60,000 ÷ 82 ≈ 732 → 90,000 ÷ 109 ≈ 826), roughly $319 each, against $120 of value for a paid parent. The 2.2 ratio blends sources and uses the average cost, so tripling would burn cash on parents who mostly leave. Sends the squads to: getting new tutors their first booking and review (for example, routing demand in under-supplied subjects to new tutors, with a threshold such as the share of new profiles booked within 60 days rising from 39% to 55%); concentrating search value on reviewed profiles (keeping unreviewed profiles out of the index, and a fix for the click-through drop); and pausing or targeting the tutor-referral bounty while improving the parent loop. Each comes with a stop condition, and the memo is honest about what 50,000 parents by the raise would really take, without rewarding reviews.

Critical failures (cap the score at 40)
  • Proposes paying, crediting or otherwise rewarding parents or tutors for reviews
  • Treats the 40% growth in organic sessions as proof the content loop is healthy
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit8 of 12 checks passed

The memo covers all major analytical demands and squad plans with only a small gap (the explicit per‑profile sessions‑decline calculation); a PM could add that sentence in a quick edit without revisiting evidence or reasoning.

Output

MEMO

To: Rachel Dunn (CEO) and the Executive Team From: Staff Product Manager Date: October 24, 2023 Subject: Pre-Offsite Analysis: How Tutorly Actually Grows & Q1–Q2 Squad Allocation

Rachel, you asked for an honest view of our growth engine ahead of the Series B planning offsite, specifically looking at the proposal to triple paid acquisition to reach 50,000 booking parents.

After digging into the last twelve months of data, my conclusion is clear: tripling paid spend will burn roughly $1.1M in extra cash, turn our marginal unit economics negative, and still leave us thousands of parents short of 50,000. Worse, it will present a leaky bucket to Series B investors.

We do have a path to an incredible Series B story, and a shot at 50,000 high-value parents, but it requires fixing our broken marketplace liquidity and organic search loops rather than buying low-retention traffic. Here is the breakdown.

---

1. How Tutorly Actually Grows: Our Four Loops

#### Loop A: The Review-SEO Loop (PRIMARY — Decaying Efficiency) ```text [New/Active Tutors] ──> [Complete Paid Lessons] ──> [Earn ≥3 Parent Reviews] ▲ │ │ ▼ [Marketplace Revenue] ◄── [New Booking Parents] ◄── [Google Ranks Profiles] ``` * Why it’s primary: Organic search drives 52% of new booking parents at a $260 net LTV and 48% 6-month retention. It is the foundation of our enterprise value. * Status: Compounding in volume, but decaying in efficiency. Organic sessions grew 40%, and total bookings grew ~20% (from ~129,220/mo to ~156,400/mo). However, the loop is choking at two bottlenecks: 1. The Cold-Start Wall: Profiles with $\ge3$ reviews capture 81% of organic traffic; those with 0 get 4%. Because we grew tutors (+62%) three times faster than demand (+20%), 61% of new tutors over the last 6 months didn’t get a single booking in their first 60 days. They sit at 0 reviews, invisible to Google. 2. SERP CTR Drop: Since March’s search results changes, CTR on our top 200 keywords fell from 6.1% to 5.0% (an 18% relative drop in top-of-funnel traffic).

#### Loop B: Paid Acquisition Loop (Linear/Contributing — Decaying) ```text [Working Capital] ──> [Paid Ads ($90k/mo)] ──> [New Booking Parents (39% share)] ▲ │ └────── [18% Commission ($7.20/lesson)] ◄────────┘ ``` * Status: Decaying. Paid is a linear lever, not a true compounding loop, because our 6-month retention on paid parents is just 22% (meaning we don't recapture cash fast enough to reinvest). Last quarter, raising spend 50% ($60k to $90k/mo) caused CAC to jump 33% ($82 to $109).

#### Loop C: Parent Viral Loop (Contributing — Sub-viral) ```text [Active Booking Parent] ──(0.31 invites/mo)──> [14% Convert to Booking] ──> [New Active Parent] ``` * Status: Contributing ($K < 1$). Each active parent generates 0.0434 new parents per month ($0.31 \times 14\%$). While sub-viral, these are our best users ($275 LTV, 51% 6-month retention), driving 9% of new parents (~1,600/mo) at a bargain $20 CAC.

#### Loop D: Tutor Viral Loop (Compounding — Uncalibrated) ```text [Onboarded Tutor] ──($50 Referral)──> [New Tutor Onboards (44% of supply growth)] ``` * Status: Compounding, but value-destructive right now. We added 8,800 net active tutors (+62%) largely by paying $50 for referrals upon onboarding rather than booking. This flooded the marketplace with unbooked supply, dragging monthly bookings per active tutor down 25% (from 9.1 to 6.8).

---

2. Why We Must Not Triple Paid Acquisition

The thesis that "our LTV:CAC is 2.2x, so tripling paid to $270k/mo gets us to 50k parents" rests on three mathematical traps:

  1. The Blended LTV Fallacy: The 2.2x ratio divides our blended LTV ($210—heavily propped up by $260 Organic and $275 Referral users) by our historical average Paid CAC ($95). If you isolate the paid channel, Paid LTV is $120 and last quarter’s Paid CAC was $109. Our true Paid LTV:CAC is 1.1x—and on a marginal basis, it is already below 1.0x.
  2. Diminishing Returns on Ad Spend: When we increased spend by $30k/mo last quarter, we only got ~94 incremental parents (731/mo at $60k spend vs. 825/mo at $90k spend). That means our marginal CAC on that extra $30k was ~$319—for a user worth $120. Tripling spend to $270k/mo will push blended CAC well past $160–$180. We will lose $40–$60 on every parent we acquire.
  3. The 22% Retention Treadmill: Even if CAC miraculously stayed flat at $109, adding $180k/mo in spend for 6 months buys ~9,900 gross parents. Because 78% of paid parents churn within 6 months, over half of those cohorts will have churned before we close the Series B. Net active parents would only grow by ~4,000–4,500—leaving us well short of 50,000 and down $1.08M in cash.

What Series B investors will see: Growth investors run cohort and channel-level unit economics in diligence. If they see negative marginal return on ad spend, 22% paid retention, and tutor utilization falling from 9.1 to 6.8, they will heavily discount our valuation.

Recommendation: Roll paid spend back to its efficient frontier ($60k/mo at ~$82 CAC, saving $180k/quarter vs. current run rate) and deploy our three product squads to fix the high-LTV organic and referral engines.

---

3. Six-Month Product Plan: Where Our Three Squads Go

To add 13,000 net active parents in six months (hitting 50k), we need to unlock the 61% of stranded tutors so Google indexes them, reverse the search CTR drop, and reduce churn.

#### Squad 1: Marketplace Liquidity & Cold-Start (Fixing the 61% Unbooked Tutors) * Objective: Route existing parent demand to new/unreviewed tutors so they cross the 3-review threshold (where 81% of SEO traffic lives), without violating review-incentive laws. * The Tests: 1. Internal Search Boost & "New to Tutorly" Badging: Give tutors with $<3$ reviews a temporary ranking boost in internal marketplace search, paired with a "Verified Background & Vetted Credential" badge to substitute for missing social proof. 2. First-Lesson Intro Pricing: Prompt new tutors during onboarding to opt into a 25% discounted introductory rate for their first 5 bookings, plus a "100% Parent Satisfaction Replacement Guarantee" underwritten by Tutorly. 3. Post-Lesson Review Friction: We cannot incentivize reviews, but we can mandate a 1-click rating prompt in the post-lesson video/checkout flow before booking the next session. 4. Fix Tutor Referral Payouts: Change the $50 tutor referral reward trigger from completed onboarding to first completed lesson to stop flooding the site with low-intent supply. * Success Threshold: Reduce 60-day unbooked new tutor rate from 61% to $\le$35% within 90 days; increase the share of active tutors reaching $\ge3$ reviews within 60 days by 50%. * Stop Condition: If routing traffic to 0-review tutors reduces overall site conversion rate by $>5\%$ relative or drops 2nd-lesson parent retention on those bookings by $>10\%$ after 4 weeks, roll back the search boost and restrict routing to high-intent overflow categories.

#### Squad 2: Organic Search & SERP Conversion (Fixing the 6.1% $\rightarrow$ 5.0% CTR Drop) * Objective: Recover and expand the top of our primary funnel (52% of parents, $260 LTV). Restoring CTR from 5.0% to 6.1% alone represents a 22% instant lift in organic traffic on our top keywords. * The Tests: 1. March Rollback / SERP Audit: Audit the March search results page changes and Google’s SERP layout shifts. Run SEO split-tests (via SearchPilot or cohort bucketing) on title tags, meta descriptions, and `AggregateRating` / `PriceRange` schema markup so star ratings and "$40/hr" show directly in Google results. 2. Category Aggregation Pages: Profiles with 0 reviews get 4% of traffic. Instead of relying solely on individual tutor profiles to rank, build programmatic local/subject landing pages (e.g., "Top 10 GCSE Maths Tutors — From $35/hr") that aggregate reviews across tutors and capture top-200 keyword clicks. * Success Threshold: Restore top-200 keyword CTR from 5.0% to $\ge$6.0% within 8 weeks; accelerate organic session growth from +40% YoY to +65% YoY by end of Q2. * Stop Condition: If schema/title split-tests fail to move CTR by at least +0.4 percentage points after 6 weeks of indexing, halt template-level tinkering and shift 100% of squad capacity to conversion-rate optimization (CRO) on existing organic landing pages.

#### Squad 3: Parent Retention & Viral Loop ($K$-Factor & LTV Expansion) * Objective: Every 1% improvement in monthly retention keeps ~370 parents active automatically. Simultaneously, we must fix the 22% paid retention cliff and scale our highest-LTV channel (Parent Referrals, $275 LTV). * The Tests: 1. Recurring Booking Default: Shift the checkout flow from single-lesson booking to a default weekly recurring slot (with free 24-hour cancellation). Most churn happens in the friction between Lesson 1 and Lesson 2. 2. Paid Landing Page Alignment: Paid users churn at 78% because ads drop them into generic flows with high tutor mismatch. Route paid traffic to curated tutor shortlists with $\ge3$ reviews and high repeat-booking rates. 3. Peak-Delight Referral Triggers: Move the $20 parent referral prompt out of the nav bar and trigger it immediately after a parent leaves a 5-star review or completes their 3rd lesson with the same tutor, utilizing pre-written WhatsApp/SMS share links for parent school groups. * Success Threshold: Lift monthly parent invite rate from 0.31 to $\ge$0.50 invites/parent (growing referral share from 9% to 15% of new parents); lift Paid 6-month retention trajectory (measured via Month-2 repeat rate proxy) by +10 percentage points. * Stop Condition: If recurring-by-default drops initial checkout conversion by $>8\%$ relative without a compensating lift in 30-day total lessons per parent after 3 weeks, revert to single-lesson checkout and trigger the recurring upsell immediately after Lesson 1 completes.

Grades and run details

Decision model 75 · LLM judge 77
Decision model checks
  • failUses the supplied evidence correctly23%
  • passAddresses the actual decision98%
  • passRespects explicit constraints19%
  • passIdentifies material uncertainty67%
  • partialAvoids unsupported claims45%
  • passProduces the required deliverable73%
  • partialFinds the decay behind the growth48%
  • passSees the cross-side effect100%
  • passAnswers the CEO on paid100%
  • passA closed loop, not a channel66%
  • partialThe loop maths holds22%
  • partialProposes tests that could fail69%
Run
Run
#1
API response time
34 s
Submitted
13 Oct 2026

Needs more than a quick edit7 of 12 checks passed

Contains a factual error on organic retention (46% vs 48%), mislabels the primary loop as compounding, and miscalculates marginal paid cost. A PM would need to re-check evidence and redo the loop and paid analysis.

Output

MEMORANDUM

TO: Rachel Dunn, CEO, and Executive Team FROM: Staff Product Manager, Tutorly DATE: October 24, 2023 SUBJECT: Growth Analysis, Paid Acquisition Plan Review, and Q1–Q2 Squad Roadmap

*

Executive Summary

To reach 50,000 booking parents before the Series B, we must separate our top-line metrics from the health of our marketplace engine. While growing booking parents from 37,000 to 50,000 (+35%) in two quarters is mathematically possible, tripling paid acquisition to achieve it will accelerate churn, drain cash, and mask structural supply-side liquidity issues that threaten our Series B valuation.

This memo maps our true growth mechanics, evaluates the paid acquisition proposal, and redeploys our three engineering squads to fix marketplace liquidity before we scale demand.

---

Part 1: Mapping Our Growth Loops

Tutorly operates on three core loops. The numbers reveal a marketplace suffering from top-heavy demand generation sitting atop a leaking supply foundation.

``` [1. SEO & Tutor Profile Loop] (PRIMARY) Tutors onboard (44% via referral) --> Create Public Profile Pages --> Indexed by Search Engines --> Organic Search Sessions (52% of parents) --> Completed Lessons & Reviews (81% traffic to profiles w/ 3+ reviews) --> Higher Search Rankings & Marketplace Trust --> More Parent Bookings --> Tutors earn & refer peers (Loop Compounding)

[2. Parent Referral Loop] (CONTRIBUTING) Booking Parents --> Send Invites (0.31/mo) --> Referred Parents Book (14% conversion) --> High Retention (51% at 6 months, $275 LTV) --> More Parents --> More Invites (Loop Compounding, but low volume at 9% share)

[3. Paid Acquisition Loop] (DECAYING AT SCALE) Capital ($90k/mo) --> Paid Search & Social (39% of parents) --> New Booking Parents --> Low Retention (22% at 6 months, $120 LTV) --> Blended LTV:CAC drops from 2.2 to 1.1 at scale --> Capital Inefficient (Loop Decaying) ```

#### Loop Status and Metrics

1. SEO & Tutor Profile Loop (Primary | Compounding): * Status: Compounding at the top, but restricted by a liquidity bottleneck at the bottom. * The Numbers: Organic search drives 52% of new booking parents with a healthy 46% 6-month retention and $260 net LTV. However, active tutors grew +62% (14.2k to 23k), while bookings per tutor dropped from 9.1 to 6.8/month. Because 61% of new tutors get zero bookings in 60 days, they never reach 3+ reviews. Since profiles with 3+ reviews capture 81% of organic profile traffic (vs. 4% for zero reviews), our SEO asset is trapped behind unactivated supply. Furthermore, CTR on our top 200 keywords dropped from 6.1% to 5.0% due to recent SERP changes.

2. Parent Referral Loop (Contributing | Compounding): * Status: Healthy unit economics, but low absolute volume. * The Numbers: Generates 9% of new booking parents. Each active parent sends 0.31 invites/month with a 14% conversion rate. Retains exceptionally well (51% after six months) with a net LTV of $275. It compounds efficiently but lacks the velocity to move the top line alone.

3. Paid Acquisition Loop (Secondary | Decaying): * Status: Decaying rapidly at marginal scale. * The Numbers: Drives 39% of new booking parents, but retention crashes to 22% at six months (net LTV of $120). When we raised spend from $60k to $90k/month last quarter, the cost per new booking parent ($CAC$) jumped from $82 to $109.

---

Part 2: Response to the Plan to Triple Paid Acquisition

Rachel, your note argues: "Our LTV to CAC is 2.2. Every dollar we don't put into paid is growth we're leaving on the table. Triple it to $270k a month and we hit 50,000 parents before the raise."

An honest view of the data shows this plan will undermine our Series B narrative. Here is why:

  1. Blended LTV is a dangerous illusion: Our blended LTV is $210 against a blended CAC of $95 (yielding 2.2). But paid acquisition cohorts have an LTV of only $120. When we pushed spend to $90k/month last quarter, marginal CAC rose to $109. Tripling spend to $270k/month will push marginal CAC past $160–$180, well above the $120 LTV of paid users. We would be acquiring customers at a net loss ($40+ loss per paid parent).
  2. Exacerbating the supply-side liquidity crisis: Active tutors are already seeing their monthly bookings fall from 9.1 to 6.8 (-25%) because supply (23,000) is growing faster than demand (37,000 parents). Pouring cheap, low-intent paid traffic into the top of the funnel without fixing tutor activation will worsen tutor churn, starving our primary SEO loop (which relies on active tutors getting reviews).
  3. Series B Risk: Investors scrutinize cohort retention and payback periods. A 22% six-month retention rate on 50,000 paid parents signals a leaky bucket that will cause churn spikes right after our Series B closes.

Conclusion on Paid: We should cap paid acquisition at a disciplined $90k/month, focusing strictly on high-intent brand and bottom-funnel keywords where marginal CAC remains under $85.

---

Part 3: Q1–Q2 Squad Roadmap

To reach 50,000 booking parents sustainably for the Series B, we must unlock supply liquidity and fix our primary organic engine. We will deploy our three squads to specific operational bottlenecks.

---

#### Squad 1: Search & Discovery Squad * Mission: Reverse the CTR decline on organic search results and optimize tutor profile visibility to capture existing search demand without increasing paid spend. * Test: * Hypothesis: Redesigning the search results page (SERP) layout and introducing structured review snippets will recover our CTR from 5.0% back to baseline (6.1%) and increase profile-to-booking conversion. * Action: A/B test a revised SERP layout prioritizing tutor responsiveness and verified subject badges. * Threshold (Go/No-Go): Recovers organic traffic conversion by $\ge$ 15% within 45 days. * Stop Condition: If CTR remains flat or declines further after two iterative algorithm tests, pivot the squad to programmatic landing page generation for long-tail tutoring subjects.

---

#### Squad 2: Tutor Activation Squad * Mission: Ensure new tutors get their first booking within 30 days, breaking the cold-start problem so they accumulate the 3 reviews required to unlock 81% of organic profile traffic. * Test: * Hypothesis: Lowering the barrier to initial trial lessons (e.g., introducing a standardized 30-minute introductory lesson priced at $15, where Tutorly waives commission) will drive the first completed lesson within 30 days for >50% of new tutors. * Action: Launch "Introductory Lesson Pass" for new tutors to accelerate time-to-first-review. * Threshold (Go/No-Group): Reduces the 60-day zero-booking rate from 61% down to $\le$ 40%. * Stop Condition: If zero-booking rates do not drop below 50% after 60 days, sunset the trial pass and test supply-capping (restricting new tutor onboarding in over-supplied subjects to concentrate demand).

---

#### Squad 3: Parent Retention & Referral Squad * Mission: Improve early-stage retention for paid-acquired parents (who drop to 22% at 6 months) and optimize the parent referral loop. * Test: * Hypothesis: Onboarding paid-acquired parents with an automated, subject-specific tutor matching wizard rather than self-serve browsing will increase 3-month retention by 20%. * Action: Implement an AI/rules-based tutor matching onboarding flow for all non-organic acquisition channels. * Threshold (Go/No-Go): Increases 90-day retention of paid cohorts from current baseline to $\ge$ 35%. * Stop Condition: If paid cohort retention fails to improve, reduce paid spend further to $60k/month and reallocate budget to scaling our Parent Referral program (which yields a $275 LTV and 51% retention).

---

Summary Table: Q1–Q2 Roadmap

SquadFocus AreaCore TestSuccess ThresholdStop Condition
:---:---:---:---:---
1. Search & DiscoverySERP CTR & Profile ConversionSERP layout redesign & review snippets+15% conversion lift in 45 daysPivot to programmatic landing pages
2. Tutor Activation60-Day Zero-Booking Rate"Intro Pass" (zero-commission first lesson)60-day zero bookings drops from 61% to $\le$40%Implement strict supply-capping
3. Retention & ReferralPaid Cohort RetentionGuided tutor-matching onboarding flowPaid cohort 90-day retention $\ge$35%Cut paid spend to $60k; scale referrals

Final Word

Rachel, we can hit 50,000 booking parents before the Series B, but not by brute-forcing paid channels that destroy our unit economics. By fixing tutor activation (Squad 2) and recovering search efficiency (Squad 1), we supercharge our compounding SEO loop. That is the growth story that commands a premium Series B valuation.

Claims the judge couldn’t find in the brief

  • Organic search retention is 46% at six months

Grades and run details

Decision model 83 · LLM judge 54
Decision model checks
  • passUses the supplied evidence correctly1%
  • passAddresses the actual decision94%
  • passRespects explicit constraints9%
  • passIdentifies material uncertainty31%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable51%
  • partialFinds the decay behind the growth91%
  • passSees the cross-side effect94%
  • partialAnswers the CEO on paid64%
  • passA closed loop, not a channel64%
  • partialThe loop maths holds20%
  • passProposes tests that could fail30%
Run
Run
#1
Time to output
12 s
Submitted
2 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 1

Answers the CEO on paidRightWrong
Gemini 4 Argon · API

The memo calculates the marginal CAC (~$319) from the last spend increase, compares it with the paid parent's $120 LTV, explains the 2.2 blended-LTV fallacy, and firmly advises against tripling ad spend.

Gemini 3.5 Flash-Lite · Gemini

Treats the $109 average cost as the marginal cost and does not compute the true marginal cost (~$319) of the last spend increment; the marginal analysis is incorrect.

All got wrong 2

Finds the decay behind the growthWrongWrong
Gemini 4 Argon · API

The memo shows the 61% unbooked rate and the fall in bookings per tutor but does not compute the per-profile organic session decline (approximately −14%), which is required to fully show decay.

Gemini 3.5 Flash-Lite · Gemini

Labels the primary SEO loop as 'Compounding' and does not conclude it is decaying, despite having the numbers to show weakening per profile.

The loop maths holdsWrongWrong
Gemini 4 Argon · API

Only the parent viral loop coefficient is computed; the primary content loop and the paid loop lack the required quantitative yield, coefficient or payback figures computed from the pack with retention applied.

Gemini 3.5 Flash-Lite · Gemini

No loop yields, coefficients or payback periods are calculated; the verdicts (compounding/contributing/decaying) are stated without supporting maths.

All mixed 2

Uses the supplied evidence correctlyMixedMixed
Gemini 4 Argon · API

All key factual claims about the current situation are taken from the supplied context or derived from it by arithmetic; no invented facts.

Gemini 3.5 Flash-Lite · Gemini

States organic retention as 46% instead of the correct 48% from the brief, a factual error about the current situation.

A closed loop, not a channelMixedMixed
Gemini 4 Argon · API

It names the Review-SEO loop as the primary closed loop but does not work out a yield or coefficient with retention applied for that loop (e.g., sessions per profile or new parents per review), only stating traffic share and LTV.

Gemini 3.5 Flash-Lite · Gemini

Does not compute a yield or coefficient for the primary loop with retention applied; only describes the loop qualitatively.

All got right 7

Addresses the actual decisionRightRight
Gemini 4 Argon · API

The memo commits early and clearly to rolling back paid spend and reinvesting in organic/referral loops, and identifies stop conditions that could change the call.

Gemini 3.5 Flash-Lite · Gemini

Commits to capping paid at $90k/month and gives a squad roadmap with clear conditions that would change the call.

Respects explicit constraintsRightRight
Gemini 4 Argon · API

No proposal rewards or pays for reviews; the payout change for tutor referrals is bounty-timing, not a review incentive; the commission stays 18%; the memo is in the requested form.

Gemini 3.5 Flash-Lite · Gemini

Memo is within 1,300 words, addressed to Rachel and exec team, respects no-paying-for-reviews rule, and uses three squads.

Identifies material uncertaintyRightRight
Gemini 4 Argon · API

Each squad proposal includes thresholds and what results would trigger a rollback or pivot, making unknowns and their resolution explicit.

Gemini 3.5 Flash-Lite · Gemini

Each squad proposal includes a stop condition that names what result would change the course of action.

Avoids unsupported claimsRightRight
Gemini 4 Argon · API

Causal claims are grounded in the data (oversupply from tutor referrals linked to bookings-per-tutor drop, etc.) and are not presented as certainty without evidence.

Gemini 3.5 Flash-Lite · Gemini

Forecasts and hypotheses are clearly labelled; the only error is a factual mistake, not an unsupported interpretation.

Produces the required deliverableRightRight
Gemini 4 Argon · API

The memo is addressed to Rachel and the exec team, follows a clear structure, and appears within the 1,300-word limit; a PM could act on it directly.

Gemini 3.5 Flash-Lite · Gemini

Delivers a complete memo with loops, paid response, and three squad plans with tests, thresholds and stop conditions.

Sees the cross-side effectRightRight
Gemini 4 Argon · API

It traces the chain from the $50 tutor referral paid on onboarding, to supply outgrowing demand, to thinner bookings per tutor, to new profiles without reviews, and to weaker ranking, and proposes acting on the tutor-referral trigger.

Gemini 3.5 Flash-Lite · Gemini

Connects tutor oversupply (+62% tutors vs +20% parents) and the tutor referral bounty to falling bookings per tutor and the review bottleneck.

Proposes tests that could failRightRight
Gemini 4 Argon · API

Every squad initiative includes a numeric threshold, a time window sufficient for readout (4–12 weeks), and the specific action to take if the result crosses the threshold.

Gemini 3.5 Flash-Lite · Gemini

Each squad test has a numeric threshold, a measurement window (45, 60, 90 days), and a stop condition with a clear next action.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI91.596.22None
2GPT-6.1 SolwithAPI89.287.82None
3GPT-6 AstrawithChatGPT84.784.32None
4Opus 5.5withClaude89.264.42None
5Haiku 5.5withAPI80.567.92None
6GPT-6 LunawithAPI78.064.12None
7Gemini 4 ArgonwithAPI67.067.62None
8Gemini 3.8 FlashwithAPI73.363.52None
9Gemini 3.5 Flash-LitewithGemini57.635.32None

What you get for the cost

GPT-6.1 Sol · API scores within 5 points of Sonnet 5.5 · API at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 92, 44× the cheapestGPT-6.1 Sol · API: task score 87, 9.3× the cheapestGPT-6 Astra · ChatGPT: task score 85, 53× the cheapestOpus 5.5 · Claude: task score 73, 54× the cheapestHaiku 5.5 · API: task score 71, 2.1× the cheapestGPT-6 Luna · API: task score 68, the cheapestGemini 4 Argon · API: task score 67, 9.3× the cheapestGemini 3.8 Flash · API: task score 65, 8.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.7× the cheapestSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API681.0×
Gemini 3.5 Flash-Lite · Gemini441.7×
Haiku 5.5 · API712.1×
Gemini 3.8 Flash · API658.2×
Gemini 4 Argon · API679.3×
GPT-6.1 Sol · API879.3×
Sonnet 5.5 · API9244×
GPT-6 Astra · ChatGPT8553×
Opus 5.5 · Claude7354×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Working out what actually drives growth, and where to push.

Why it matters

Teams tune funnel steps while the loop that compounds goes unmeasured. Mistaking a channel for a loop can cost a year.

What good looks like

  • A closed loop: each cycle's output feeds the next
  • The primary loop, traced from where the best users come from
  • The loop sized: cycle time, conversion, amplification
  • Retention in the maths
  • One lever, with a test that could fail

Deliberately not measured

  • Building a full growth model in a spreadsheet
  • Channel-level media planning
Capability tested

Growth systems thinking

The failure we’re looking for

Calls a channel a loop, or a referral button a viral loop

Grading

Decision model and LLM judge, calibrated against a blind PM review