Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Strategy

Call product-market fit

Can the model tell real product-market fit from a blended survey, a launch spike or a flattering metric, and say where to scale?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    The output is a memo addressed to Sam, within 1,200 words, and respects the form and length requirements.
    GPT-6 Astra · ChatGPT · 38%, and the retention that isn't
  2. Finds who it's a must-have for100% pass
    It identifies recruiters as the must-have segment, noting what they value (candidate summaries, ATS scorecards, focus on the candidate) and builds the plan around them.
    GPT-6 Astra · ChatGPT · 38%, and the retention that isn't
  3. Spots false fit100% pass
    The memo calls out the August promotional MRR spike, the active-definition change, and the early erosion as misleading signs of fit.
    GPT-6 Astra · ChatGPT · 38%, and the retention that isn't

Where it slips

  1. Uses the supplied evidence correctly28% pass
    The claim that the cohort jumps line up with the August 3 definition change is not supported; the jumps occur in the week of July 27–August 2, one week before the change, so the evidence is misused.
    GPT-6.1 Sol · API · 38%, and the retention that isn't
  2. Gets the base of every number right44% pass
    The month-6 retention percentages are computed from incorrect sums and denominators, so the base is wrong.
    GPT-6 Luna · API · Do we have fit, or a spike?
  3. Unpicks the 38%53% pass
    It separates by persona but does not break down the 38% by tenure into ~11% very disappointed for users under two weeks and ~47% for established users, which the criterion explicitly requires.
    Haiku 5.5 · API · 38%, and the retention that isn't

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Judges fit by retention, not sign-ups

    Does the call on product-market fit rest on whether retention curves flatten, segment by segment, rather than on sign-ups, revenue growth or a survey alone?

    Passes when Reads retention by segment from the data and lets it decide where fit exists, with the survey as supporting evidence.

    Todd JacksonLenny’s PodcastSean EllisLenny’s Podcast

  2. Finds who it's a must-have for

    Does the output profile the users who would be very disappointed without the product, and aim the plan at them and at the users closest to them?

    Passes when Names the segment or persona where the product is a must-have, what they value, and builds the plan around them.

    Rahul VohraLenny’s Podcast

  3. Spots false fit

    Does the output catch evidence that looks like fit but isn't, such as revenue won through custom work, a launch spike, a promotion, or a metric whose definition changed?

    Passes when Names each piece of flattering evidence in the material and explains why it isn't fit.

    Product and growth leadersOn Lenny’s Podcast

  4. Doesn't scale before fit

    Does the recommendation avoid pouring spend or headcount into segments that haven't shown fit, and say what would justify scaling them later?

    Passes when Scales only where fit is shown, and sets the evidence that would justify widening.

    Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Relay. Our CEO, Dana Kowalski, wants to triple the sales team next quarter, "because we clearly have product-market fit", and has asked for your view before Thursday's leadership meeting. Write a memo for her of no more than 900 words: do we have product-market fit, for whom, and what should we do next quarter instead of (or as well as) tripling sales? The files are attached below. Work from the data, not from anyone's summary of it.

What the model was given5 items: About Relay, pmf_survey_responses.csv (sent in September to every paying account's admin), logo_retention_by_cohort.csv (paying accounts still active, by sign-up month and segment), signups_by_month.csv, q3_new_business.md
About RelayShift scheduling software for teams that work in shifts: healthcare clinics, hospitality, retail and others. Sold per location per month. 1,120 paying accounts.
pmf_survey_responses.csv (sent in September to every paying account's admin)response_id,segment,company_size,role,months_using,answer,main_benefit R1001,Other,10-50,Operations manager,12,very disappointed,Fine for now R1002,Hospitality,50-200,Area manager,12,somewhat disappointed,Staff like it R1003,Retail,50-200,Store manager,7,not disappointed,Staff can see the rota R1004,Healthcare clinic,50-200,Office manager,8,very disappointed,On-call swaps that follow our rules R1005,Retail,50-200,Regional manager,8,not disappointed,Shift swaps R1006,Hospitality,200-500,Owner,2,very disappointed,Not sure yet R1007,Retail,50-200,Store manager,4,very disappointed,Easier than paper R1008,Other,50-200,Owner,12,very disappointed,Fine for now R1009,Retail,10-50,Store manager,1,not disappointed,Fine for now R1010,Hospitality,10-50,Head chef,11,not disappointed,Shift swaps R1011,Hospitality,50-200,Owner,8,somewhat disappointed,Shift swaps R1012,Retail,50-200,Store manager,1,very disappointed,Shift swaps R1013,Healthcare clinic,50-200,Operations manager,12,very disappointed,"Rota done in an hour, not a day" R1014,Hospitality,50-200,Owner,12,somewhat disappointed,Easier than the spreadsheet R1015,Hospitality,200-500,Area manager,5,not disappointed,Nice app R1016,Healthcare clinic,200-500,Office manager,9,somewhat disappointed,Staff stop texting me at night R1017,Hospitality,10-50,Head chef,5,somewhat disappointed,Staff like it R1018,Hospitality,200-500,Area manager,12,somewhat disappointed,Not sure yet R1019,Healthcare clinic,50-200,Practice manager,14,very disappointed,Staff stop texting me at night R1020,Other,10-50,Team lead,6,not disappointed,Shift swaps R1021,Healthcare clinic,200-500,Operations manager,3,somewhat disappointed,Covering sick calls fast R1022,Retail,50-200,Owner,3,somewhat disappointed,Shift swaps R1023,Healthcare clinic,200-500,Office manager,4,very disappointed,Covering sick calls fast R1024,Hospitality,200-500,Owner,1,somewhat disappointed,Staff like it R1025,Other,50-200,Team lead,11,somewhat disappointed,Shift swaps R1026,Hospitality,10-50,General manager,13,very disappointed,Easier than the spreadsheet R1027,Hospitality,10-50,General manager,4,somewhat disappointed,Shift swaps R1028,Hospitality,10-50,Head chef,6,somewhat disappointed,Not sure yet R1029,Hospitality,10-50,Owner,6,somewhat disappointed,Easier than the spreadsheet R1030,Healthcare clinic,50-200,Office manager,2,not disappointed,On-call swaps that follow our rules R1031,Retail,50-200,Regional manager,14,very disappointed,Staff can see the rota R1032,Healthcare clinic,50-200,Office manager,2,somewhat disappointed,On-call swaps that follow our rules R1033,Retail,10-50,Store manager,3,somewhat disappointed,Shift swaps R1034,Hospitality,10-50,Area manager,13,not disappointed,Easier than the spreadsheet R1035,Hospitality,10-50,Area manager,9,not disappointed,Shift swaps R1036,Other,50-200,Owner,13,not disappointed,Saves time R1037,Retail,10-50,Store manager,11,very disappointed,Staff can see the rota R1038,Retail,50-200,Store manager,10,somewhat disappointed,Easier than paper R1039,Other,50-200,Owner,5,very disappointed,Shift swaps R1040,Other,50-200,Team lead,3,not disappointed,Saves time R1041,Retail,10-50,Owner,10,not disappointed,Shift swaps R1042,Other,10-50,Owner,6,somewhat disappointed,Saves time R1043,Healthcare clinic,200-500,Clinical lead,6,very disappointed,On-call swaps that follow our rules R1044,Healthcare clinic,50-200,Office manager,12,very disappointed,Credential checks before a shift R1045,Hospitality,200-500,Area manager,12,not disappointed,Not sure yet R1046,Healthcare clinic,50-200,Office manager,5,somewhat disappointed,Covering sick calls fast R1047,Healthcare clinic,50-200,Operations manager,10,somewhat disappointed,Covering sick calls fast R1048,Healthcare clinic,200-500,Clinical lead,12,not disappointed,Covering sick calls fast R1049,Hospitality,10-50,Area manager,11,somewhat disappointed,Nice app R1050,Other,50-200,Team lead,6,somewhat disappointed,Saves time R1051,Hospitality,10-50,Head chef,9,somewhat disappointed,Shift swaps R1052,Hospitality,10-50,Area manager,3,not disappointed,Shift swaps R1053,Retail,50-200,Regional manager,7,not disappointed,Fine for now R1054,Retail,10-50,Regional manager,13,somewhat disappointed,Fine for now R1055,Retail,50-200,Store manager,4,very disappointed,Fine for now R1056,Healthcare clinic,50-200,Clinical lead,4,very disappointed,Covering sick calls fast R1057,Healthcare clinic,50-200,Clinical lead,6,very disappointed,On-call swaps that follow our rules R1058,Hospitality,10-50,Area manager,11,somewhat disappointed,Nice app R1059,Healthcare clinic,200-500,Operations manager,14,somewhat disappointed,Credential checks before a shift R1060,Hospitality,10-50,Owner,5,very disappointed,Staff like it R1061,Healthcare clinic,50-200,Clinical lead,1,very disappointed,Covering sick calls fast R1062,Retail,50-200,Regional manager,6,somewhat disappointed,Easier than paper R1063,Hospitality,200-500,Area manager,7,somewhat disappointed,Nice app R1064,Other,10-50,Team lead,7,somewhat disappointed,Saves time R1065,Healthcare clinic,50-200,Clinical lead,10,very disappointed,Staff stop texting me at night R1066,Retail,10-50,Store manager,4,not disappointed,Fine for now R1067,Hospitality,10-50,General manager,8,very disappointed,Not sure yet R1068,Other,50-200,Owner,4,somewhat disappointed,Fine for now R1069,Hospitality,50-200,General manager,5,not disappointed,Nice app R1070,Healthcare clinic,50-200,Office manager,5,somewhat disappointed,Covering sick calls fast R1071,Retail,50-200,Store manager,11,not disappointed,Staff can see the rota R1072,Hospitality,10-50,Head chef,14,somewhat disappointed,Easier than the spreadsheet R1073,Healthcare clinic,50-200,Operations manager,14,somewhat disappointed,"Rota done in an hour, not a day" R1074,Hospitality,10-50,General manager,13,not disappointed,Easier than the spreadsheet R1075,Retail,50-200,Owner,7,not disappointed,Staff can see the rota R1076,Hospitality,200-500,Area manager,1,very disappointed,Shift swaps R1077,Retail,50-200,Owner,1,not disappointed,Shift swaps R1078,Healthcare clinic,50-200,Practice manager,12,somewhat disappointed,Credential checks before a shift R1079,Healthcare clinic,200-500,Operations manager,9,very disappointed,"Rota done in an hour, not a day" R1080,Hospitality,50-200,General manager,11,not disappointed,Not sure yet R1081,Retail,10-50,Regional manager,11,not disappointed,Fine for now R1082,Retail,10-50,Store manager,8,somewhat disappointed,Easier than paper R1083,Hospitality,200-500,Area manager,5,not disappointed,Shift swaps R1084,Retail,10-50,Store manager,10,somewhat disappointed,Easier than paper R1085,Hospitality,200-500,Owner,1,not disappointed,Nice app R1086,Healthcare clinic,200-500,Practice manager,14,very disappointed,Covering sick calls fast R1087,Hospitality,50-200,Owner,4,somewhat disappointed,Nice app R1088,Hospitality,10-50,General manager,4,somewhat disappointed,Staff like it R1089,Hospitality,50-200,Area manager,13,somewhat disappointed,Nice app R1090,Healthcare clinic,200-500,Clinical lead,2,very disappointed,Staff stop texting me at night R1091,Hospitality,200-500,Owner,4,not disappointed,Nice app R1092,Other,10-50,Operations manager,1,somewhat disappointed,Saves time R1093,Retail,50-200,Store manager,1,somewhat disappointed,Easier than paper R1094,Retail,50-200,Owner,3,not disappointed,Easier than paper R1095,Healthcare clinic,50-200,Practice manager,10,very disappointed,On-call swaps that follow our rules R1096,Hospitality,10-50,General manager,4,not disappointed,Easier than the spreadsheet R1097,Hospitality,10-50,General manager,10,very disappointed,Staff like it R1098,Healthcare clinic,200-500,Practice manager,8,very disappointed,Covering sick calls fast R1099,Other,50-200,Owner,5,very disappointed,Saves time R1100,Healthcare clinic,50-200,Operations manager,8,not disappointed,Staff stop texting me at night R1101,Hospitality,200-500,General manager,5,not disappointed,Nice app R1102,Retail,50-200,Owner,9,not disappointed,Staff can see the rota R1103,Retail,50-200,Store manager,4,not disappointed,Staff can see the rota R1104,Healthcare clinic,200-500,Clinical lead,8,somewhat disappointed,Credential checks before a shift R1105,Retail,50-200,Regional manager,8,somewhat disappointed,Fine for now R1106,Retail,50-200,Owner,10,very disappointed,Shift swaps R1107,Other,50-200,Operations manager,5,very disappointed,Saves time R1108,Retail,10-50,Store manager,13,not disappointed,Shift swaps R1109,Hospitality,200-500,General manager,12,somewhat disappointed,Staff like it R1110,Other,50-200,Team lead,3,somewhat disappointed,Shift swaps R1111,Hospitality,50-200,Owner,13,not disappointed,Staff like it R1112,Hospitality,10-50,Area manager,14,somewhat disappointed,Staff like it R1113,Retail,10-50,Regional manager,2,somewhat disappointed,Fine for now R1114,Hospitality,10-50,Head chef,10,very disappointed,Staff like it R1115,Retail,50-200,Store manager,12,not disappointed,Staff can see the rota R1116,Hospitality,10-50,Area manager,11,somewhat disappointed,Not sure yet R1117,Retail,10-50,Owner,1,not disappointed,Fine for now R1118,Other,10-50,Operations manager,13,somewhat disappointed,Shift swaps R1119,Hospitality,50-200,Area manager,8,very disappointed,Shift swaps R1120,Hospitality,50-200,Area manager,3,very disappointed,Nice app R1121,Hospitality,50-200,Head chef,7,not disappointed,Shift swaps R1122,Other,50-200,Operations manager,7,somewhat disappointed,Shift swaps R1123,Other,50-200,Operations manager,9,not disappointed,Shift swaps R1124,Retail,50-200,Owner,9,somewhat disappointed,Fine for now R1125,Retail,50-200,Store manager,11,somewhat disappointed,Fine for now R1126,Healthcare clinic,50-200,Office manager,5,not disappointed,Staff stop texting me at night R1127,Healthcare clinic,50-200,Office manager,13,very disappointed,Staff stop texting me at night R1128,Retail,50-200,Store manager,3,very disappointed,Easier than paper R1129,Healthcare clinic,50-200,Office manager,14,somewhat disappointed,"Rota done in an hour, not a day" R1130,Hospitality,50-200,Area manager,10,not disappointed,Shift swaps R1131,Healthcare clinic,50-200,Office manager,8,very disappointed,Staff stop texting me at night R1132,Healthcare clinic,50-200,Operations manager,5,somewhat disappointed,Covering sick calls fast R1133,Other,50-200,Team lead,11,not disappointed,Shift swaps R1134,Healthcare clinic,200-500,Practice manager,12,somewhat disappointed,On-call swaps that follow our rules R1135,Hospitality,10-50,General manager,11,very disappointed,Easier than the spreadsheet R1136,Healthcare clinic,200-500,Office manager,8,very disappointed,Staff stop texting me at night R1137,Retail,10-50,Store manager,12,somewhat disappointed,Shift swaps R1138,Retail,50-200,Owner,5,somewhat disappointed,Fine for now R1139,Hospitality,10-50,Owner,8,somewhat disappointed,Not sure yet R1140,Retail,10-50,Owner,9,not disappointed,Staff can see the rota R1141,Healthcare clinic,200-500,Operations manager,6,not disappointed,"Rota done in an hour, not a day" R1142,Other,10-50,Owner,3,not disappointed,Shift swaps R1143,Retail,10-50,Owner,2,very disappointed,Staff can see the rota R1144,Retail,10-50,Store manager,12,somewhat disappointed,Easier than paper R1145,Hospitality,10-50,Owner,12,very disappointed,Shift swaps R1146,Retail,10-50,Owner,6,somewhat disappointed,Fine for now R1147,Other,50-200,Owner,7,not disappointed,Fine for now R1148,Retail,10-50,Regional manager,12,very disappointed,Easier than paper R1149,Hospitality,10-50,Owner,9,somewhat disappointed,Nice app R1150,Hospitality,10-50,Head chef,11,not disappointed,Nice app R1151,Retail,10-50,Regional manager,2,not disappointed,Shift swaps R1152,Retail,50-200,Store manager,8,not disappointed,Easier than paper R1153,Healthcare clinic,50-200,Office manager,2,very disappointed,On-call swaps that follow our rules R1154,Hospitality,200-500,Owner,13,somewhat disappointed,Not sure yet R1155,Healthcare clinic,50-200,Practice manager,2,very disappointed,"Rota done in an hour, not a day" R1156,Retail,50-200,Store manager,1,not disappointed,Easier than paper R1157,Retail,10-50,Regional manager,12,not disappointed,Fine for now R1158,Healthcare clinic,200-500,Office manager,14,very disappointed,On-call swaps that follow our rules R1159,Hospitality,200-500,Head chef,1,somewhat disappointed,Shift swaps R1160,Retail,50-200,Owner,14,somewhat disappointed,Staff can see the rota R1161,Retail,10-50,Owner,7,very disappointed,Shift swaps R1162,Healthcare clinic,200-500,Clinical lead,8,somewhat disappointed,On-call swaps that follow our rules R1163,Healthcare clinic,50-200,Clinical lead,9,very disappointed,"Rota done in an hour, not a day" R1164,Retail,10-50,Regional manager,1,somewhat disappointed,Easier than paper R1165,Healthcare clinic,200-500,Operations manager,3,very disappointed,"Rota done in an hour, not a day" R1166,Healthcare clinic,50-200,Office manager,3,not disappointed,Credential checks before a shift R1167,Retail,10-50,Store manager,5,somewhat disappointed,Staff can see the rota R1168,Hospitality,10-50,Owner,14,not disappointed,Not sure yet R1169,Hospitality,50-200,General manager,2,not disappointed,Easier than the spreadsheet R1170,Other,10-50,Owner,14,somewhat disappointed,Saves time R1171,Hospitality,200-500,Head chef,13,not disappointed,Nice app R1172,Healthcare clinic,50-200,Practice manager,2,somewhat disappointed,"Rota done in an hour, not a day" R1173,Healthcare clinic,50-200,Practice manager,7,very disappointed,"Rota done in an hour, not a day" R1174,Healthcare clinic,50-200,Clinical lead,9,somewhat disappointed,Staff stop texting me at night R1175,Healthcare clinic,200-500,Practice manager,8,very disappointed,Staff stop texting me at night R1176,Retail,10-50,Store manager,11,very disappointed,Shift swaps R1177,Healthcare clinic,200-500,Clinical lead,4,very disappointed,Staff stop texting me at night R1178,Healthcare clinic,50-200,Clinical lead,7,very disappointed,On-call swaps that follow our rules R1179,Healthcare clinic,200-500,Clinical lead,4,very disappointed,Staff stop texting me at night R1180,Healthcare clinic,50-200,Office manager,1,somewhat disappointed,"Rota done in an hour, not a day" R1181,Healthcare clinic,200-500,Clinical lead,12,not disappointed,Staff stop texting me at night R1182,Retail,10-50,Store manager,12,somewhat disappointed,Fine for now R1183,Hospitality,10-50,Owner,1,somewhat disappointed,Nice app R1184,Healthcare clinic,50-200,Clinical lead,3,very disappointed,Staff stop texting me at night R1185,Hospitality,10-50,Area manager,13,somewhat disappointed,Staff like it R1186,Healthcare clinic,200-500,Clinical lead,1,somewhat disappointed,"Rota done in an hour, not a day" R1187,Retail,10-50,Regional manager,3,somewhat disappointed,Shift swaps R1188,Hospitality,50-200,Head chef,7,not disappointed,Not sure yet R1189,Hospitality,10-50,General manager,11,not disappointed,Shift swaps R1190,Retail,10-50,Store manager,11,somewhat disappointed,Shift swaps R1191,Hospitality,50-200,Head chef,3,not disappointed,Easier than the spreadsheet R1192,Retail,10-50,Owner,14,somewhat disappointed,Staff can see the rota R1193,Hospitality,50-200,Head chef,10,not disappointed,Shift swaps R1194,Hospitality,10-50,Head chef,1,not disappointed,Easier than the spreadsheet R1195,Healthcare clinic,50-200,Clinical lead,2,not disappointed,Credential checks before a shift R1196,Hospitality,50-200,Head chef,8,not disappointed,Easier than the spreadsheet R1197,Retail,50-200,Store manager,8,not disappointed,Easier than paper R1198,Hospitality,200-500,Area manager,14,somewhat disappointed,Nice app R1199,Healthcare clinic,50-200,Office manager,1,very disappointed,Covering sick calls fast R1200,Healthcare clinic,200-500,Office manager,1,very disappointed,"Rota done in an hour, not a day" R1201,Hospitality,10-50,General manager,2,not disappointed,Nice app R1202,Hospitality,10-50,Head chef,5,somewhat disappointed,Shift swaps R1203,Hospitality,50-200,Owner,2,not disappointed,Nice app R1204,Other,10-50,Owner,3,somewhat disappointed,Shift swaps R1205,Retail,50-200,Store manager,10,somewhat disappointed,Easier than paper R1206,Hospitality,10-50,Area manager,4,not disappointed,Not sure yet R1207,Other,10-50,Operations manager,3,very disappointed,Fine for now R1208,Hospitality,50-200,Owner,7,somewhat disappointed,Not sure yet R1209,Retail,10-50,Regional manager,10,not disappointed,Staff can see the rota R1210,Other,50-200,Owner,13,very disappointed,Fine for now R1211,Other,10-50,Team lead,14,very disappointed,Fine for now R1212,Healthcare clinic,50-200,Office manager,6,somewhat disappointed,Credential checks before a shift R1213,Retail,50-200,Regional manager,7,somewhat disappointed,Shift swaps R1214,Retail,10-50,Regional manager,7,very disappointed,Shift swaps R1215,Other,50-200,Operations manager,9,somewhat disappointed,Saves time R1216,Healthcare clinic,200-500,Practice manager,14,somewhat disappointed,Credential checks before a shift R1217,Hospitality,50-200,Head chef,7,very disappointed,Not sure yet R1218,Healthcare clinic,50-200,Clinical lead,12,somewhat disappointed,Staff stop texting me at night R1219,Healthcare clinic,50-200,Operations manager,12,not disappointed,Credential checks before a shift R1220,Hospitality,10-50,Area manager,9,very disappointed,Shift swaps R1221,Hospitality,10-50,Owner,8,not disappointed,Nice app R1222,Hospitality,50-200,General manager,4,somewhat disappointed,Not sure yet R1223,Retail,50-200,Regional manager,8,somewhat disappointed,Shift swaps R1224,Hospitality,10-50,Head chef,9,somewhat disappointed,Not sure yet R1225,Retail,10-50,Regional manager,7,somewhat disappointed,Staff can see the rota R1226,Hospitality,50-200,Head chef,11,somewhat disappointed,Staff like it R1227,Other,50-200,Team lead,8,very disappointed,Saves time R1228,Hospitality,10-50,Head chef,12,somewhat disappointed,Nice app R1229,Hospitality,10-50,Head chef,9,not disappointed,Easier than the spreadsheet R1230,Hospitality,200-500,General manager,12,somewhat disappointed,Easier than the spreadsheet R1231,Hospitality,10-50,General manager,3,somewhat disappointed,Easier than the spreadsheet R1232,Hospitality,10-50,Area manager,11,very disappointed,Staff like it R1233,Healthcare clinic,50-200,Operations manager,3,somewhat disappointed,"Rota done in an hour, not a day" R1234,Other,50-200,Owner,9,somewhat disappointed,Saves time R1235,Healthcare clinic,200-500,Practice manager,13,very disappointed,On-call swaps that follow our rules R1236,Hospitality,10-50,General manager,11,not disappointed,Not sure yet R1237,Retail,50-200,Store manager,10,not disappointed,Staff can see the rota R1238,Healthcare clinic,50-200,Office manager,6,very disappointed,On-call swaps that follow our rules R1239,Hospitality,200-500,Head chef,5,not disappointed,Nice app R1240,Healthcare clinic,200-500,Operations manager,8,very disappointed,On-call swaps that follow our rules R1241,Retail,50-200,Store manager,14,somewhat disappointed,Easier than paper R1242,Hospitality,50-200,Area manager,5,somewhat disappointed,Easier than the spreadsheet R1243,Healthcare clinic,200-500,Office manager,4,not disappointed,"Rota done in an hour, not a day" R1244,Hospitality,200-500,General manager,1,very disappointed,Nice app R1245,Other,50-200,Team lead,9,not disappointed,Fine for now R1246,Healthcare clinic,50-200,Office manager,6,somewhat disappointed,Credential checks before a shift R1247,Retail,50-200,Store manager,10,somewhat disappointed,Fine for now R1248,Retail,10-50,Owner,12,somewhat disappointed,Fine for now R1249,Healthcare clinic,50-200,Operations manager,13,not disappointed,"Rota done in an hour, not a day" R1250,Hospitality,50-200,Area manager,2,somewhat disappointed,Easier than the spreadsheet R1251,Healthcare clinic,200-500,Practice manager,7,not disappointed,Credential checks before a shift R1252,Other,50-200,Operations manager,11,somewhat disappointed,Fine for now R1253,Healthcare clinic,200-500,Office manager,13,very disappointed,Covering sick calls fast R1254,Retail,50-200,Regional manager,11,not disappointed,Staff can see the rota R1255,Healthcare clinic,50-200,Clinical lead,5,somewhat disappointed,Staff stop texting me at night R1256,Hospitality,200-500,Head chef,1,somewhat disappointed,Not sure yet R1257,Hospitality,200-500,Owner,2,very disappointed,Not sure yet R1258,Hospitality,50-200,Owner,6,very disappointed,Easier than the spreadsheet R1259,Retail,10-50,Regional manager,2,somewhat disappointed,Staff can see the rota R1260,Hospitality,200-500,Head chef,7,somewhat disappointed,Nice app R1261,Healthcare clinic,50-200,Practice manager,13,very disappointed,Staff stop texting me at night R1262,Retail,10-50,Regional manager,12,very disappointed,Shift swaps R1263,Retail,10-50,Regional manager,8,not disappointed,Staff can see the rota R1264,Other,10-50,Owner,1,not disappointed,Shift swaps R1265,Other,50-200,Owner,9,very disappointed,Saves time R1266,Healthcare clinic,200-500,Operations manager,11,very disappointed,"Rota done in an hour, not a day" R1267,Hospitality,10-50,General manager,14,not disappointed,Not sure yet R1268,Hospitality,10-50,Owner,3,somewhat disappointed,Staff like it R1269,Hospitality,200-500,Head chef,9,somewhat disappointed,Nice app R1270,Hospitality,10-50,Owner,4,somewhat disappointed,Staff like it R1271,Hospitality,200-500,Owner,2,not disappointed,Staff like it R1272,Healthcare clinic,50-200,Operations manager,3,very disappointed,Covering sick calls fast R1273,Other,50-200,Team lead,3,very disappointed,Saves time R1274,Healthcare clinic,50-200,Operations manager,7,not disappointed,"Rota done in an hour, not a day" R1275,Healthcare clinic,200-500,Practice manager,12,somewhat disappointed,On-call swaps that follow our rules R1276,Healthcare clinic,50-200,Office manager,8,somewhat disappointed,Credential checks before a shift R1277,Hospitality,200-500,Area manager,12,not disappointed,Nice app R1278,Hospitality,200-500,Area manager,2,somewhat disappointed,Staff like it R1279,Healthcare clinic,200-500,Clinical lead,12,very disappointed,Covering sick calls fast R1280,Hospitality,10-50,Head chef,3,very disappointed,Shift swaps R1281,Other,50-200,Owner,2,very disappointed,Saves time R1282,Healthcare clinic,200-500,Practice manager,13,very disappointed,On-call swaps that follow our rules R1283,Other,50-200,Owner,9,somewhat disappointed,Fine for now R1284,Retail,50-200,Regional manager,12,not disappointed,Staff can see the rota R1285,Retail,10-50,Owner,2,somewhat disappointed,Fine for now R1286,Hospitality,50-200,Area manager,7,somewhat disappointed,Staff like it R1287,Retail,10-50,Owner,13,somewhat disappointed,Fine for now R1288,Healthcare clinic,200-500,Practice manager,13,somewhat disappointed,"Rota done in an hour, not a day" R1289,Other,10-50,Operations manager,4,not disappointed,Saves time R1290,Healthcare clinic,50-200,Practice manager,14,somewhat disappointed,Credential checks before a shift R1291,Healthcare clinic,200-500,Clinical lead,14,very disappointed,Credential checks before a shift R1292,Hospitality,50-200,Head chef,12,very disappointed,Not sure yet R1293,Other,10-50,Operations manager,11,very disappointed,Fine for now R1294,Hospitality,200-500,Head chef,1,somewhat disappointed,Easier than the spreadsheet R1295,Hospitality,50-200,Head chef,13,not disappointed,Staff like it R1296,Retail,10-50,Regional manager,2,somewhat disappointed,Shift swaps R1297,Retail,10-50,Regional manager,1,not disappointed,Shift swaps R1298,Healthcare clinic,50-200,Operations manager,10,somewhat disappointed,Credential checks before a shift R1299,Retail,10-50,Owner,10,somewhat disappointed,Fine for now R1300,Retail,10-50,Owner,14,not disappointed,Shift swaps R1301,Hospitality,200-500,Owner,11,not disappointed,Nice app R1302,Healthcare clinic,50-200,Operations manager,8,somewhat disappointed,Credential checks before a shift R1303,Retail,50-200,Regional manager,13,somewhat disappointed,Fine for now R1304,Hospitality,50-200,Head chef,13,not disappointed,Nice app R1305,Healthcare clinic,200-500,Operations manager,2,somewhat disappointed,On-call swaps that follow our rules R1306,Retail,10-50,Owner,11,not disappointed,Fine for now R1307,Healthcare clinic,200-500,Practice manager,14,very disappointed,Covering sick calls fast R1308,Retail,10-50,Regional manager,9,somewhat disappointed,Staff can see the rota R1309,Hospitality,10-50,Head chef,4,somewhat disappointed,Easier than the spreadsheet R1310,Healthcare clinic,50-200,Clinical lead,10,very disappointed,"Rota done in an hour, not a day" R1311,Retail,50-200,Store manager,7,not disappointed,Fine for now R1312,Hospitality,50-200,Area manager,1,very disappointed,Easier than the spreadsheet R1313,Other,10-50,Owner,3,somewhat disappointed,Fine for now R1314,Retail,10-50,Store manager,7,very disappointed,Easier than paper R1315,Hospitality,10-50,Head chef,11,very disappointed,Not sure yet R1316,Retail,50-200,Regional manager,14,somewhat disappointed,Staff can see the rota R1317,Retail,50-200,Store manager,13,somewhat disappointed,Shift swaps R1318,Retail,10-50,Store manager,9,very disappointed,Staff can see the rota R1319,Other,50-200,Owner,2,not disappointed,Shift swaps R1320,Hospitality,10-50,General manager,5,somewhat disappointed,Not sure yet R1321,Hospitality,200-500,General manager,7,very disappointed,Staff like it R1322,Healthcare clinic,200-500,Clinical lead,4,somewhat disappointed,"Rota done in an hour, not a day" R1323,Other,50-200,Operations manager,7,very disappointed,Saves time R1324,Healthcare clinic,50-200,Operations manager,8,very disappointed,On-call swaps that follow our rules R1325,Healthcare clinic,50-200,Clinical lead,14,very disappointed,"Rota done in an hour, not a day" R1326,Healthcare clinic,50-200,Office manager,11,not disappointed,Credential checks before a shift R1327,Healthcare clinic,200-500,Clinical lead,2,very disappointed,"Rota done in an hour, not a day" R1328,Other,10-50,Team lead,8,not disappointed,Shift swaps R1329,Healthcare clinic,200-500,Clinical lead,3,somewhat disappointed,Credential checks before a shift R1330,Healthcare clinic,200-500,Clinical lead,7,somewhat disappointed,Staff stop texting me at night R1331,Hospitality,200-500,Head chef,14,somewhat disappointed,Easier than the spreadsheet R1332,Healthcare clinic,200-500,Operations manager,3,somewhat disappointed,"Rota done in an hour, not a day" R1333,Retail,10-50,Regional manager,13,very disappointed,Shift swaps R1334,Hospitality,50-200,Head chef,6,somewhat disappointed,Nice app R1335,Retail,50-200,Owner,14,somewhat disappointed,Shift swaps R1336,Healthcare clinic,50-200,Clinical lead,8,very disappointed,Staff stop texting me at night R1337,Retail,10-50,Regional manager,12,not disappointed,Fine for now R1338,Retail,10-50,Store manager,5,not disappointed,Shift swaps R1339,Hospitality,50-200,Area manager,3,not disappointed,Staff like it R1340,Other,50-200,Owner,7,not disappointed,Saves time R1341,Healthcare clinic,200-500,Practice manager,12,very disappointed,Credential checks before a shift R1342,Hospitality,50-200,General manager,4,not disappointed,Nice app R1343,Other,10-50,Operations manager,2,somewhat disappointed,Fine for now R1344,Hospitality,10-50,Area manager,13,not disappointed,Easier than the spreadsheet R1345,Other,10-50,Operations manager,1,not disappointed,Saves time R1346,Retail,10-50,Regional manager,11,somewhat disappointed,Shift swaps R1347,Retail,50-200,Regional manager,5,somewhat disappointed,Staff can see the rota R1348,Hospitality,10-50,Area manager,13,somewhat disappointed,Nice app R1349,Hospitality,200-500,Head chef,14,not disappointed,Nice app R1350,Healthcare clinic,50-200,Clinical lead,5,somewhat disappointed,Covering sick calls fast R1351,Healthcare clinic,200-500,Office manager,2,very disappointed,On-call swaps that follow our rules R1352,Retail,50-200,Regional manager,11,not disappointed,Easier than paper R1353,Hospitality,10-50,General manager,2,somewhat disappointed,Not sure yet R1354,Hospitality,10-50,Owner,3,very disappointed,Easier than the spreadsheet R1355,Hospitality,10-50,Owner,1,somewhat disappointed,Nice app R1356,Retail,10-50,Store manager,10,somewhat disappointed,Fine for now R1357,Hospitality,200-500,Head chef,13,not disappointed,Shift swaps R1358,Retail,10-50,Owner,13,somewhat disappointed,Shift swaps R1359,Other,50-200,Operations manager,6,very disappointed,Shift swaps R1360,Other,10-50,Owner,1,somewhat disappointed,Saves time R1361,Hospitality,200-500,Head chef,5,not disappointed,Staff like it R1362,Hospitality,10-50,Head chef,6,not disappointed,Staff like it R1363,Hospitality,10-50,Head chef,12,very disappointed,Easier than the spreadsheet R1364,Healthcare clinic,200-500,Practice manager,6,very disappointed,"Rota done in an hour, not a day" R1365,Healthcare clinic,200-500,Clinical lead,3,not disappointed,Credential checks before a shift R1366,Retail,50-200,Owner,12,very disappointed,Fine for now R1367,Healthcare clinic,200-500,Operations manager,9,very disappointed,Covering sick calls fast R1368,Hospitality,200-500,Area manager,13,not disappointed,Easier than the spreadsheet R1369,Hospitality,10-50,Owner,2,very disappointed,Not sure yet R1370,Healthcare clinic,200-500,Clinical lead,5,very disappointed,Credential checks before a shift R1371,Hospitality,50-200,Owner,7,somewhat disappointed,Easier than the spreadsheet R1372,Hospitality,10-50,Head chef,9,somewhat disappointed,Staff like it R1373,Healthcare clinic,200-500,Practice manager,12,very disappointed,On-call swaps that follow our rules R1374,Hospitality,10-50,Owner,4,very disappointed,Staff like it R1375,Hospitality,200-500,Head chef,1,somewhat disappointed,Not sure yet R1376,Healthcare clinic,50-200,Operations manager,5,very disappointed,Credential checks before a shift R1377,Hospitality,10-50,General manager,11,somewhat disappointed,Staff like it R1378,Healthcare clinic,200-500,Practice manager,13,very disappointed,On-call swaps that follow our rules R1379,Hospitality,50-200,Head chef,7,very disappointed,Easier than the spreadsheet R1380,Healthcare clinic,50-200,Practice manager,1,very disappointed,"Rota done in an hour, not a day" R1381,Hospitality,200-500,General manager,6,somewhat disappointed,Shift swaps R1382,Hospitality,50-200,Owner,6,somewhat disappointed,Nice app R1383,Healthcare clinic,50-200,Clinical lead,13,not disappointed,On-call swaps that follow our rules R1384,Hospitality,200-500,Owner,10,not disappointed,Staff like it R1385,Hospitality,200-500,Area manager,3,somewhat disappointed,Staff like it R1386,Healthcare clinic,200-500,Office manager,12,very disappointed,Staff stop texting me at night R1387,Hospitality,10-50,Owner,7,very disappointed,Not sure yet R1388,Hospitality,50-200,Area manager,5,not disappointed,Staff like it R1389,Hospitality,200-500,General manager,9,somewhat disappointed,Easier than the spreadsheet R1390,Healthcare clinic,50-200,Office manager,2,very disappointed,Staff stop texting me at night R1391,Retail,10-50,Regional manager,1,not disappointed,Staff can see the rota R1392,Healthcare clinic,50-200,Practice manager,8,very disappointed,On-call swaps that follow our rules R1393,Retail,50-200,Owner,13,not disappointed,Easier than paper R1394,Healthcare clinic,200-500,Clinical lead,7,very disappointed,On-call swaps that follow our rules R1395,Retail,50-200,Store manager,2,not disappointed,Fine for now R1396,Retail,10-50,Regional manager,4,not disappointed,Fine for now R1397,Hospitality,10-50,Area manager,3,somewhat disappointed,Not sure yet R1398,Other,10-50,Operations manager,4,somewhat disappointed,Fine for now R1399,Retail,10-50,Owner,2,very disappointed,Shift swaps R1400,Retail,10-50,Store manager,2,somewhat disappointed,Staff can see the rota R1401,Healthcare clinic,200-500,Operations manager,10,very disappointed,"Rota done in an hour, not a day" R1402,Healthcare clinic,200-500,Practice manager,1,somewhat disappointed,On-call swaps that follow our rules R1403,Other,10-50,Operations manager,3,somewhat disappointed,Shift swaps R1404,Other,50-200,Team lead,6,very disappointed,Shift swaps R1405,Healthcare clinic,50-200,Practice manager,7,very disappointed,"Rota done in an hour, not a day" R1406,Retail,10-50,Store manager,14,not disappointed,Fine for now R1407,Healthcare clinic,200-500,Operations manager,4,very disappointed,Covering sick calls fast R1408,Healthcare clinic,50-200,Operations manager,8,somewhat disappointed,Covering sick calls fast R1409,Healthcare clinic,200-500,Operations manager,2,very disappointed,"Rota done in an hour, not a day" R1410,Retail,10-50,Owner,1,not disappointed,Shift swaps R1411,Retail,10-50,Owner,2,somewhat disappointed,Fine for now R1412,Hospitality,200-500,Head chef,4,somewhat disappointed,Easier than the spreadsheet
logo_retention_by_cohort.csv (paying accounts still active, by sign-up month and segment)cohort_month,segment,accounts_at_start,active_month_1,active_month_2,active_month_3,active_month_4,active_month_5,active_month_6,active_month_7,active_month_8,active_month_9 2025-10,Healthcare clinic,18,17,15,14,13,13,13,13,13,13 2025-10,Hospitality,23,20,17,15,13,12,11,10,9,8 2025-10,Retail,21,18,15,13,11,9,8,8,7,6 2025-11,Healthcare clinic,18,17,15,14,13,13,13,13,13,13 2025-11,Hospitality,23,20,17,15,13,12,11,10,9,8 2025-11,Retail,15,13,11,9,8,7,6,5,5,4 2025-12,Healthcare clinic,16,15,14,12,12,11,11,11,11,11 2025-12,Hospitality,25,22,19,17,14,13,12,11,10,9 2025-12,Retail,23,20,16,14,12,10,9,8,7,7 2026-01,Healthcare clinic,16,15,14,12,12,11,11,11,11, 2026-01,Hospitality,27,24,21,18,16,14,12,11,10, 2026-01,Retail,17,14,12,10,9,8,7,6,5, 2026-02,Healthcare clinic,14,13,12,11,10,10,10,10,, 2026-02,Hospitality,25,22,19,17,14,13,12,11,, 2026-02,Retail,21,18,15,13,11,9,8,8,, 2026-03,Healthcare clinic,15,14,13,12,11,11,11,,, 2026-03,Hospitality,27,24,21,18,16,14,12,,, 2026-03,Retail,22,19,16,13,11,10,9,,, 2026-04,Healthcare clinic,12,11,10,9,9,9,,,, 2026-04,Hospitality,30,26,23,20,17,15,,,, 2026-04,Retail,24,20,17,14,12,11,,,, 2026-05,Healthcare clinic,12,11,10,9,9,,,,, 2026-05,Hospitality,30,26,23,20,17,,,,, 2026-05,Retail,19,16,13,11,10,,,,, 2026-06,Healthcare clinic,14,13,12,11,,,,,, 2026-06,Hospitality,29,26,22,19,,,,,, 2026-06,Retail,15,13,11,9,,,,,,
signups_by_month.csv9 rows · Download
month,trial_signups,new_paying_accounts
2026-01,312,44
2026-02,298,41
2026-03,341,47
2026-04,356,49
2026-05,330,46
2026-06,372,52
…
q3_new_business.md10 lines · Download
# Q3 2026 new business (from the CRO)

New ARR in Q3: $412,000 across 61 new accounts.

Largest deals:
- Harbour Hotels Group (hospitality, 34 sites): $96,000 ARR. Closed after we built a custom integration with their payroll system (9 engineer-weeks). Not available to other customers.
- Pinewood Leisure (hospitality, 18 sites): $61,000 ARR. Closed after a custom export to their HR platform (6 engineer-weeks).
…
What a strong answer doesThe answer key the graders mark against

Says Relay has fit in one segment, not across the board. The survey is 31% 'very disappointed' overall (126 of 412), below the 40% bar, but 54% among healthcare clinics (64 of 118) against 20% in hospitality and 17% in retail. Clinic cohorts' logo retention flattens at about 71% from month 4, while hospitality and retail keep falling (around 35% and 29% by month 9), so the behavioural evidence agrees. August's tripling of trial sign-ups (1,140 against about 350) came from the Product Hunt launch and barely moved new paying accounts (55 against about 50), so it isn't evidence of fit. The two biggest Q3 deals (38% of new ARR) were won with custom engineering, which is services revenue, not fit, and the 14 groups asking for the same are a warning, not a pipeline. Recommends against tripling sales across all segments: scale only where retention flattens (clinics), with sales hires focused there, stop custom integrations, and work on what clinics value (on-call swaps, credential checks) while learning why hospitality and retail don't stick. Sets a test or threshold for revisiting.

Critical failures (cap the score at 40)
  • Recommends tripling sales across every segment as the CEO proposed
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT93Usable with a quick editSonnet 5.5API91Usable with a quick editOpus 5.5Claude84Usable with a quick editGPT-6 LunaAPI80Needs more workHaiku 5.5API77Needs more workGPT-6.1 SolAPI75Needs more workGemini 3.8 FlashAPI68Usable with a quick editGemini 3.5 Flash-LiteGemini63Needs more work
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit7 of 13 checks passed

The memo lacks the quantitative survey split by segment and does not set conditions for revisiting the scaling decision, so a PM would need to rework the evidence and add thresholds.

Output

MEMORANDUM

TO: Dana Kowalski, CEO FROM: Head of Product DATE: September 24, 2026 SUBJECT: Product-Market Fit Analysis & Q4 Recommendations (Leadership Meeting Prep)

---

Dana,

Ahead of Thursday’s leadership meeting, you asked for my view on tripling the sales team next quarter based on the assumption that we clearly have product-market fit (PMF).

Looking strictly at the data across our 1,120 paying accounts, the reality is more nuanced. We do not have broad PMF across all segments, but we have achieved strong, durable PMF in one specific wedge: Healthcare Clinics.

Tripling the sales team right now would pour expensive fuel onto a leaky bucket in Retail and Hospitality, while starving Healthcare—our only truly healthy segment—of the product enhancements it needs to scale.

Here is what the data tells us about who we are building for, and what we should do instead next quarter.

---

1. Do we have Product-Market Fit, and for whom?

No, not globally. Our overall retention and PMF survey data reveal a stark divide between Healthcare and our other segments.

  • Healthcare Clinics (Strong PMF):
  • Retention: Healthcare cohorts show exceptional long-term stability. Across 2025–2026 cohorts, accounts that survive the first 3 months flatline into rock-solid retention. For example, the Oct/Nov 2025 Healthcare cohorts retain 72% of their accounts at Month 9 (13 out of 18).
  • PMF Survey: Healthcare admins cite acute, high-stakes pain points that Relay solves: "Staff stop texting me at night," "On-call swaps that follow our rules," "Credential checks before a shift," and "Rota done in an hour, not a day."
  • Retail & Hospitality (Weak/Fading PMF):
  • Retention: The data is concerning. Retail cohorts bleed logos continuously month-over-month. The Oct 2025 Retail cohort dropped from 21 accounts to 6 at Month 9 (28% retention). Nov 2025 dropped to 26% retention.
  • PMF Survey & Behavior: In retail and hospitality, the primary benefits cited are generic ("Shift swaps," "Staff can see the rota," "Fine for now"). Furthermore, despite a massive marketing/top-of-funnel spike in August 2026 (1,140 trial signups up from ~350), new paying accounts barely budged (55 in Aug, 47 in Sept). The market is looking at us, but they aren't buying or staying because "nice app" and "shift swaps" are table stakes, not pain-killers.

---

2. The Danger of the "Enterprise Hospitality" Trap

The CRO’s Q3 report highlights $412k in new ARR, driven largely by massive hospitality groups (Harbour Hotels: $96k ARR; Pinewood Leisure: $61k ARR). On paper, this looks like explosive enterprise traction.

Look closer: Both deals required heavy engineering customisation (9 engineer-weeks for Harbour's payroll integration; 6 weeks for Pinewood's HR export) that is not available to other customers. Worse, 14 other hospitality groups are now demanding that exact custom integration.

If we triple the sales team now, sales will chase more logo-logo hospitality groups demanding bespoke engineering. We will inadvertently turn Relay into a custom IT shop for hotels, pulling engineering away from core product growth.

---

3. What we should do in Q4 instead of (or alongside) tripling sales

Instead of a blind 3x sales expansion, Q4 should be a targeted pivot toward operational efficiency, product standardisation, and doubling down on Healthcare. I recommend four concrete shifts:

#### A. Moderate the Sales Expansion; Retrain on ICP (Ideal Customer Profile) Do not triple the sales team blindly. Instead, cap sales headcount growth at a controlled 50% increase, and redirect sales incentives away from chasing logo-heavy hospitality groups that require custom engineering. Focus inbound and outbound motion strictly on Healthcare Clinics and mid-market operations where standard features win deals.

#### B. Productise the Custom Integrations (or Walk Away) The pipeline of 14 hospitality groups wanting Harbour-style integrations is a flashing warning light. Product and Engineering must evaluate whether payroll and HR integrations can be built as a standard, self-serve module available to all tiers. If the market requires custom engineering to close, we must reject those deals rather than compromise our product roadmap.

#### C. Double Down on Healthcare Product Capabilities Healthcare users are telling us precisely what keeps them loyal (credential checks, rule-abiding on-call swaps, eliminating night texts). In Q4, Engineering should lean entirely into hardening Healthcare-specific workflows. If we make Relay the undisputed operating system for healthcare clinic scheduling, we build a defensible moat with predictable, high-retention revenue.

#### D. Fix Retail/Hospitality Through Self-Serve, Not Sales For Retail and Hospitality, the churn data proves that throwing human sales reps at the problem won’t fix it. The product needs to solve retention via self-serve onboarding and better workflow automation. We should keep acquisition lean here and let product-led growth (PLG) iterate on the feedback rather than burning capital on enterprise sales reps.

---

Summary: We have clear PMF in Healthcare Clinics, and perilous, high-churn, high-customization noise in Hospitality and Retail. Let’s protect our engineering bandwidth, fix our ICP, and scale sales intelligently rather than rushing into a costly expansion.

I look forward to discussing this further on Thursday.

Grades and run details

Decision model 73 · LLM judge 57
Decision model checks
  • failUses the supplied evidence correctly3%
  • partialAddresses the actual decision33%
  • passRespects explicit constraints45%
  • partialIdentifies material uncertainty44%
  • partialAvoids unsupported claims44%
  • passProduces the required deliverable59%
  • partialFinds the segment the product fits61%
  • passDiscounts the sign-up spike100%
  • passJudges fit by retention, not sign-ups72%
  • passGets the base of every number right24%
  • passFinds who it's a must-have for96%
  • passSpots false fit94%
  • partialDoesn't scale before fit84%
Run
Run
#1
Time to output
7 s
Submitted
5 Oct 2026
Opus 5.5Claude

Usable with a quick edit10 of 13 checks passed

The memo is well-structured, data-driven, and actionable. A quick edit to note the 40% benchmark as an external standard and to soften the pipeline assumption would make it fully ready.

Output

MEMO

To: Dana Kowalski From: Head of Product Re: Product-market fit, and what to do next quarter

---

Bottom line

We have product-market fit in healthcare clinics. We do not have it in hospitality or retail. Those two segments bring in most of our new accounts and both of our biggest Q3 deals. If we triple sales across the board, we will mostly buy customers who leave within a year. I recommend a smaller sales expansion focused on healthcare, plus product and data work, with clear gates for scaling further.

---

1. Do we have PMF? By segment, no and yes

The September survey reached 412 of 1,120 paying admins (37%). The usual benchmark for fit is 40% of users saying they'd be "very disappointed" without the product.

SegmentSurvey nVery disappointedLogo retention, month 6Month 9
Healthcare clinic11854%72%71%
Hospitality14020%47%35%
Retail10417%41%29%
Other5032%n/an/a
All41231%

Month 6 pools the Oct–Apr cohorts; month 9 pools Oct–Dec.

Three things stand out:

  • Healthcare retention flattens; the others don't. The October 2025 healthcare cohort lost 5 of 18 accounts by month 4 and none since. Hospitality and retail cohorts are still losing roughly one account a month at month 9, with no floor in sight.
  • Healthcare customers name a job we do for them. Their main benefits are "on-call swaps that follow our rules", "covering sick calls fast", "credential checks before a shift", "rota done in an hour, not a day" and "staff stop texting me at night". Hospitality and retail answers are "Nice app", "Staff like it", "Fine for now" and "Not sure yet". That is satisfaction, not dependence.
  • The survey flatters the weak segments. It only went to current payers, so churned accounts aren't counted. Even the hospitality and retail customers who stayed mostly wouldn't miss us. Large hospitality (200–500 staff) is the weakest group: only about 1 in 8 is very disappointed.

"Other" scores 32%, but the sample is small and the benefits are vague ("Saves time", "Fine for now"). I wouldn't act on it.

---

2. What the growth numbers actually say

  • New paying accounts are flat, at 41–55 a month all year.
  • The August signup spike was not buyers. 1,140 trials converted at 4.8%, against a steady 14% every other month. September converted at 9%.
  • The mix is drifting the wrong way.
  • Healthcare cohorts have shrunk from 18 accounts to 12–14.
  • Hospitality cohorts have grown from 23 to about 30.
  • 38% of Q3 new ARR came from two custom deals.
  • Harbour and Pinewood brought in $157k of the $412k.
  • Both are hospitality, and both required custom builds (15 engineer-weeks in total) that no other customer can use.
  • The repeatable business is the other 59 deals: about $255k, averaging roughly $4,300 each.
  • The pipeline is mostly bespoke work. The 14 groups asking for "the Harbour integration" will each have their own payroll system. At 6–9 engineer-weeks apiece, that is 84–126 engineer-weeks of custom work for the segment with the weakest fit.

Tripling sales now would mean:

  • selling mainly to segments where about half of accounts are gone by month 6;
  • pulling engineering into one-off integrations;
  • under-investing in the one segment where the product clearly works.

---

3. What I recommend for next quarter

a) Expand sales, but aim it at clinics. - Hire a small dedicated healthcare team (2–3 reps) selling the product as it is. - Agree targets with the CRO now: bookings, sales cycle, and month-1 retention. - If the team hits those targets, we scale hard in the following quarter. Then we'd be tripling into a segment that retains 70%+.

b) Point product and marketing at the healthcare job. - Deepen the features clinics already name: on-call rules, sick-call cover, credential checks. - Build clinic-specific onboarding and positioning. - Produce case studies from our long-tenured clinics.

c) No more bespoke integrations. - Pause the 14 requests until we know which payroll systems they use. - If 3–4 systems cover most of them, build one productised integration available to every customer and price it as an add-on. - First, check how Harbour and Pinewood are actually using Relay. A $96k logo that behaves like our other hospitality accounts is a churn risk, not a template.

d) Diagnose hospitality and retail before buying more of it. - Interview about 20 churned accounts and about 20 "very disappointed" ones. - Look for a sub-segment that genuinely depends on us, and for what drives the month 1–4 drop-off. - Hold paid acquisition aimed at these segments until we have answers.

e) Fix the numbers before Thursday. The files don't reconcile: - Monthly new accounts: the retention file shows 58–66 new accounts a month for January–June, while the signups file shows 41–52. - Q3 new accounts: the CRO counts 61, against 152 in the signups file. That may simply be sales-led versus self-serve, but we should know. - Revenue retention: we have no revenue retention by segment. With per-location pricing, expansion revenue could change the picture, in either direction. - The August spike: we should find out what caused it.

---

The decision I'm asking for

  1. Approve a healthcare-focused sales pilot instead of tripling now.
  2. Pause bespoke integration work.
  3. Commit to revisiting the tripling question at the end of next quarter, against the agreed gates.

That keeps the ambition and points it where the evidence supports it.

Claims the judge couldn’t find in the brief

  • The usual benchmark for product-market fit is 40% of users saying they'd be 'very disappointed' without the product.
  • The pipeline is mostly bespoke work; each group will have its own payroll system, requiring 84–126 engineer-weeks of custom work.

Grades and run details

Decision model 81 · LLM judge 86
Decision model checks
  • failUses the supplied evidence correctly44%
  • passAddresses the actual decision98%
  • passRespects explicit constraints41%
  • passIdentifies material uncertainty91%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable68%
  • passFinds the segment the product fits97%
  • passDiscounts the sign-up spike99%
  • passJudges fit by retention, not sign-ups88%
  • failGets the base of every number right55%
  • passFinds who it's a must-have for100%
  • passSpots false fit96%
  • passDoesn't scale before fit97%
Run
Run
#1
Time to output
1.7 min
Submitted
5 Oct 2026
Sonnet 5.5API

Usable with a quick edit11 of 13 checks passed

The memo is data-driven, well-structured, and directly answers the CEO's question with a clear call and actionable next steps. A product manager could use it with minor wording tweaks and no re-checking of evidence.

Output

To: Dana Kowalski From: Head of Product Re: Product-market fit and next quarter's plan

Bottom line

We have product-market fit in one segment, healthcare clinics, and not yet anywhere else. Tripling sales across the board would mostly buy more hospitality and retail accounts that churn. I recommend a targeted healthcare push plus fixes to the rest, not a general tripling.

Do we have PMF?

The standard test is whether at least 40% of users would be "very disappointed" to lose the product. Across all 412 survey responses, 30.6% (126) said so, which is below the bar. By segment:

SegmentResponsesVery disappointed6-month logo retention*
Healthcare clinic11854%71%
Other5032%not tracked
Hospitality14020%47%
Retail10417%40%

*Cohorts Oct 2025 to Mar 2026.

The retention curves tell the same story:

  • Healthcare drops to about 77% by month 3, then goes flat. Most cohorts hold their numbers from month 4 on (e.g., Oct-25 sits at 13 of 18 for six straight months). A curve that flattens is what PMF looks like.
  • Hospitality and retail never flatten. The Oct-25 cohorts are at 35% and 29% by month 9 and still falling.

The reasons people give differ too. Healthcare users name specific, hard-to-replace workflows: on-call swaps that follow their rules, credential checks, fast sick-call cover, and "rota done in an hour, not a day". Hospitality and retail answers are mostly generic: "Nice app", "Staff like it", "Fine for now", "Not sure yet". Eight of 28 "very disappointed" hospitality respondents said "Not sure yet". Healthcare's strong scores also aren't just long-tenured survivors, because they appear at one to three months of use as well as at 12 to 14.

"Other" (32%, n=50) is a mixed bag. Six of its 16 "very disappointed" answers are "Fine for now", so I wouldn't read it as a second wedge.

Why tripling sales is the wrong move

  1. We're feeding the wrong segments. Healthcare is our best segment but our smallest and shrinking source of new accounts: 18 a month in the Oct/Nov-25 cohorts, 12–14 in the last four. Hospitality has grown from 23 to about 30. More sales capacity without direction amplifies that mix.
  2. Q3 revenue is less repeatable than it looks. Two hospitality deals, Harbour ($96k) and Pinewood ($61k), are 38% of the $412k new ARR. Both closed only after 15 engineer-weeks of custom work. The other 59 deals average about $4.3k, and none of the big deals came from healthcare, our proven segment.
  3. Top-of-funnel growth isn't accelerating. New paying accounts have been flat at 41–55 a month all year. August trials spiked to 1,140, but only 55 converted (4.8% versus about 14% normally). That spike looks like low-intent traffic, not demand. I don't yet know what caused it.

What to do next quarter

  1. Healthcare push (the "as well as"). Fund a dedicated healthcare sales pod of 2–3 reps from the tripling budget, with marketing aimed at clinics of 50–500 staff. Release further hiring only if healthcare new accounts reach about 20 a month and month-3 retention on the new cohorts stays at 75% or better.
  2. Healthcare product depth. Put the roadmap behind what healthcare users already value: rules-aware on-call swaps, credential checks and sick-call cover.
  3. Hospitality and retail: diagnose before scaling. Hold their headcount and spend flat. Product and CS should run about 20 churned-account interviews and look for a sub-segment that sticks, such as multi-site groups with payroll needs.
  4. The 14 hospitality groups asking for "the Harbour integration". This could be a second wedge, but it's untested. Freeze bespoke builds. Qualify the 14 and build one generic payroll/HR integration only if at least five commit with pricing. I also want to see whether Harbour-type accounts retain before we bet on them.
  5. Understand August. Marketing should tell us the source of the 1,140 trials and why they didn't convert.

What the data can't tell us, and where it looks wrong

  • Survey bias. Only 37% of admins responded (412 of 1,120), and churned accounts weren't surveyed. That likely flatters hospitality and retail. Healthcare may be over-represented among responders.
  • Cohort file doesn't match signups. Cohort starting counts run 10–20 per month higher than new paying accounts in signupsbymonth (e.g., 60 versus 44 in January). Several cohorts are also identical row for row (Oct-25 and Nov-25 healthcare and hospitality, for example). Before Thursday I'd like Finance to confirm against raw billing data. I'd expect the direction to hold, since it matches the survey, but I wouldn't quote exact retention figures externally yet.
  • Q3 account counts don't match. The CRO reports 61 new accounts for Q3; signups shows 152 new paying accounts (50+55+47). This is probably sales-led versus self-serve, but it needs confirming.
  • We lack revenue retention and any retention data for "Other".

Ask for Thursday

Agree that the claim is "PMF in healthcare clinics", not "PMF". Approve the healthcare pod and the gated hiring plan instead of tripling. Pause bespoke integrations pending the qualification in point 4.

Grades and run details

Decision model 81 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly43%
  • passAddresses the actual decision99%
  • passRespects explicit constraints39%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims24%
  • passProduces the required deliverable79%
  • passFinds the segment the product fits94%
  • passDiscounts the sign-up spike100%
  • passJudges fit by retention, not sign-ups75%
  • failGets the base of every number right42%
  • passFinds who it's a must-have for100%
  • passSpots false fit95%
  • passDoesn't scale before fit98%
Run
Run
#1
API response time
1.8 min
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedWrongMixed
Gemini 3.5 Flash-Lite · Gemini

All factual claims about the current situation are directly supported by the supplied data or arithmetic.

Opus 5.5 · Claude

The 40% 'very disappointed' benchmark and the claim that the pipeline is bespoke work requiring custom integrations are not in the supplied context and are not arithmetic derivations.

Sonnet 5.5 · API

All factual claims about the current situation are drawn directly from the supplied data or arithmetic on it, with no invented facts.

Addresses the actual decisionWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo does not state what result or condition would change the call against tripling sales, as required.

Opus 5.5 · Claude

The memo commits to a clear answer (PMF in healthcare only, no tripling across the board) and sets gates for revisiting the decision.

Sonnet 5.5 · API

The memo commits early to 'PMF in healthcare clinics, not yet anywhere else', recommends against tripling sales, and sets a condition (healthcare new accounts reach ~20/month with 75%+ month-3 retention) to widen hiring.

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

No specific unknowns that could change the decision are named, nor how they would be resolved.

Opus 5.5 · Claude

It names specific unknowns (data discrepancies, lack of revenue retention, cause of August spike, hospitality/retail churn drivers) and says how to resolve them.

Sonnet 5.5 · API

It names specific unknowns (survey bias, cohort vs signup mismatch, Q3 account count discrepancy, missing revenue retention, August spike source) and says how to resolve them (Finance confirmation, marketing investigation).

Avoids unsupported claimsWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

The causal claim that users aren't buying/staying 'because "nice app" and "shift swaps" are table stakes, not pain-killers' is presented as fact without labelling it as a hypothesis.

Opus 5.5 · Claude

The 40% benchmark and the bespoke pipeline forecast are presented as established facts without being labelled as assumptions or hypotheses.

Sonnet 5.5 · API

Interpretations like the August spike being 'low-intent traffic' are presented as hypotheses, not established fact, and the memo avoids overclaiming.

Finds the segment the product fitsWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo does not compute the segment split of 'very disappointed' responses (e.g., 54% in healthcare) from the survey file, using only qualitative quotes and retention.

Opus 5.5 · Claude

The memo breaks down survey and retention by segment, finds clinics at 54% very disappointed with flattening retention, and correctly notes the blended 31% hides this.

Sonnet 5.5 · API

It breaks the survey down by segment (54% very disappointed in healthcare vs 20% and 17%) and uses retention curves to show healthcare flattens while others don't, correctly identifying clinics as the fit segment.

Gets the base of every number rightRightMixedMixed
Gemini 3.5 Flash-Lite · Gemini

All derived percentages (72%, 28%, 26%) are computed correctly from the retention data with the base stated.

Opus 5.5 · Claude

All derived percentages (54%, 20%, 17%, 72%, 47%, 41%, 38%, 4.8%, etc.) check out against the supplied data with correct denominators.

Sonnet 5.5 · API

All percentages and rates (30.6%, 54%, 71%, 47%, 40%, 38%, $4.3k, 4.8%) are computed correctly from the supplied data, and bases are stated where needed.

Doesn't scale before fitWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

It does not set the evidence that would justify scaling sales in retail/hospitality later, only recommending self-serve for now.

Opus 5.5 · Claude

It recommends scaling sales only in healthcare with a pilot, not across all segments, and sets clear evidence gates for widening later.

Sonnet 5.5 · API

It recommends scaling only in healthcare, with gated hiring, and sets the evidence (20 new accounts/month, 75%+ month-3 retention) that would justify widening to other segments.

All got right 6

Respects explicit constraintsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo is addressed to Dana, is under 900 words, and respects the requested form.

Opus 5.5 · Claude

The output is a memo to the CEO, under 900 words, and respects the requested form and length.

Sonnet 5.5 · API

The output is a memo to the CEO, under 900 words, and the proposed actions (gated hiring, freeze bespoke builds) would enforce the constraints in practice.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form, and a product manager could act on it with light edits.

Opus 5.5 · Claude

The memo is complete, in the right form, within the word limit, and a product manager could act on it with light edits.

Sonnet 5.5 · API

The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.

Discounts the sign-up spikeRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It compares August's tripled trial sign-ups with flat new paying accounts and treats the spike as not evidence of fit.

Opus 5.5 · Claude

It compares August's 1,140 trials to the flat 55 new paying accounts and treats the spike as a launch effect, not evidence of fit.

Sonnet 5.5 · API

It notes the August trial spike (1,140) barely moved new paying accounts (55) and treats it as a launch effect, not evidence of fit.

Judges fit by retention, not sign-upsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The call on PMF rests on whether retention curves flatten by segment, with the survey as supporting evidence.

Opus 5.5 · Claude

The PMF call rests on retention curves flattening for clinics versus continuing decline for hospitality and retail, with the survey as supporting evidence.

Sonnet 5.5 · API

The PMF call rests on whether retention curves flatten by segment, with the survey as supporting evidence, not on sign-ups or revenue alone.

Finds who it's a must-have forRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It names healthcare clinics as the must-have segment, lists what they value, and builds the plan around them.

Opus 5.5 · Claude

It profiles healthcare clinics as the must-have segment, lists their specific valued features, and builds the plan around them.

Sonnet 5.5 · API

It profiles healthcare clinics as the must-have segment, naming the specific workflows they value, and builds the plan around them.

Spots false fitRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It identifies the custom-engineering hospitality deals and the sign-up spike as flattering but not true fit.

Opus 5.5 · Claude

It identifies the August spike, the custom deals, and the bespoke pipeline as false signals of fit and explains why each is not real PMF.

Sonnet 5.5 · API

It identifies the two large custom-work hospitality deals, the August spike, and the 14 groups asking for the same integration as false signals of fit, explaining why each is not real PMF.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT90.5100.02None
2Sonnet 5.5API77.990.02None
3Opus 5.5Claude79.782.92None
4Haiku 5.5API77.986.02None
5GPT-6.1 SolAPI90.575.72None
6GPT-6 LunaAPI92.772.42None
7Gemini 3.8 FlashAPI65.062.12None
8Gemini 3.5 Flash-LiteGemini65.151.92None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 96, 118× the cheapestSonnet 5.5 · API: task score 85, 81× the cheapestOpus 5.5 · Claude: task score 82, 118× the cheapestHaiku 5.5 · API: task score 81, 6.1× the cheapestGPT-6.1 Sol · API: task score 80, 25× the cheapestGPT-6 Luna · API: task score 79, the cheapestGemini 3.8 Flash · API: task score 63, 16× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 56, 2.8× the cheapestGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether you have product-market fit, and for whom.

Why it matters

Scaling before fit amplifies waste. Most teams read fit from sign-ups or a blended survey, and miss the one segment where it's real.

What good looks like

  • Judges fit by retention, not sign-ups
  • Breaks the survey down by segment and tenure
  • Spots revenue and metrics that flatter
  • Scales only where fit is proven

Deliberately not measured

  • Fundraising strategy beyond what the evidence supports
Capability tested

Reading fit from evidence

The failure we’re looking for

Calling fit from a blended number or a growth spike

Grading

Decision model and LLM judge, calibrated against a blind PM review