Experiment specification

Experiment spec: Order batching before the peak-season freeze

Decision owner: COO · Sign-off: COO, CFO, Head of Operations, Courier Relations Lead · Copied: Head of Growth Status: Draft for sign-off, 30 September 2026

1. Decision

Should we switch batching on in all 14 cities before the code freeze on 6 November 2026? The switch works per city, so the result can also be "on in some cities."

2. What changed from the draft plan, and why

3. Timeline

The dates are fixed by two constraints. The Porto notice cannot slip past today without losing Porto days. The test cannot run past 28 October without eating into the rollout buffer.

4. Design

• Unit: one city in one 2-hour window. Each day has six windows (11–13, 13–15, 15–17, 17–19, 19–21, 21–23).
• Assignment: randomised 50/50 within each city and time slot. Each city gets a balanced mix of on and off across lunch, dinner, weekdays and weekends.
• Switch behaviour: new batches form only while the switch is on. Batches already in progress when it turns off complete normally.
• Washout: the first 30 minutes of each window are excluded from analysis. This leaves 90 analysed minutes per window.
• Sample: 13 cities × 6 windows × 21 days plus Porto × 6 × 15 days gives 1,728 windows. That is about 2.3× the analyst's 740. The extra covers the washout and the correlation between windows. Analytics to confirm the power calculation using the correlation seen in the first week's data. If power is short, we extend to 30 October and give up one buffer day.
• Analysis: compare window-level metrics with adjustments for city, day and time slot. Treat all windows in the same city on the same day as one group when calculating uncertainty. Report the network result and each city's result.

5. Metrics and decision rule

Rollout rule: a city rolls out on 3 Nov if the network result passes the primary metric and both guardrails, and that city's own result shows no clear guardrail failure.

The 1.0pp and €0.40 margins are proposals. Signers must confirm or change them before 8 October. They cannot be changed after the test starts.

6. Tensions signers should know about

What "at least 5%" can mean. We can reliably detect whether batching cuts cost by 5% versus not at all. We cannot prove the true effect is above 5% if it is only 6%, which is the low end of the simulation. The decision rule therefore uses the point estimate. If the estimate lands between 4% and 6%, the COO decides with that uncertainty in view.

Cost savings and courier earnings pull in opposite directions. Couriers are paid per order plus distance, so a lower cost per order is lower pay per order. Earnings per hour only hold up if couriers complete more orders per hour.

• During the test, the same couriers are logged in whether batching is on or off. That makes the test a fair early read on hourly earnings.
• After rollout, the saving only stays a saving if we need fewer courier hours. Fewer hours affects couriers' income, even if the hourly rate is protected.
• The CFO and Courier Relations Lead should agree on this before sign-off, not after results.

Porto. "Must not reduce average hourly earnings" is a contractual condition, not a statistical margin. Porto gives us only ~90 windows, too few to prove "no reduction."

• If network earnings per hour fall at all, Porto stays off unless it comes with an earnings protection mechanism. One example is a per-batch supplement, which legal and Courier Relations should scope now.
• If Porto rolls out, we measure its hourly earnings against the pre-batching baseline every week.

"Noticeably longer." We are defining this as the >45-minute late share plus the complaint rate. We are not using average delivery time, because a 3–6 minute delay on second orders is expected by design.

7. Stop rules (checked daily by the on-call analyst)

• City level: the >45-minute late share in "on" windows exceeds 10% for two consecutive days. Batching goes off in that city pending review.
• Network level:
• Any courier safety incident linked to batching, or a formal complaint from a couriers' association: pause everywhere.
• Earnings per hour in "on" windows below €12.50 for three consecutive days: pause everywhere.
• Porto only: any clear drop in hourly earnings leads to an immediate pause and notification to Courier Relations.

8. What each signer is agreeing to

• COO: the dates in §3, the rollout rule in §5, and making the call if the result falls between 4% and 6%.
• CFO: cost per order as the primary metric, with the ≥5% rule. Also acknowledges that post-rollout savings depend on how many courier hours we buy (§6).
• Head of Operations: the 1.0pp late-share margin and the stop rules.
• Courier Relations Lead:
• the €0.40 margin;
• the separate rule for Porto;
• sending the Porto notice today.

9. Open items

1. Legal, today: Porto notice wording, and whether one notice covers both the test and the rollout. 2. Analytics, by 7 Oct: - power calculation that accounts for correlated windows; - randomisation schedule; - dashboard. 3. Engineering, by 7 Oct: per-city switch with switch times logged, tested in one city after hours. 4. Courier Relations and Legal, by 2 Nov: Porto earnings protection option. 5. Signers, by 8 Oct: confirm the guardrail margins in §5.

Brisk order-batching experiment spec

Decision required: Whether to enable batching in all 14 cities on 5 November 2026, ahead of the 6 November code freeze. Batching pairs two eligible orders from nearby restaurants for one courier.

Decision principles and sign-off

Launch only if the experiment shows a worthwhile courier-cost reduction, customer lateness stays within an agreed limit, and courier earnings protections are met. An inconclusive result is not a pass.

Before launch, the COO and Head of Operations must approve the customer-delay limit below; the courier relations lead and Legal must confirm the Porto protocol in writing. If those approvals or the schedule cannot be met, do not claim evidence for an all-city rollout before the freeze.

Design

Randomise city × two-hour operating windows, not individual orders. Orders interact through shared courier supply, and individual randomisation would contaminate the comparison. In each treatment window, the batching policy is enabled; in each control window, batching is disabled. Randomise and balance windows within each city, weekday and daypart, aiming for an even treatment/control split. Keep the assignment schedule fixed before the test begins.

An order is analysed according to its city-window at placement (intention to treat). Batch only orders placed in the same treatment window; do not pair across windows or with control orders. This limits cross-arm contamination at switchovers. Record actual batching eligibility and use, but do not reclassify orders based on whether they were batched.

Timeline and Porto condition

• 30 September: Send Porto’s couriers’ association written notice. The notice requirement applies to the experiment as well as any rollout. Start Porto treatment no earlier than 15 October, allowing 14 full days.
• By 7 October: Complete the per-city switch, monitoring, and rollback controls; test the assignment and cost calculations.
• 15 October–4 November: Run the experiment in all cities, subject to Porto approval.
• 5 November: Review results and, only if all launch gates pass, enable batching in all 14 cities in one day. Otherwise keep it off. Do not shorten the experiment to meet the freeze.

Porto must not enter treatment until Legal and the courier relations lead confirm the notice and trial protocol comply with the agreement. Agree a written earnings-protection mechanism with the association before starting; any top-ups count in courier cost. If this cannot be agreed, exclude Porto from treatment and do not describe the result as support for a 14-city rollout.

Measures and launch gates

Estimate treatment effects against control, with confidence intervals and pre-specified analysis. Baselines: courier cost €7.40/order, orders over 45 minutes 7.5%, courier earnings €13.20/active hour.

The proposed 0.5pp lateness margin and the earnings non-inferiority margin are policy choices, not facts in the brief; the accountable leads must sign them off before randomisation. Do not change margins after seeing results.

Analysis and power

The schedule contains 1,764 city-windows (14 cities × 6 windows/day × 21 days), before any exclusions. The analyst’s estimate of 740 assumes independent windows; successive windows may be correlated, and couriers and demand carry across time. Before the test, use historical city-window data and the planned randomisation to simulate power, accounting for serial correlation and city/daypart differences. Use blocked randomisation-based inference or an appropriate model with uncertainty at the randomised-window level. Do not claim the test is powered merely because 1,764 exceeds 740. If adjusted power is inadequate, or results are too imprecise to meet a gate by 5 November, the decision is inconclusive: no rollout before freeze.

Operations and rollback

Monitor treatment uptake, cost, lateness and earnings daily, but do not stop early for a favourable result. Pause a city for a safety, service or earnings breach under pre-agreed operational thresholds; document the incident and retain it in intention-to-treat analysis. Keep the city switch available for immediate rollback. Sign-off: COO, CFO, Head of Operations, courier relations lead; Growth acknowledges the delayed reorder readout.

Order Batching Experiment Spec

Decision: Roll out batching to all 14 cities before the 6 Nov 2026 code freeze, or not. Owner: Staff PM. Sign-off: COO, CFO, Head of Operations, Courier Relations Lead. Head of Growth is copied in.

1. What changed from the intern's draft

2. Design

• Unit: city × two-hour window. The windows are 11–13, 13–15, 15–17, 17–19, 19–21 and 21–23, so six per city per day.
• Assignment: Within each city-day, the windows form three adjacent pairs. In each pair, one window is randomly batching-on and the other is off. This balances lunch, afternoon and dinner across arms and controls for city and weekday.
• Attribution: An order belongs to the window in which it was placed. The pre-specified sensitivity analysis drops orders placed in the first 30 minutes of each window, because in-flight batches and courier positioning carry over from the previous window.
• Exposure: Batching pairs only nearby restaurants, using the same rules the simulation assumed. The batching configuration is frozen before the test starts.

3. Duration and power

The analyst's figure of about 740 windows assumes windows are independent. They are not: adjacent windows share couriers and demand, and cities differ. I therefore treat 740 as a floor and plan for up to double that.

• Engineering needs one week, so the switch is ready on 7 Oct and the test runs Thu 8 Oct – Wed 28 Oct (21 days, three of every weekday).
• 13 cities × 6 windows × 21 days = 1,638 windows. Porto adds 6 × 14 days = 84. The total is about 1,720 windows, roughly 2.3× the floor.
• We do not stop early on a good-looking interim result. The end date is fixed.
• Analysis: A window-level regression on total cost ÷ total orders, with city and weekday × hour fixed effects and standard errors clustered by city-day. We will confirm with randomisation inference.

4. Metrics and decision rules

All four sign-offs are needed on these thresholds by Tue 6 Oct, before any data exist.

The COO's "without making customers wait noticeably longer" is operationalised as the +1.0 pp cap. The simulations add 3–6 minutes to the second order in each batch, so this cap is the threshold most likely to be contested. The Head of Ops should confirm or change it before the start.

Tension to surface: Cost per order is forecast to fall 6–11%, and couriers are paid per order plus distance. Earnings per hour therefore stay flat only if batching raises orders per courier-hour by at least as much as pay per order falls. The test settles this, and the CFO's and Courier Relations' metrics are not independent.

Secondary metrics (descriptive only): - Mean delivery time, and delivery time for the second order in a batch. - Share of orders batched. - Orders per courier-hour. - Customer support contacts. - Cancellations.

Reorder rate: We report the 7-day rate for test customers at the decision. The 30-day rate is tracked after rollout for customers by batched-order exposure, and it cannot be treated as causal. Growth should know this now.

5. Porto

• Written notice goes to the couriers' association today (30 Sept). Legal should word it to cover both the test and permanent rollout. The earliest compliant start is 14 Oct, so Porto joins the test on 15 Oct.
• The contract says changes must not reduce average hourly earnings. Porto therefore has a stricter rule than other cities: the estimated earnings difference must be ≥ €0 for Porto to ship.
• Porto has only about 84 windows, so its own estimate will be noisy. We pool it for cost and lateness. For earnings we report Porto separately, daily, with the stop rule below.
• If Porto fails or is ambiguous, we ship in the other 13 cities and keep Porto off. Legal should confirm that running the test in Porto does not itself breach the agreement. If it might, exclude Porto from the test and roll out there only on the pooled evidence plus the earnings rule.

6. Stop rules

The per-city switch lets us turn batching off immediately. Any of these triggers a pause in that city and a review within 24 hours:

• The >45-minute share in batching-on windows exceeds off windows by more than 3 pp over two consecutive days.
• Earnings per active hour in batching-on windows are more than 5% below off windows over three consecutive days. In Porto, the threshold is any sustained shortfall that the Courier Relations Lead judges real.
• Any safety or serious courier-welfare incident tied to batching.

7. Timeline

The 29 Oct decision leaves a week of buffer. If the test slips past about 2 Nov, there is no safe rollout before the freeze.

8. Risks and limits

• Seasonality: October demand is below peak. Higher order density should raise batch rates, but it also strains courier supply. We track batch rate by hour and expect the peak effect to differ from the test estimate.
• Carryover: Some carryover between windows will remain even with the 30-minute washout. The sensitivity analysis shows how much it matters.
• Short-run effects: Couriers may change their behaviour as they learn batching, and three weeks may not capture that. We monitor weekly trends within the test.
• Possible outcomes: The result can be a full rollout, a partial rollout (for example excluding Porto or any city that fails a guardrail), or no rollout. Partial rollout by city is allowed by the switch, but only for cities that individually pass the stop rules. City-level results will be noisy, so we do not make ship decisions city by city beyond this.

9. Sign-off