Experiment specification

Experiment Spec: Order Batching Rollout Evaluation

Author: Staff Product Manager, Core Logistics Target Decision Date: 4 November 2026 (Rollout: 5 November; Peak Code Freeze: 6 November) Stakeholders / Sign-offs: COO, CFO, Head of Operations, Courier Relations Lead Informed: Head of Growth

---

1. Executive Summary & Decision Framework

We are evaluating courier order batching (one courier delivering two proximate orders) across our 14 European markets (~1.9M monthly orders). Our goal is to determine whether to ship batching globally prior to the 6 November code freeze.

Go / No-Go Decision Criteria (Sign-off Contract)

Batching will roll out to all 14 cities on 5 November only if all four conditions are met: 1. Cost Efficiency (CFO & COO): Courier cost per order decreases by $\ge$ 5.0% (baseline: €7.40; target: $\le$ €7.03) at $p < 0.05$. 2. Customer Experience Guardrail (COO & Head of Ops): Share of orders delivered in >45 minutes increases by no more than 0.8 percentage points (baseline: 7.5%; ceiling: $\le$ 8.3%) with no statistically significant drop in order ratings. 3. Courier Welfare & Legal Compliance (Courier Relations & Porto Agreement): Courier earnings per active hour do not decline in any market (baseline: €13.20; target: $\ge$ €13.20, non-inferiority margin of 0%). 4. Leading Retention Indicator (Head of Growth): 7-day reorder rate exhibits no statistically significant negative degradation ($\alpha = 0.05$).

---

2. Flaws in the Intern’s Draft & Required Fixes

---

3. Experiment Design & Methodology

Design: Stratified City-Time Switchback

• Unit of Randomization: City $\times$ 2-hour window (11:00–13:00, 13:00–15:00, ..., 21:00–23:00 = 6 windows/day per city).
• Intervention:
• Treatment Window: Batching algorithm active (up to 2 orders per courier from nearby pickup points).
• Control Window: Pure single-order dispatch (status quo).
• Buffer/Washout: 15-minute dispatch transition buffer between blocks. Orders placed in the final 15 minutes of a window cannot be batched if the subsequent window is Control, eliminating spillover across windows.

Power & Sample Size

• Required Sample: 740 two-hour windows to detect a 5% shift in cost per order ($\alpha = 0.05, \beta = 0.80$).
• Capacity:
• 13 cities $\times$ 6 windows/day = 78 windows/day.
• Porto joins on Day 8 (15 October): 14 cities $\times$ 6 windows/day = 84 windows/day.
• Duration: 14 days (8 October – 21 October).
• 7 days $\times$ 78 windows + 7 days $\times$ 84 windows = 1,134 windows.
• This provides >150% of required sample, allowing high statistical power to detect sub-1% changes in courier hourly earnings and Ops tail-delays.

---

4. Stakeholder Metrics & Success Criteria

``` +---------------------------------------------------------------------------------------+ | PRIMARY METRIC | | Courier Cost per Order (CFO/COO) | Target: >= 5% reduction (<= €7.03 vs. €7.40) | +---------------------------------------------------------------------------------------+ | +---------------------------------+---------------------------------+ | | | +------------------+ +-------------------+ +--------------------+ | OPS GUARDRAIL | | COURIER GUARDRAIL | | GROWTH GUARDRAILS | | (Head of Ops) | | (Courier Lead) | | (Head of Growth) | | >45m Late Share | | Courier Earnings | | • 7-day Reorder | | Baseline: 7.5% | | per Active Hour | | • Order Rating | | Ceiling: <= 8.3% | | Baseline: €13.20 | | • Missing/Cold Food| | Second-order ETA | | Floor: >= €13.20 | | Refund Rate | | impact: <= +5m | | (Non-inferior) | | | +------------------+ +-------------------+ +--------------------+ ```

Metric Definitions & Instruments

1. Courier Cost per Order (CFO): Total courier payout (base + distance fees + batch bonuses) divided by completed orders within the window. 2. Late Delivery Share >45m (Head of Ops): P90 delivery time and proportion of orders delivered >45 minutes from placement. Simulation projects a 3–6 min increase for the 2nd order; total late share must not breach 8.3%. 3. Courier Earnings per Active Hour (Courier Relations): Gross earnings divided by active hours (online and on-trip/assigned). Risk: Because couriers are paid per order + distance, batching two orders saves distance, potentially reducing payout unless batch base rates or drop-off multipliers offset it. 4. Customer Experience & Retention Proxy (Growth): * 7-day reorder rate for cohorts exposed to $\ge 2$ batched deliveries. * Delivery-related customer support contacts per 1,000 orders.

---

5. Porto Compliance & Courier Safeguards

Our Porto agreement mandates a 14-day notice period for dispatch adjustments and guarantees that courier average hourly earnings will not fall.

• Timeline Action: Formal written notice delivered to the Porto Couriers’ Association on Wednesday, 1 October 2026.
• Phased Rollout: Porto remains 100% Control through 14 October. On 15 October, Porto enters the switchback rotation.
• Earnings Floor Mechanism: If average courier earnings per active hour in treatment windows drop below €13.20 in Porto (or any city), an automatic “Batch Adjustment Top-Up” will credit couriers the difference at end-of-day. This eliminates legal exposure while allowing us to evaluate the raw algorithm’s unit economics.

---

6. Execution Timeline

---

7. Circuit Breakers & Rollback Criteria

The experiment toggle will automatically shut down batching within 15 minutes if any of the following triggers trip during live operations: * Severe Operations Failure: City-level late delivery rate (>45 min) exceeds 12.0% across two consecutive peak windows. * Courier Backlash/Disruption: Courier unassigned order rate spikes by $>25\%$ or courier offline churn spikes by $>15\%$ compared to historical control. * Severe Food Quality Impact: Customer support contact rate for "cold food" or "late arrival" doubles ($>2.0\times$).

---

8. Stakeholder Sign-Off

By signing below, stakeholders agree that if the primary and guardrail criteria in Section 1 are satisfied by 3 November, order batching will be deployed to all 14 cities on 5 November 2026.

• Chief Operating Officer: _________
• Chief Financial Officer: _________
• Head of Operations: _________
• Courier Relations Lead: _________

Order Batching Experiment Spec

Decision: Roll out batching to all 14 cities before the 6 Nov 2026 code freeze, or not. Owner: Staff PM. Sign-off: COO, CFO, Head of Operations, Courier Relations Lead. Head of Growth is copied in.

1. What changed from the intern's draft

2. Design

• Unit: city × two-hour window. The windows are 11–13, 13–15, 15–17, 17–19, 19–21 and 21–23, so six per city per day.
• Assignment: Within each city-day, the windows form three adjacent pairs. In each pair, one window is randomly batching-on and the other is off. This balances lunch, afternoon and dinner across arms and controls for city and weekday.
• Attribution: An order belongs to the window in which it was placed. The pre-specified sensitivity analysis drops orders placed in the first 30 minutes of each window, because in-flight batches and courier positioning carry over from the previous window.
• Exposure: Batching pairs only nearby restaurants, using the same rules the simulation assumed. The batching configuration is frozen before the test starts.

3. Duration and power

The analyst's figure of about 740 windows assumes windows are independent. They are not: adjacent windows share couriers and demand, and cities differ. I therefore treat 740 as a floor and plan for up to double that.

• Engineering needs one week, so the switch is ready on 7 Oct and the test runs Thu 8 Oct – Wed 28 Oct (21 days, three of every weekday).
• 13 cities × 6 windows × 21 days = 1,638 windows. Porto adds 6 × 14 days = 84. The total is about 1,720 windows, roughly 2.3× the floor.
• We do not stop early on a good-looking interim result. The end date is fixed.
• Analysis: A window-level regression on total cost ÷ total orders, with city and weekday × hour fixed effects and standard errors clustered by city-day. We will confirm with randomisation inference.

4. Metrics and decision rules

All four sign-offs are needed on these thresholds by Tue 6 Oct, before any data exist.

The COO's "without making customers wait noticeably longer" is operationalised as the +1.0 pp cap. The simulations add 3–6 minutes to the second order in each batch, so this cap is the threshold most likely to be contested. The Head of Ops should confirm or change it before the start.

Tension to surface: Cost per order is forecast to fall 6–11%, and couriers are paid per order plus distance. Earnings per hour therefore stay flat only if batching raises orders per courier-hour by at least as much as pay per order falls. The test settles this, and the CFO's and Courier Relations' metrics are not independent.

Secondary metrics (descriptive only): - Mean delivery time, and delivery time for the second order in a batch. - Share of orders batched. - Orders per courier-hour. - Customer support contacts. - Cancellations.

Reorder rate: We report the 7-day rate for test customers at the decision. The 30-day rate is tracked after rollout for customers by batched-order exposure, and it cannot be treated as causal. Growth should know this now.

5. Porto

• Written notice goes to the couriers' association today (30 Sept). Legal should word it to cover both the test and permanent rollout. The earliest compliant start is 14 Oct, so Porto joins the test on 15 Oct.
• The contract says changes must not reduce average hourly earnings. Porto therefore has a stricter rule than other cities: the estimated earnings difference must be ≥ €0 for Porto to ship.
• Porto has only about 84 windows, so its own estimate will be noisy. We pool it for cost and lateness. For earnings we report Porto separately, daily, with the stop rule below.
• If Porto fails or is ambiguous, we ship in the other 13 cities and keep Porto off. Legal should confirm that running the test in Porto does not itself breach the agreement. If it might, exclude Porto from the test and roll out there only on the pooled evidence plus the earnings rule.

6. Stop rules

The per-city switch lets us turn batching off immediately. Any of these triggers a pause in that city and a review within 24 hours:

• The >45-minute share in batching-on windows exceeds off windows by more than 3 pp over two consecutive days.
• Earnings per active hour in batching-on windows are more than 5% below off windows over three consecutive days. In Porto, the threshold is any sustained shortfall that the Courier Relations Lead judges real.
• Any safety or serious courier-welfare incident tied to batching.

7. Timeline

The 29 Oct decision leaves a week of buffer. If the test slips past about 2 Nov, there is no safe rollout before the freeze.

8. Risks and limits

• Seasonality: October demand is below peak. Higher order density should raise batch rates, but it also strains courier supply. We track batch rate by hour and expect the peak effect to differ from the test estimate.
• Carryover: Some carryover between windows will remain even with the 30-minute washout. The sensitivity analysis shows how much it matters.
• Short-run effects: Couriers may change their behaviour as they learn batching, and three weeks may not capture that. We monitor weekly trends within the test.
• Possible outcomes: The result can be a full rollout, a partial rollout (for example excluding Porto or any city that fails a guardrail), or no rollout. Partial rollout by city is allowed by the switch, but only for cities that individually pass the stop rules. City-level results will be noisy, so we do not make ship decisions city by city beyond this.

9. Sign-off

Brisk order-batching experiment spec

Decision owner: Staff PM Sign-off: COO, CFO, Head of Operations, Courier Relations Decision deadline: 4 November 2026; rollout on 5 November, before the 6 November freeze.

1. Decision and hypothesis

Test whether batching can reduce total courier cost per delivered order by at least 5%, without materially worsening delivery times or reducing courier earnings.

At today’s €7.40 baseline, 5% means approximately €0.37 saved per order. Simulations are directional evidence, not sufficient grounds for rollout.

Decision: launch across all 14 cities only if the economic, customer, courier and operational gates below pass. Otherwise, leave batching off before the freeze. A promising but inconclusive result is not a pass.

2. Treatment and eligibility

Treatment enables one courier to carry at most two orders from nearby restaurants. Control retains current assignment.

Before testing, Operations and Engineering will lock:

• Restaurant-proximity, pickup-readiness and maximum-detour rules.
• Maximum predicted delivery times for both orders.
• Courier payment rules, including distance calculation.
• Exception handling, cancellation treatment and customer communications.

Do not change these rules during the confirmatory experiment. A material change requires a new test.

Measure the policy’s effect across all orders, not merely successfully batched orders. Report batching rate and first-/second-order outcomes as diagnostics; comparing batched orders with unbatched orders is selection-biased.

3. Experimental design

Replace order-level randomization with city-wide switchbacks. Treatment orders would otherwise change courier availability and dispatch conditions for control orders, contaminating the comparison.

Each city has six two-hour blocks daily between 11:00 and 23:00. Randomize batching on/off by city-block, with balanced assignment across cities, dates and time of day. Use constrained random schedules—not deterministic alternation.

Before the confirmatory test:

• Set a washout period using historical order/trip completion data and the pilot.
• Exclude the same initial portion of every block from measurement, whether or not its state changes.
• Validate that residual trips and courier repositioning do not materially contaminate measurement. If they do, lengthen blocks or washout and recalculate feasibility.

Attribute order outcomes to order-placement time; follow included orders through completion even after their block ends. Define payout and active-hour attribution consistently before launch.

Analysis is intention-to-treat by assigned city-block. Weight results to the expected rollout order mix, rather than weighting small and large cities equally. Report city-level results and heterogeneity alongside the aggregate.

4. Metrics and acceptance gates

The following customer thresholds are proposed definitions of “not noticeably longer” and require sign-off before testing.

Define active hours identically in both arms, including available waiting time—not just time carrying orders. Report earnings distribution and effects by city and courier segment, not only the pooled mean.

Use baseline-adjusted treatment/control comparisons, not historical averages as the counterfactual. Log cancellations, failed deliveries, assignment failures and complaints as safety diagnostics; cost savings driven by worse completion are unacceptable.

Porto is a separate launch gate: pooled earnings gains cannot override a Porto earnings reduction. Legal and Courier Relations must approve the compliance interpretation and protection mechanism. Any guarantee or top-up must be part of the tested policy and counted in courier cost.

Growth: 30-day reorder is exploratory, not a pre-freeze gate. Even customers first exposed on 14 October do not mature until 13 November. Report available short-term indicators separately and agree a post-freeze retention study; do not present immature reorder data as a 30-day result.

5. Power and analysis

The intern’s “two weeks” is not justified. The analyst’s 740 windows assumes independence and addresses detecting a 5% effect—not necessarily proving savings exceed 5%.

From 14 October through 2 November, the maximum is:

14 cities × 6 blocks × 20 days = 1,680 blocks, before data losses.

That is only 2.27 times the nominal requirement. Serial correlation, washout, uneven volume and separate guardrail requirements can consume this headroom.

By 13 October, Data Science must replay the proposed schedule on historical/pilot data and establish:

• Power to pass the savings threshold under plausible true savings, including 6%, 8% and 11%.
• Power for delivery and earnings non-inferiority, especially Porto.
• Sensitivity to carryover, missing data and city differences.

Use inference respecting the randomized schedule, with serial-correlation and small-number-of-cities sensitivity checks. Pre-register estimators, confidence bounds, exclusions, city safety rules and any multiplicity adjustment for city-specific claims.

Feasibility gate: if adequate power cannot fit before the deadline, do not promise an all-city rollout. Extend testing beyond the freeze or return with a narrower decision.

6. Timeline and operating controls

• 30 September: obtain metric-threshold agreement; send Porto’s written assignment-change notice. Confirm legally valid delivery and when its 14-day period expires.
• By 7 October: deliver city switches, instrumentation, audit logs and rollback.
• 8–13 October: technical/safety pilot outside Porto; validate washout, payments and power. Pilot data excluded from confirmatory results.
• 14 October–2 November: confirmatory test, with Porto starting only after the full notice period has elapsed.
• 3–4 November: finalize outcomes, review compliance and make the signed decision.
• 5 November: one-day rollout only if all gates pass.

Operations owns live monitoring; Engineering owns rollback; Courier Relations owns courier communications and payment disputes. Stop affected cities immediately for payment errors, serious safety incidents or predefined severe service degradation. Such stops are recorded, not silently excluded.

No repeated efficacy peeking. If the test ends early for safety or loses its required sample, it does not automatically qualify for launch. Keep the kill switch available throughout peak season.