Experiment specification

Brisk order-batching experiment: pre-freeze decision

Decision owner: Staff PM Required approvers: COO, CFO, Head of Operations, courier relations lead Decision deadline: 4 November 2026; rollout on 5 November, before the 6 November freeze.

1. Decision and hypothesis

Ship batching to all 14 cities only if it demonstrates at least 5% lower courier cost per order, with acceptable customer experience and protected courier earnings.

At today’s €7.40 baseline, 5% is approximately €0.37 per order. Simulated savings are not sufficient evidence.

The thresholds below are proposed sign-off terms, not established definitions of “noticeably longer.” Approvers must agree them before randomisation. An inconclusive result means no all-city rollout before the freeze, not permission to lower the bar.

2. Replace order randomisation with city switchbacks

Order-level randomisation is unsuitable: treatment and control orders share couriers, restaurant queues and dispatch capacity. Batching changes availability for both groups, contaminating the comparison.

Instead, switch each city between:

• Treatment: batching enabled under frozen restaurant-proximity, route-detour and eligibility rules; maximum two orders per courier.
• Control: current single-order dispatch.

Randomise six local-time blocks daily: 11–13, 13–15, …, 21–23. Balance treatment across cities, weekdays and clock times, with approximately equal exposure per city. Generate and lock the schedule in advance; do not let operators choose favourable treatment periods.

This estimates the effect of offering batching across a city, not merely its effect on successfully batched orders. Analyse all orders in the measurement cohort, including those never batched.

Carryover: Complete existing assignments safely at switches; never split an accepted batch. Set a fixed washout period using historical and shadow-mode assignment-to-completion times, before observing treatment results. Exclude order arrivals during that period from the primary cohort in every block, regardless of whether the state changed. Follow included orders through completion; do not truncate outcomes at block boundaries. If carryover cannot be contained within two-hour blocks, use longer blocks and re-power.

3. Metrics and launch gates

Use concurrent controls and pre-specified, one-sided 95% confidence bounds. Every gate must pass.

Cost includes distance pay, incentives, failed-delivery compensation and any batching-related earnings protection. Finance freezes allocation rules before launch. Track cancellation/completion rates so apparent savings cannot come from abandoning orders.

Define active hours as available-for-work plus engaged time, excluding genuine offline breaks—not just delivery time. Use consistent definitions across arms. Report both total earnings/total hours and courier-level distributions.

Report results by city, peak/off-peak, first/second batch position and courier earnings decile. Batch-position comparisons are descriptive, not randomised causal estimates. Examine late-delivery tails and cancellations; averages must not conceal material local harm.

Growth: Thirty-day reorder is exploratory, not a freeze gate. The first main-trial cohort matures on 14 November. Customer crossover also limits interpretation; do not present this switchback as a clean test of persistent customer retention.

4. Porto is a contractual prerequisite

Send written notice on 30 September, covering experimental on/off assignment changes and potential rollout. Legal and courier relations must confirm valid receipt, scope and the earliest permitted activation. Plan Porto activation for 15 October, providing a buffer beyond 14 days; late receipt moves that date.

Before activation, agree the contract’s hourly-earnings definition and comparison method with the association. The network’s €13.20 is not a Porto-specific contractual baseline.

Implement a funded make-whole mechanism accepted as satisfying the no-reduction requirement. Include those payments in experimental cost. A statistical non-inferiority margin cannot waive the agreement. Without compliant notice and earnings protection, Porto stays off—and the all-14-city decision is no-go.

5. Power, analysis and validity

The analyst’s 740 windows is not a sufficient sample-size justification:

• Adjacent windows share demand, weather, couriers and backlog.
• Washout reduces usable exposure.
• Detecting a 5% saving versus zero differs from proving a saving of at least 5%.
• Customer and earnings gates may require more data than cost.

Before launch, replay historical city-time data under the proposed schedule. Incorporate autocorrelation, city heterogeneity, washout and realistic take-up. Estimate power for every gate and the probability of passing them jointly, using simulation scenarios spanning 6–11% savings. Target at least 90% joint power under an explicitly agreed planning scenario.

Estimate intention-to-treat effects using city and calendar/clock-time adjustment, with fleet weights fixed from pre-experiment volume. Use schedule-respecting randomisation inference and dependence-aware uncertainty—not independent-window standard errors.

Publish city-level uncertainty. Pre-agree local safety thresholds and multiplicity handling; a pooled pass does not automatically authorise a city showing credible harm.

Feasibility gate: If adequate evidence cannot fit before the deadline, recommend no all-city pre-freeze launch.

6. Calendar and operating controls

• 30 September: Approve protocol and proposed margins; send Porto notice.
• 1–7 October: Build city switches, assignment logging, earnings protection and tested emergency rollback.
• 8–14 October: Shadow-mode validation; finalise washout, power, schedule and registered analysis.
• 15 October–2 November: Main experiment: 19 days, at most 1,596 nominal city-blocks before exclusions—not 1,596 independent observations.
• 3–4 November: Reconcile outcomes and compensation; analyse; obtain four signatures.
• 5 November: If approved, stage rollout across cities within the day, with monitored checkpoints.

Operations owns live monitoring. Pre-register severe-delay, cancellation, earnings and system-failure stop rules and a sequential-monitoring procedure. Emergency pauses remain permissible; log overrides and retain affected observations in assignment-based analysis. No opportunistic early declaration of success.

7. Sign-off package

Deliver effect estimates and confidence bounds, city diagnostics, notice/compliance evidence, incident logs, and Finance’s net savings forecast including protection payments and operating costs.

Go: Every statistical, contractual, local-safety and operational gate passes. No-go: Any gate fails, remains inconclusive, or cannot be evaluated reliably by 4 November. A subset rollout requires a separate, explicitly approved decision; it is not success against the all-city objective.

Experiment spec: Order batching before the peak-season freeze

Decision owner: COO · Sign-off: COO, CFO, Head of Operations, Courier Relations Lead · Copied: Head of Growth Status: Draft for sign-off, 30 September 2026

1. Decision

Should we switch batching on in all 14 cities before the code freeze on 6 November 2026? The switch works per city, so the result can also be "on in some cities."

2. What changed from the draft plan, and why

3. Timeline

The dates are fixed by two constraints. The Porto notice cannot slip past today without losing Porto days. The test cannot run past 28 October without eating into the rollout buffer.

4. Design

• Unit: one city in one 2-hour window. Each day has six windows (11–13, 13–15, 15–17, 17–19, 19–21, 21–23).
• Assignment: randomised 50/50 within each city and time slot. Each city gets a balanced mix of on and off across lunch, dinner, weekdays and weekends.
• Switch behaviour: new batches form only while the switch is on. Batches already in progress when it turns off complete normally.
• Washout: the first 30 minutes of each window are excluded from analysis. This leaves 90 analysed minutes per window.
• Sample: 13 cities × 6 windows × 21 days plus Porto × 6 × 15 days gives 1,728 windows. That is about 2.3× the analyst's 740. The extra covers the washout and the correlation between windows. Analytics to confirm the power calculation using the correlation seen in the first week's data. If power is short, we extend to 30 October and give up one buffer day.
• Analysis: compare window-level metrics with adjustments for city, day and time slot. Treat all windows in the same city on the same day as one group when calculating uncertainty. Report the network result and each city's result.

5. Metrics and decision rule

Rollout rule: a city rolls out on 3 Nov if the network result passes the primary metric and both guardrails, and that city's own result shows no clear guardrail failure.

The 1.0pp and €0.40 margins are proposals. Signers must confirm or change them before 8 October. They cannot be changed after the test starts.

6. Tensions signers should know about

What "at least 5%" can mean. We can reliably detect whether batching cuts cost by 5% versus not at all. We cannot prove the true effect is above 5% if it is only 6%, which is the low end of the simulation. The decision rule therefore uses the point estimate. If the estimate lands between 4% and 6%, the COO decides with that uncertainty in view.

Cost savings and courier earnings pull in opposite directions. Couriers are paid per order plus distance, so a lower cost per order is lower pay per order. Earnings per hour only hold up if couriers complete more orders per hour.

• During the test, the same couriers are logged in whether batching is on or off. That makes the test a fair early read on hourly earnings.
• After rollout, the saving only stays a saving if we need fewer courier hours. Fewer hours affects couriers' income, even if the hourly rate is protected.
• The CFO and Courier Relations Lead should agree on this before sign-off, not after results.

Porto. "Must not reduce average hourly earnings" is a contractual condition, not a statistical margin. Porto gives us only ~90 windows, too few to prove "no reduction."

• If network earnings per hour fall at all, Porto stays off unless it comes with an earnings protection mechanism. One example is a per-batch supplement, which legal and Courier Relations should scope now.
• If Porto rolls out, we measure its hourly earnings against the pre-batching baseline every week.

"Noticeably longer." We are defining this as the >45-minute late share plus the complaint rate. We are not using average delivery time, because a 3–6 minute delay on second orders is expected by design.

7. Stop rules (checked daily by the on-call analyst)

• City level: the >45-minute late share in "on" windows exceeds 10% for two consecutive days. Batching goes off in that city pending review.
• Network level:
• Any courier safety incident linked to batching, or a formal complaint from a couriers' association: pause everywhere.
• Earnings per hour in "on" windows below €12.50 for three consecutive days: pause everywhere.
• Porto only: any clear drop in hourly earnings leads to an immediate pause and notification to Courier Relations.

8. What each signer is agreeing to

• COO: the dates in §3, the rollout rule in §5, and making the call if the result falls between 4% and 6%.
• CFO: cost per order as the primary metric, with the ≥5% rule. Also acknowledges that post-rollout savings depend on how many courier hours we buy (§6).
• Head of Operations: the 1.0pp late-share margin and the stop rules.
• Courier Relations Lead:
• the €0.40 margin;
• the separate rule for Porto;
• sending the Porto notice today.

9. Open items

1. Legal, today: Porto notice wording, and whether one notice covers both the test and the rollout. 2. Analytics, by 7 Oct: - power calculation that accounts for correlated windows; - randomisation schedule; - dashboard. 3. Engineering, by 7 Oct: per-city switch with switch times logged, tested in one city after hours. 4. Courier Relations and Legal, by 2 Nov: Porto earnings protection option. 5. Signers, by 8 Oct: confirm the guardrail margins in §5.