Experiment specification

EXPERIMENT SPECIFICATION: EUROPEAN ORDER BATCHING ROLLOUT

Author: Staff PM, Brisk Date: 30 September 2026 Target Decision Date: 27 October 2026 Code Freeze Deadline: 6 November 2026 Stakeholder Sign-Off: COO, CFO, Head of Operations, Courier Relations Lead CC: Head of Growth

---

1. Executive Summary & Objective

Brisk processes approximately 1.9 million orders per month across 14 European cities. To improve unit economics ahead of peak season, we are evaluating the rollout of Order Batching—allowing a single courier to carry two orders from nearby restaurants along an optimized route.

Simulations indicate batching can reduce courier cost per order by 6–11% while adding 3–6 minutes to the second delivery in a batch.

This spec establishes a statistically rigorous, city-level alternating rollout to determine whether we can achieve the COO’s launch criteria—a $\ge 5\%$ reduction in courier cost per order without degrading customer experience—before the 6 November 2026 code freeze.

---

2. Fixing the Intern’s Draft: Key Methodology Corrections

The intern’s initial proposal (a 50/50 randomized order-level split within cities) is fundamentally flawed for three operational reasons: 1. The Interference Problem: An order-level split means control and treatment orders are competing for the exact same couriers in real-time. This distorts courier utilization, falsely inflates earnings for treatment assignments, and poisons the control group data. 2. The Porto Legal Trap: An order-level split disrupts dispatch algorithms dynamically without the required 14 days' written notice to the Porto couriers' association regarding structural assignment changes. 3. Operational Reality: Batching is a network-level routing feature. It must be tested at the geographic/time-window level, not the individual order level.

---

3. Executive Dashboard: Addressing Stakeholder Requirements

To secure cross-functional sign-off, the experiment is instrumented to explicitly track the core metrics demanded by each stakeholder:

---

4. Experimental Design: City-Level Time-Window Switchback

To achieve statistical power while preventing network interference, we will use a Switchback Design across all 14 cities simultaneously.

• Unit of Assignment: 2-hour operational windows (e.g., 11:00–13:00, 13:00–15:00, etc., running through the 11:00–23:00 operating day). This yields 6 windows per city per day.
• Treatment Assignment: Within each city, 2-hour windows are randomly assigned to either Treatment (Batching algorithm active) or Control (Standard single-order dispatch).
• Statistical Power Justification: Per our Data Analyst’s note, detecting a 5% shift in courier cost requires approximately 740 independent windows. Across 14 cities running 6 windows per day (84 windows/day total), we will accumulate >1,100 windows in a 14-day test period (13 October – 26 October), guaranteeing sufficient statistical power.

Compliance & Regulatory Notice (Porto)

To satisfy our agreement with the Porto couriers' association, formal written notice of a temporary dispatch optimization test will be dispatched on 1 October 2026 (providing a 12-day buffer ahead of the 13 October test start, and well within our internal timelines). The notice confirms that average hourly earnings will be monitored in real-time and protected.

---

5. Timeline to 6 November 2026 Code Freeze

• 30 September (Today): Finalize and sign off experiment spec.
• 1 October: Issue 14-day advance notice to Porto couriers' association. Engineering begins building the feature toggle.
• 7 October: Engineering completes work. Feature deployed behind a global kill-switch. Dry run / QA across staging environments.
• 13 October: Experiment Launch. Switchback testing goes live across all 14 cities.
• 26 October: Experiment concludes. Data pipeline freezes at 23:59.
• 27 October: Decision Gate. Data Science, Product, and Ops analyze results against executive criteria. If criteria are met, COO/CFO/Ops/Courier Leads sign off on permanent activation.
• 28 October – 5 November: Buffer for any bug fixes, edge-case routing refinements, or full-city rollout execution (which takes 1 day).
• 6 November: Peak-Season Code Freeze. System stable, batched, and optimized.

---

6. Risk Mitigation & Kill-Switch Protocols

The experiment will be automatically aborted in a specific city or globally if any of the following tripwires are breached over a rolling 24-hour window: 1. Courier Earnings Drop: If courier earnings per active hour in any city drop below €13.20 over a 48-hour average (violating Porto terms and general labor health). 2. Severe Latency Spike: If the share of orders delivered $>45$ minutes exceeds 10% overall. 3. Technical Failure: Any dispatch deadlock or routing loop triggers an immediate reversion to standard single-order dispatch via the engineering kill-switch.

---

7. Sign-Off & Approval

By signing below, executive stakeholders approve this experiment spec, timeline, and success criteria for the Brisk Order Batching Rollout.

• COO: _________ (Date: ____)
• CFO: _________ (Date: ____)
• Head of Operations: _________ (Date: ____)
• Courier Relations Lead: _________ (Date: ____)

Brisk order-batching experiment: pre-freeze decision

Decision owner: Staff PM Required approvers: COO, CFO, Head of Operations, courier relations lead Decision deadline: 4 November 2026; rollout on 5 November, before the 6 November freeze.

1. Decision and hypothesis

Ship batching to all 14 cities only if it demonstrates at least 5% lower courier cost per order, with acceptable customer experience and protected courier earnings.

At today’s €7.40 baseline, 5% is approximately €0.37 per order. Simulated savings are not sufficient evidence.

The thresholds below are proposed sign-off terms, not established definitions of “noticeably longer.” Approvers must agree them before randomisation. An inconclusive result means no all-city rollout before the freeze, not permission to lower the bar.

2. Replace order randomisation with city switchbacks

Order-level randomisation is unsuitable: treatment and control orders share couriers, restaurant queues and dispatch capacity. Batching changes availability for both groups, contaminating the comparison.

Instead, switch each city between:

• Treatment: batching enabled under frozen restaurant-proximity, route-detour and eligibility rules; maximum two orders per courier.
• Control: current single-order dispatch.

Randomise six local-time blocks daily: 11–13, 13–15, …, 21–23. Balance treatment across cities, weekdays and clock times, with approximately equal exposure per city. Generate and lock the schedule in advance; do not let operators choose favourable treatment periods.

This estimates the effect of offering batching across a city, not merely its effect on successfully batched orders. Analyse all orders in the measurement cohort, including those never batched.

Carryover: Complete existing assignments safely at switches; never split an accepted batch. Set a fixed washout period using historical and shadow-mode assignment-to-completion times, before observing treatment results. Exclude order arrivals during that period from the primary cohort in every block, regardless of whether the state changed. Follow included orders through completion; do not truncate outcomes at block boundaries. If carryover cannot be contained within two-hour blocks, use longer blocks and re-power.

3. Metrics and launch gates

Use concurrent controls and pre-specified, one-sided 95% confidence bounds. Every gate must pass.

Cost includes distance pay, incentives, failed-delivery compensation and any batching-related earnings protection. Finance freezes allocation rules before launch. Track cancellation/completion rates so apparent savings cannot come from abandoning orders.

Define active hours as available-for-work plus engaged time, excluding genuine offline breaks—not just delivery time. Use consistent definitions across arms. Report both total earnings/total hours and courier-level distributions.

Report results by city, peak/off-peak, first/second batch position and courier earnings decile. Batch-position comparisons are descriptive, not randomised causal estimates. Examine late-delivery tails and cancellations; averages must not conceal material local harm.

Growth: Thirty-day reorder is exploratory, not a freeze gate. The first main-trial cohort matures on 14 November. Customer crossover also limits interpretation; do not present this switchback as a clean test of persistent customer retention.

4. Porto is a contractual prerequisite

Send written notice on 30 September, covering experimental on/off assignment changes and potential rollout. Legal and courier relations must confirm valid receipt, scope and the earliest permitted activation. Plan Porto activation for 15 October, providing a buffer beyond 14 days; late receipt moves that date.

Before activation, agree the contract’s hourly-earnings definition and comparison method with the association. The network’s €13.20 is not a Porto-specific contractual baseline.

Implement a funded make-whole mechanism accepted as satisfying the no-reduction requirement. Include those payments in experimental cost. A statistical non-inferiority margin cannot waive the agreement. Without compliant notice and earnings protection, Porto stays off—and the all-14-city decision is no-go.

5. Power, analysis and validity

The analyst’s 740 windows is not a sufficient sample-size justification:

• Adjacent windows share demand, weather, couriers and backlog.
• Washout reduces usable exposure.
• Detecting a 5% saving versus zero differs from proving a saving of at least 5%.
• Customer and earnings gates may require more data than cost.

Before launch, replay historical city-time data under the proposed schedule. Incorporate autocorrelation, city heterogeneity, washout and realistic take-up. Estimate power for every gate and the probability of passing them jointly, using simulation scenarios spanning 6–11% savings. Target at least 90% joint power under an explicitly agreed planning scenario.

Estimate intention-to-treat effects using city and calendar/clock-time adjustment, with fleet weights fixed from pre-experiment volume. Use schedule-respecting randomisation inference and dependence-aware uncertainty—not independent-window standard errors.

Publish city-level uncertainty. Pre-agree local safety thresholds and multiplicity handling; a pooled pass does not automatically authorise a city showing credible harm.

Feasibility gate: If adequate evidence cannot fit before the deadline, recommend no all-city pre-freeze launch.

6. Calendar and operating controls

• 30 September: Approve protocol and proposed margins; send Porto notice.
• 1–7 October: Build city switches, assignment logging, earnings protection and tested emergency rollback.
• 8–14 October: Shadow-mode validation; finalise washout, power, schedule and registered analysis.
• 15 October–2 November: Main experiment: 19 days, at most 1,596 nominal city-blocks before exclusions—not 1,596 independent observations.
• 3–4 November: Reconcile outcomes and compensation; analyse; obtain four signatures.
• 5 November: If approved, stage rollout across cities within the day, with monitored checkpoints.

Operations owns live monitoring. Pre-register severe-delay, cancellation, earnings and system-failure stop rules and a sequential-monitoring procedure. Emergency pauses remain permissible; log overrides and retain affected observations in assignment-based analysis. No opportunistic early declaration of success.

7. Sign-off package

Deliver effect estimates and confidence bounds, city diagnostics, notice/compliance evidence, incident logs, and Finance’s net savings forecast including protection payments and operating costs.

Go: Every statistical, contractual, local-safety and operational gate passes. No-go: Any gate fails, remains inconclusive, or cannot be evaluated reliably by 4 November. A subset rollout requires a separate, explicitly approved decision; it is not success against the all-city objective.

Brisk order-batching experiment spec

Decision required: Whether to enable batching in all 14 cities on 5 November 2026, ahead of the 6 November code freeze. Batching pairs two eligible orders from nearby restaurants for one courier.

Decision principles and sign-off

Launch only if the experiment shows a worthwhile courier-cost reduction, customer lateness stays within an agreed limit, and courier earnings protections are met. An inconclusive result is not a pass.

Before launch, the COO and Head of Operations must approve the customer-delay limit below; the courier relations lead and Legal must confirm the Porto protocol in writing. If those approvals or the schedule cannot be met, do not claim evidence for an all-city rollout before the freeze.

Design

Randomise city × two-hour operating windows, not individual orders. Orders interact through shared courier supply, and individual randomisation would contaminate the comparison. In each treatment window, the batching policy is enabled; in each control window, batching is disabled. Randomise and balance windows within each city, weekday and daypart, aiming for an even treatment/control split. Keep the assignment schedule fixed before the test begins.

An order is analysed according to its city-window at placement (intention to treat). Batch only orders placed in the same treatment window; do not pair across windows or with control orders. This limits cross-arm contamination at switchovers. Record actual batching eligibility and use, but do not reclassify orders based on whether they were batched.

Timeline and Porto condition

• 30 September: Send Porto’s couriers’ association written notice. The notice requirement applies to the experiment as well as any rollout. Start Porto treatment no earlier than 15 October, allowing 14 full days.
• By 7 October: Complete the per-city switch, monitoring, and rollback controls; test the assignment and cost calculations.
• 15 October–4 November: Run the experiment in all cities, subject to Porto approval.
• 5 November: Review results and, only if all launch gates pass, enable batching in all 14 cities in one day. Otherwise keep it off. Do not shorten the experiment to meet the freeze.

Porto must not enter treatment until Legal and the courier relations lead confirm the notice and trial protocol comply with the agreement. Agree a written earnings-protection mechanism with the association before starting; any top-ups count in courier cost. If this cannot be agreed, exclude Porto from treatment and do not describe the result as support for a 14-city rollout.

Measures and launch gates

Estimate treatment effects against control, with confidence intervals and pre-specified analysis. Baselines: courier cost €7.40/order, orders over 45 minutes 7.5%, courier earnings €13.20/active hour.

The proposed 0.5pp lateness margin and the earnings non-inferiority margin are policy choices, not facts in the brief; the accountable leads must sign them off before randomisation. Do not change margins after seeing results.

Analysis and power

The schedule contains 1,764 city-windows (14 cities × 6 windows/day × 21 days), before any exclusions. The analyst’s estimate of 740 assumes independent windows; successive windows may be correlated, and couriers and demand carry across time. Before the test, use historical city-window data and the planned randomisation to simulate power, accounting for serial correlation and city/daypart differences. Use blocked randomisation-based inference or an appropriate model with uncertainty at the randomised-window level. Do not claim the test is powered merely because 1,764 exceeds 740. If adjusted power is inadequate, or results are too imprecise to meet a gate by 5 November, the decision is inconclusive: no rollout before freeze.

Operations and rollback

Monitor treatment uptake, cost, lateness and earnings daily, but do not stop early for a favourable result. Pause a city for a safety, service or earnings breach under pre-agreed operational thresholds; document the incident and retain it in intention-to-treat analysis. Keep the city switch available for immediate rollback. Sign-off: COO, CFO, Head of Operations, courier relations lead; Growth acknowledges the delayed reorder readout.