Experiment specification

Brisk order-batching experiment spec

Decision owner: Staff PM Sign-off: COO, CFO, Head of Operations, Courier Relations Decision deadline: 4 November 2026; rollout on 5 November, before the 6 November freeze.

1. Decision and hypothesis

Test whether batching can reduce total courier cost per delivered order by at least 5%, without materially worsening delivery times or reducing courier earnings.

At today’s €7.40 baseline, 5% means approximately €0.37 saved per order. Simulations are directional evidence, not sufficient grounds for rollout.

Decision: launch across all 14 cities only if the economic, customer, courier and operational gates below pass. Otherwise, leave batching off before the freeze. A promising but inconclusive result is not a pass.

2. Treatment and eligibility

Treatment enables one courier to carry at most two orders from nearby restaurants. Control retains current assignment.

Before testing, Operations and Engineering will lock:

• Restaurant-proximity, pickup-readiness and maximum-detour rules.
• Maximum predicted delivery times for both orders.
• Courier payment rules, including distance calculation.
• Exception handling, cancellation treatment and customer communications.

Do not change these rules during the confirmatory experiment. A material change requires a new test.

Measure the policy’s effect across all orders, not merely successfully batched orders. Report batching rate and first-/second-order outcomes as diagnostics; comparing batched orders with unbatched orders is selection-biased.

3. Experimental design

Replace order-level randomization with city-wide switchbacks. Treatment orders would otherwise change courier availability and dispatch conditions for control orders, contaminating the comparison.

Each city has six two-hour blocks daily between 11:00 and 23:00. Randomize batching on/off by city-block, with balanced assignment across cities, dates and time of day. Use constrained random schedules—not deterministic alternation.

Before the confirmatory test:

• Set a washout period using historical order/trip completion data and the pilot.
• Exclude the same initial portion of every block from measurement, whether or not its state changes.
• Validate that residual trips and courier repositioning do not materially contaminate measurement. If they do, lengthen blocks or washout and recalculate feasibility.

Attribute order outcomes to order-placement time; follow included orders through completion even after their block ends. Define payout and active-hour attribution consistently before launch.

Analysis is intention-to-treat by assigned city-block. Weight results to the expected rollout order mix, rather than weighting small and large cities equally. Report city-level results and heterogeneity alongside the aggregate.

4. Metrics and acceptance gates

The following customer thresholds are proposed definitions of “not noticeably longer” and require sign-off before testing.

Define active hours identically in both arms, including available waiting time—not just time carrying orders. Report earnings distribution and effects by city and courier segment, not only the pooled mean.

Use baseline-adjusted treatment/control comparisons, not historical averages as the counterfactual. Log cancellations, failed deliveries, assignment failures and complaints as safety diagnostics; cost savings driven by worse completion are unacceptable.

Porto is a separate launch gate: pooled earnings gains cannot override a Porto earnings reduction. Legal and Courier Relations must approve the compliance interpretation and protection mechanism. Any guarantee or top-up must be part of the tested policy and counted in courier cost.

Growth: 30-day reorder is exploratory, not a pre-freeze gate. Even customers first exposed on 14 October do not mature until 13 November. Report available short-term indicators separately and agree a post-freeze retention study; do not present immature reorder data as a 30-day result.

5. Power and analysis

The intern’s “two weeks” is not justified. The analyst’s 740 windows assumes independence and addresses detecting a 5% effect—not necessarily proving savings exceed 5%.

From 14 October through 2 November, the maximum is:

14 cities × 6 blocks × 20 days = 1,680 blocks, before data losses.

That is only 2.27 times the nominal requirement. Serial correlation, washout, uneven volume and separate guardrail requirements can consume this headroom.

By 13 October, Data Science must replay the proposed schedule on historical/pilot data and establish:

• Power to pass the savings threshold under plausible true savings, including 6%, 8% and 11%.
• Power for delivery and earnings non-inferiority, especially Porto.
• Sensitivity to carryover, missing data and city differences.

Use inference respecting the randomized schedule, with serial-correlation and small-number-of-cities sensitivity checks. Pre-register estimators, confidence bounds, exclusions, city safety rules and any multiplicity adjustment for city-specific claims.

Feasibility gate: if adequate power cannot fit before the deadline, do not promise an all-city rollout. Extend testing beyond the freeze or return with a narrower decision.

6. Timeline and operating controls

• 30 September: obtain metric-threshold agreement; send Porto’s written assignment-change notice. Confirm legally valid delivery and when its 14-day period expires.
• By 7 October: deliver city switches, instrumentation, audit logs and rollback.
• 8–13 October: technical/safety pilot outside Porto; validate washout, payments and power. Pilot data excluded from confirmatory results.
• 14 October–2 November: confirmatory test, with Porto starting only after the full notice period has elapsed.
• 3–4 November: finalize outcomes, review compliance and make the signed decision.
• 5 November: one-day rollout only if all gates pass.

Operations owns live monitoring; Engineering owns rollback; Courier Relations owns courier communications and payment disputes. Stop affected cities immediately for payment errors, serious safety incidents or predefined severe service degradation. Such stops are recorded, not silently excluded.

No repeated efficacy peeking. If the test ends early for safety or loses its required sample, it does not automatically qualify for launch. Keep the kill switch available throughout peak season.

Experiment spec: Order batching before the peak-season freeze

Decision owner: COO · Sign-off: COO, CFO, Head of Operations, Courier Relations Lead · Copied: Head of Growth Status: Draft for sign-off, 30 September 2026

1. Decision

Should we switch batching on in all 14 cities before the code freeze on 6 November 2026? The switch works per city, so the result can also be "on in some cities."

2. What changed from the draft plan, and why

3. Timeline

The dates are fixed by two constraints. The Porto notice cannot slip past today without losing Porto days. The test cannot run past 28 October without eating into the rollout buffer.

4. Design

• Unit: one city in one 2-hour window. Each day has six windows (11–13, 13–15, 15–17, 17–19, 19–21, 21–23).
• Assignment: randomised 50/50 within each city and time slot. Each city gets a balanced mix of on and off across lunch, dinner, weekdays and weekends.
• Switch behaviour: new batches form only while the switch is on. Batches already in progress when it turns off complete normally.
• Washout: the first 30 minutes of each window are excluded from analysis. This leaves 90 analysed minutes per window.
• Sample: 13 cities × 6 windows × 21 days plus Porto × 6 × 15 days gives 1,728 windows. That is about 2.3× the analyst's 740. The extra covers the washout and the correlation between windows. Analytics to confirm the power calculation using the correlation seen in the first week's data. If power is short, we extend to 30 October and give up one buffer day.
• Analysis: compare window-level metrics with adjustments for city, day and time slot. Treat all windows in the same city on the same day as one group when calculating uncertainty. Report the network result and each city's result.

5. Metrics and decision rule

Rollout rule: a city rolls out on 3 Nov if the network result passes the primary metric and both guardrails, and that city's own result shows no clear guardrail failure.

The 1.0pp and €0.40 margins are proposals. Signers must confirm or change them before 8 October. They cannot be changed after the test starts.

6. Tensions signers should know about

What "at least 5%" can mean. We can reliably detect whether batching cuts cost by 5% versus not at all. We cannot prove the true effect is above 5% if it is only 6%, which is the low end of the simulation. The decision rule therefore uses the point estimate. If the estimate lands between 4% and 6%, the COO decides with that uncertainty in view.

Cost savings and courier earnings pull in opposite directions. Couriers are paid per order plus distance, so a lower cost per order is lower pay per order. Earnings per hour only hold up if couriers complete more orders per hour.

• During the test, the same couriers are logged in whether batching is on or off. That makes the test a fair early read on hourly earnings.
• After rollout, the saving only stays a saving if we need fewer courier hours. Fewer hours affects couriers' income, even if the hourly rate is protected.
• The CFO and Courier Relations Lead should agree on this before sign-off, not after results.

Porto. "Must not reduce average hourly earnings" is a contractual condition, not a statistical margin. Porto gives us only ~90 windows, too few to prove "no reduction."

• If network earnings per hour fall at all, Porto stays off unless it comes with an earnings protection mechanism. One example is a per-batch supplement, which legal and Courier Relations should scope now.
• If Porto rolls out, we measure its hourly earnings against the pre-batching baseline every week.

"Noticeably longer." We are defining this as the >45-minute late share plus the complaint rate. We are not using average delivery time, because a 3–6 minute delay on second orders is expected by design.

7. Stop rules (checked daily by the on-call analyst)

• City level: the >45-minute late share in "on" windows exceeds 10% for two consecutive days. Batching goes off in that city pending review.
• Network level:
• Any courier safety incident linked to batching, or a formal complaint from a couriers' association: pause everywhere.
• Earnings per hour in "on" windows below €12.50 for three consecutive days: pause everywhere.
• Porto only: any clear drop in hourly earnings leads to an immediate pause and notification to Courier Relations.

8. What each signer is agreeing to

• COO: the dates in §3, the rollout rule in §5, and making the call if the result falls between 4% and 6%.
• CFO: cost per order as the primary metric, with the ≥5% rule. Also acknowledges that post-rollout savings depend on how many courier hours we buy (§6).
• Head of Operations: the 1.0pp late-share margin and the stop rules.
• Courier Relations Lead:
• the €0.40 margin;
• the separate rule for Porto;
• sending the Porto notice today.

9. Open items

1. Legal, today: Porto notice wording, and whether one notice covers both the test and the rollout. 2. Analytics, by 7 Oct: - power calculation that accounts for correlated windows; - randomisation schedule; - dashboard. 3. Engineering, by 7 Oct: per-city switch with switch times logged, tested in one city after hours. 4. Courier Relations and Legal, by 2 Nov: Porto earnings protection option. 5. Signers, by 8 Oct: confirm the guardrail margins in §5.

EXPERIMENT SPECIFICATION: EUROPEAN ORDER BATCHING ROLLOUT

Author: Staff PM, Brisk Date: 30 September 2026 Target Decision Date: 27 October 2026 Code Freeze Deadline: 6 November 2026 Stakeholder Sign-Off: COO, CFO, Head of Operations, Courier Relations Lead CC: Head of Growth

---

1. Executive Summary & Objective

Brisk processes approximately 1.9 million orders per month across 14 European cities. To improve unit economics ahead of peak season, we are evaluating the rollout of Order Batching—allowing a single courier to carry two orders from nearby restaurants along an optimized route.

Simulations indicate batching can reduce courier cost per order by 6–11% while adding 3–6 minutes to the second delivery in a batch.

This spec establishes a statistically rigorous, city-level alternating rollout to determine whether we can achieve the COO’s launch criteria—a $\ge 5\%$ reduction in courier cost per order without degrading customer experience—before the 6 November 2026 code freeze.

---

2. Fixing the Intern’s Draft: Key Methodology Corrections

The intern’s initial proposal (a 50/50 randomized order-level split within cities) is fundamentally flawed for three operational reasons: 1. The Interference Problem: An order-level split means control and treatment orders are competing for the exact same couriers in real-time. This distorts courier utilization, falsely inflates earnings for treatment assignments, and poisons the control group data. 2. The Porto Legal Trap: An order-level split disrupts dispatch algorithms dynamically without the required 14 days' written notice to the Porto couriers' association regarding structural assignment changes. 3. Operational Reality: Batching is a network-level routing feature. It must be tested at the geographic/time-window level, not the individual order level.

---

3. Executive Dashboard: Addressing Stakeholder Requirements

To secure cross-functional sign-off, the experiment is instrumented to explicitly track the core metrics demanded by each stakeholder:

---

4. Experimental Design: City-Level Time-Window Switchback

To achieve statistical power while preventing network interference, we will use a Switchback Design across all 14 cities simultaneously.

• Unit of Assignment: 2-hour operational windows (e.g., 11:00–13:00, 13:00–15:00, etc., running through the 11:00–23:00 operating day). This yields 6 windows per city per day.
• Treatment Assignment: Within each city, 2-hour windows are randomly assigned to either Treatment (Batching algorithm active) or Control (Standard single-order dispatch).
• Statistical Power Justification: Per our Data Analyst’s note, detecting a 5% shift in courier cost requires approximately 740 independent windows. Across 14 cities running 6 windows per day (84 windows/day total), we will accumulate >1,100 windows in a 14-day test period (13 October – 26 October), guaranteeing sufficient statistical power.

Compliance & Regulatory Notice (Porto)

To satisfy our agreement with the Porto couriers' association, formal written notice of a temporary dispatch optimization test will be dispatched on 1 October 2026 (providing a 12-day buffer ahead of the 13 October test start, and well within our internal timelines). The notice confirms that average hourly earnings will be monitored in real-time and protected.

---

5. Timeline to 6 November 2026 Code Freeze

• 30 September (Today): Finalize and sign off experiment spec.
• 1 October: Issue 14-day advance notice to Porto couriers' association. Engineering begins building the feature toggle.
• 7 October: Engineering completes work. Feature deployed behind a global kill-switch. Dry run / QA across staging environments.
• 13 October: Experiment Launch. Switchback testing goes live across all 14 cities.
• 26 October: Experiment concludes. Data pipeline freezes at 23:59.
• 27 October: Decision Gate. Data Science, Product, and Ops analyze results against executive criteria. If criteria are met, COO/CFO/Ops/Courier Leads sign off on permanent activation.
• 28 October – 5 November: Buffer for any bug fixes, edge-case routing refinements, or full-city rollout execution (which takes 1 day).
• 6 November: Peak-Season Code Freeze. System stable, batched, and optimized.

---

6. Risk Mitigation & Kill-Switch Protocols

The experiment will be automatically aborted in a specific city or globally if any of the following tripwires are breached over a rolling 24-hour window: 1. Courier Earnings Drop: If courier earnings per active hour in any city drop below €13.20 over a 48-hour average (violating Porto terms and general labor health). 2. Severe Latency Spike: If the share of orders delivered $>45$ minutes exceeds 10% overall. 3. Technical Failure: Any dispatch deadlock or routing loop triggers an immediate reversion to standard single-order dispatch via the engineering kill-switch.

---

7. Sign-Off & Approval

By signing below, executive stakeholders approve this experiment spec, timeline, and success criteria for the Brisk Order Batching Rollout.

• COO: _________ (Date: ____)
• CFO: _________ (Date: ____)
• Head of Operations: _________ (Date: ____)
• Courier Relations Lead: _________ (Date: ____)