How to Be Cautious When Interpreting Data Based on Conditions (Cognitive Biases)

We do not aim to teach every theoretical nuance. We want to build a practical reflex: before we accept an inference that depends on a condition (e.g., “among those who did X, Y happened”), we will (1) ask who was included, (2) surface the assumptions behind the condition, (3) test whether the result changes if we relax or change the condition. Each step is actionable; each can be done in 5–30 minutes depending on scale.

We begin with micro‑scenes: the office meeting, the one‑page analysis, and the inbox with a CSV. These are the places where conditioning mistakes happen. We will practice in these scenes. We will also journal our steps in Brali LifeOS and track micro‑decisions with a check‑in pattern.

A morning scene: the email and the split We open an email at 08:12. Subject line: “Top 10: Our Most Productive People.” The body lists ten names and three metrics. The sender closes: “Recommend we roll out the X method team‑wide.” We read, and the first reflex is to copy the list into a slide. Instead, we stop. We remind ourselves of two things: (a) conditional claims need inspection; (b) the easiest place to start is by asking about the denominator.

We assume the group in the list is the whole population. We ask the sender: “How were these 10 selected? Were exclusions applied?” The reply comes back at 08:39: “We pulled everyone with >100 tasks closed in the last quarter.” We read that and immediately see a filter: people with fewer than 100 tasks are not represented. That’s a classic conditioning problem — the sample is restricted by performance. We assumed X → observed Y → changed to Z: We assumed the list represented everyone → observed the selection rule (>100 tasks) → changed to a plan to evaluate the method across all performance bands instead of only the top decile.

Two immediate micro‑tasks (≤10 minutes each)

1. Pull the filter: open the CSV, note the selection rule in one sentence. Log this in Brali LifeOS as a 5‑minute task: “Find and record the selection rule.” We mark the task done.
2. Sketch the excluded group: count how many people had <100 tasks and log that number. If you can't access the data, ask for it and record the request.

Why we do this now: finding the rule and quantifying the excluded group shifts us from impression to evidence. The numbers constrain our hypotheses.

We can name common conditional filters. Naming helps us notice them in real time. This list is short — the rest of the piece will refer to these categories as we practice.

• Outcome conditioning: looking only at those who succeeded (survivors). Example: top performers.
• Prior filtering: excluding cases before the outcome occurs (e.g., selecting only customers who bought before a promotion).
• Post‑selection filtering: using a variable that is downstream of the outcome or correlated with it as a filter.
• Convenience samples: whoever replied, whoever completed a survey, whoever showed up.
• Rule‑based exclusions: thresholds, quotas, or administrative flags.

After this list we remind ourselves: naming is not enough. We must test. Names dissolve into decisions: can we include the excluded group? Can we proxy for them? Can we at least estimate how the filter shifts the outcome?

We practice on a short example: the sales funnel micro‑scene Imagine we analyze conversion among leads who reached the demo stage. The claim: “Conversion from demo is 40% — it works.” We ask: who reached demo? If only leads that passed a prequalification were shipped to demos, and prequalification excluded many cold leads, then 40% is conditional on a prior filter. We can do two things today.

• Recompute conversion from initial contact to purchase (full funnel) and record both numbers.
• If raw data is unavailable, estimate via sample: take 50 recent leads, trace them through the steps, count conversions.

We do the math: if 1,000 leads contacted → 200 reached demo → 80 converted, demo conversion is 80/200 = 40%, full funnel conversion is 80/1000 = 8%. The interpretation changes. The demo stage is effective for those who pass prequalification, but overall funnel performance is different. Putting numbers in front of us makes clear the trade‑offs: improving demo scripting might increase 40% to 50% (conditional), but improving lead quality or prequalification might shift the full funnel from 8% to 12% — a relative change of 50% in total conversions versus a relative 25% change at demo stage.

A daily reflex: three short questions (time: 2–5 minutes)
When faced with a conditional result we say, aloud or in the journal: “(1) Who is included? (2) What was the selection rule? (3) How would this number change if we included the excluded cases?” We record quick answers in Brali LifeOS as a 3‑minute check‑in. This builds the habit.

From reflex to habit — the mental model we use We will think in terms of two sets: included (I) and excluded (E). The reported metric is typically M(I). We want to estimate M(all) or at least bound it: M(all) lies between M(I) and M(E), weighted by their sizes. So a simple calculation we do is:

M(all) = (|I| * M(I)
+ |E| * M(E)) / (|I| + |E|)

This requires |I| and |E| (counts)
and an estimate for M(E). If we cannot measure M(E), we consider plausible bounds: worst case and best case. This reduces speculative certainty. If M(I) = 40% and |I| = 200, |E| = 800, then:

• Best case for all: assume M(E) = M(I) = 40% → M(all) = 40%
• Worst plausible case: assume M(E) = 0% → M(all) = (20040% + 8000%)/1000 = 8%
• A more realistic middle: M(E) = 10% → M(all) = (20040% + 80010%)/1000 = (80 + 80)/1000 = 16%

These bounds tell us whether the conditional claim would matter if generalized. They also guide where to invest effort: improve M(E) (the excluded group) or change the selection rule.

A lunchtime scene: prototyping a small test At 12:40 we set up a micro‑experiment. We decide to include 20 random individuals from the excluded group E in the next week's demo invites. We will measure conversion there. This is a scaled, low‑risk way to test whether the conditional result holds beyond I. We budget 30 minutes to prepare the invite, and 20 minutes to log the plan in Brali LifeOS with a scheduled check‑in in 10 days.

We assumed X → observed Y → changed to Z: We assumed demos were fine for everyone → observed that demos were only offered to those prequalified → changed to Z: include a small random sample from the excluded group to test conversion. The pivot is explicit and reversible.

Trade‑offs: why not always include everyone? Randomly including previously excluded cases often reduces average performance and increases cost. If demos take 60 minutes of a senior rep's time and cost $100 per rep hour, including low‑probability leads increases cost per conversion. So we quantify: if a demo costs $100 and conversion among I is 40% (expected cost per conversion = $100 / 0.4 = $250), but conversion among E might be 10% (expected cost per conversion = $100 / 0.1 = $1000), then the expected cost multiplies by 4. That makes the trade‑off explicit and actionable: we can only expand if the lifetime value exceeds these numbers.

Estimate demo cost and conversion:

• Demo time: 60 minutes = $100 rep cost (example).
• Conversion I: 40% → cost per conversion = $250.
• Conversion E (assume): 10% → cost per conversion = $1000.
• If lifetime value per customer is $2,500, both groups are profitable but including E increases marginal cost per conversion by $750. We can accept or reject that based on budget.

Small numbers like minutes and dollars make the decision real. They anchor emotional responses: relief when numbers show feasibility; frustration when they don't.

Micro‑policy: a 3‑step inclusion test (≤1 hour to set up)

We set k = 20 when time or cost is small; k = 100 when we need more precision. With k = 20 and an observed M(sample) of 15%, the standard error roughly is sqrt(p*(1-p)/n) ≈ sqrt(0.15*0.85/20) ≈ 0.08 = 8 percentage points. So our estimate will be noisy but directional. That tells us whether to scale the test.

Random does not mean perfect. We often use convenience random: first 20 people in a list are often nonrandom. If we cannot generate a random sample, we at least stratify by a clear variable (geography, tenure) and note the bias. The key is transparency: record the sampling method in Brali LifeOS and note limitations.

Mini‑App Nudge Set a Brali LifeOS micro‑task: “Randomly sample 20 from excluded group E and schedule pilot invites” with a 10‑day follow‑up check‑in. This creates the small, repeatable ritual we need.

When conditioning is subtle: the case of post‑selection Sometimes the filter is downstream. For example, we analyze “job satisfaction among people promoted in the last year.” Promotion is influenced by performance and visibility; measuring satisfaction among promotees conditions on a variable correlated with both performance and other unobserved factors (mentorship, opportunities). The result is not representative of all employees. Our action today is to look for variables that are causally downstream of the outcome. We can ask: “Could the selection itself be affected by the traits we measure?” If yes, we need different comparisons (e.g., compare promoted to matched nonpromoted peers).

A practical matching exercise (30–90 minutes)

We assumed X → observed Y → changed to Z: We assumed promotion group represented all employees → observed that promotion is correlated with tenure and rating → changed to matching on those variables to measure the residual association with satisfaction. The pivot reduces confounding.

If original difference in satisfaction was 0.8 points on a 5‑point scale and matching reduces it to 0.3, selection accounts for (0.8 − 0.3)/0.8 = 62.5% of the difference. That number helps us decide whether to change promotion policy or not.

• When excluded cases are unobservable: Sometimes we have no access to E at all (deleted records, privacy limits). Then we use bounding: set plausible extremes (M(E)=0% or 100%) to see if the result is robust. If the conclusion flips under plausible extremes, we act cautiously.
• When selection is deliberate: If selection is a strategic decision (e.g., only high‑quality applicants admitted), then conditional results are the point. We must be explicit: “We analyzed the effect among admitted only.” The habit is to avoid implicit generalization: do not say “this works” without adding “for the admitted population.”
• Small samples: When I is small (n < 30), the estimate M(I) is noisy. Treat findings as tentative, run more data collection.
• Incentive traps: Sometimes rewards depend on the conditional result. If a team is rewarded for top decile metrics, they may artificially alter who gets counted. We need audit checks.

Practical journaling prompts (5–10 minutes)
In Brali LifeOS, we write one paragraph answering:

• What condition is used here?
• Who is excluded, and why?
• What small test will we run to check generality? This practice accelerates the habit; writing forces us to state assumptions.

A hardware store scene: a physical product example We visit a small retail chain interested in shelf layout. They observed that products placed at eye level sold 30% more among the featured SKUs. They propose moving all SKUs to eye level. We ask: which SKUs were featured? If the chain featured fast‑moving items already, eye level may merely reflect an existing selection. We count featured SKUs (|I|=50) and nonfeatured SKUs (|E|=450). The observed lift is 30% among I. If we estimate a modest lift of 5% among E, the overall effect is:

M(all) = (50*(1+0.30)Savg + 450(1+0.05)Savg) / (500Savg) ≈ weighted average giving a 7–8% aggregate lift, not 30%. The operational cost of reorganizing shelves might outweigh an aggregate 7% lift. We recommend a targeted pilot: relocate 50 items from E to eye level and measure for 8 weeks. We also compute a Sample Day Tally (below) for the pilot.

Goal: test moving 50 E items to eye level for 7 days and measure change in units sold.

• Baseline average units per SKU per day in E: 2 units → 2 * 50 = 100 units/day.
• Expected lift if eye level gives 5% on E: additional 0.1 units per SKU/day → +5 units/day.
• Cost of shelf rearrangement per SKU: 2 minutes per SKU → 100 minutes total (1.67 hours).
• Staff cost: $20/hour → repositioning cost ≈ $33.
• Incremental revenue per unit: $10 → expected daily incremental revenue = 5 units * $10 = $50.
• Payback: repositioning cost ($33) paid back in <1 day at the expected lift.

These concrete numbers make the decision actionable. If, instead, we had assumed a 30% lift across all 500 SKUs, we would have projected a much larger gain and possibly misallocated effort.

We write short, clear statements:

• Always include the denominator: “Among customers who used feature X (n=2,500), retention was 65%.”
• State selection rule: “Selection: customers who signed up in Q1 and completed onboarding within 14 days.”
• Provide bounds for generalization: “If excluded customers had 0% retention, overall retention would be 16%; if they had the same 65%, overall retention would be 65%.”
• State what test we will run: “We will randomly invite 400 excluded customers to onboarding next month and measure retention after 90 days.”

If we present this way, we lower the risk of overgeneralization and make next steps clear.

A weekly scene: integrating this into planning At our weekly planning meeting, we allocate 15 minutes to review any conditional claims. We ask teams to bring the denominator and selection rules. We keep a shared table in Brali LifeOS with three columns: Claim, Condition, Planned test. Each Week we tick off tests that started. Over four weeks, the table looks like a small experiments ledger.

We tracked this practice in three teams over 12 weeks. Teams that applied the inclusion test reduced rerun projects by 34% and avoided at least two misapplied rollouts that would have cost an estimated $25k in wasted effort. Those numbers are examples drawn from internal prototypes but suggest a plausible return. The lesson: early interrogation saves downstream cost; the cost of asking three questions is a few minutes; the potential benefit can be thousands in prevented missteps.

A constrained alternative: the ≤5‑minute path (for busy days)
If we have ≤5 minutes:

This tiny alternative preserves the habit under time pressure and keeps the issue visible.

• “If top performers all did X, X caused performance.” Not necessarily. Correlation may reflect selection. We address this by testing X in a randomized or matched sample.
• “If someone filtered the data, then the result is invalid.” Not always. Conditional results can be purposeful and useful — but they must be labeled and tested before scale.
• “Small samples are useless.” Small samples are noisy but informative. Use them as directional checks and report uncertainty.

• Confirmation bias: we tend to explain away exclusions that contradict our preferred story. Mitigation: assign a teammate to play the skeptic and require that person to write one paragraph listing alternative explanations.
• Incentive distortion: metrics tied to rewards create temptation to manipulate selection. Mitigation: introduce audit checks and random audits of selection procedures.
• Data unavailable: if E is inaccessible, use bounding and be conservative in claims. Mitigation: push for minimal logging (a simple count) even if content is private.

A reflective scene: when numbers make us uncomfortable We ran a pilot to include 50 previously excluded leads in demos. Conversion was 12% versus 40% for the included group. The team reacted with disappointment and relief. Disappointment because the method wasn’t a one‑size‑fits‑all. Relief because we avoided a large, expensive rollout. We logged the result in Brali LifeOS, wrote one paragraph reflecting on selection mechanisms (lead source mattered a lot), and planned a follow‑up experiment focused on improving prequalification and lead nurture.

The pivot was explicit: we assumed demos were universally beneficial → observed a lower conversion among formerly excluded leads → changed to a two‑track approach: maintain demos for prequalified leads; invest in lead nurture for the rest.

Scaling the habit: how we embed it in team routines

• Onboarding: new analysts get a one‑page checklist: “When reporting M(I), include denominator, selection rule, and one planned test.”
• Templates: our report templates require a “Selection and Generalizability” box with three lines.
• Retrospective: in postmortems, include a “Selection check” to see whether conditioning shaped decisions.

Sample Day Tally (Personal Example: health data)
We use a personal health example to make numbers relatable. Imagine we tracked sleep among days we exercised vs. days we didn’t. We found “sleep quality 8.2/10 on exercise days.” We must ask: were we only exercising on days we felt good already? Excluded days might be bad mood or illness days.

Goal: estimate average sleep quality across all days.

• Count days in month: 30.
• Exercise days (I): 12 → avg sleep = 8.2.
• Non‑exercise days (E): 18. We sample 6 non‑exercise days and get avg sleep = 6.8.
• M(all) ≈ (128.2 + 186.8)/30 = (98.4 + 122.4)/30 = 220.8/30 = 7.36.
• Interpretation: the 8.2 among exercise days overstated the monthly average by 0.84 points.

This simple tally took 15 minutes: pull calendar, sample days, compute the weighted average. It shows how conditioning can mislead even in our daily life.

Another micro‑scene: academic reading We read a paper that reports survival rates among patients who underwent a novel surgical technique. The reported survival is 85% among operated patients. The condition is clear: operated patients only. We ask: were patients selected for surgery because they were healthier? We look for a flowchart: how many were screened, how many excluded, reasons for exclusion. If the paper lacks that, we mark it as “limited generalizability” in our notebook and look for complementary evidence.

When we predefine analyses for an experiment, we should specify inclusion/exclusion criteria up front. A preanalysis plan avoids post‑hoc cherry picking. It should state:

• The population.
• Rules for excluding cases.
• Handling of missing data.
• Planned subgroup analyses.

If we run an A/B test and later notice that one variant caused different dropout rates, we examine whether conditioning on post‑treatment variables invalidates comparisons. The habit is to predefine and then document any deviations.

• When you next see a conditional claim, spend 5 minutes to write the denominator and selection rule in Brali LifeOS.
• If possible, count excluded cases and compute a simple bound for M(all) using M(I) and a plausible M(E).
• If you manage a process that applies a filter, schedule a 1‑hour pilot to test inclusion of a random sample from E.

We will practice this now: schedule the Brali task “Selection check — next conditional claim” and set a 10‑minute timer.

Quantifying uncertainty: simple rules to present ranges

• If you have no estimate for M(E): present M(all) bounds assuming M(E) = 0% and M(E) = M(I).
• If you have a small sample estimate for M(E): compute the standard error and present ±1 SE as an uncertainty band. For n=20 and p=0.15, SE ≈ 8 percentage points. Report 15% ±8pp.
• Always show counts (n) alongside percentages. 40% (n=200) is more informative than 40% alone.

We practiced on three cases: a top performers memo, a demo conversion, and a retail layout. Each time, the conditioned effect shrank when generalized. In two cases we avoided large rollouts; in one case we found a targeted subset where scaling was profitable. The behavioral pattern is consistent: interrogating selection reduces premature scaling and channels experiments toward including excluded groups.

Check‑in Block (integrate into Brali LifeOS)
Daily (3 Qs):

• Which conditional claim did we encounter today? (one sentence)
• What was the denominator and selection rule? (brief)
• Did we take one action to test generality (yes/no)? If yes, what? (one line)

Weekly (3 Qs):

• How many conditional checks did we perform this week? (count)
• How many small inclusion pilots did we run? (count)
• What was the largest change in interpretation after testing? (short description with numbers)

Metrics:

• Count: number of conditional claims inspected (daily/weekly).
• Minutes: time spent on pilots or checks (daily/weekly).

One simple alternative path for busy days (≤5 minutes)

• We will have recorded at least 10 conditional checks in Brali LifeOS.
• At least 2 small inclusion pilots started (k ≥ 20).
• We will have one concrete decision changed or avoided because of these checks.
• We will reduce the number of untested rollouts by a measurable fraction (target: 30% reduction in untested changes).

A final reflective scene: the habit at scale We picture a team meeting six months from now. A junior analyst presents a finding and begins, “Among customers who completed onboarding within 14 days (n=2,450), churn at 6 months is 12%.” The team nods, and someone asks, “How many were excluded, and what’s our plan to test their churn?” The analyst answers, “Excluded were 5,500; we will randomize 300 to a targeted nurture flow.” No surprise. This is the habit we trained: an initial tightening followed by thoughtful expansion. It feels less dramatic than a miracle claim, but it reduces costly mistakes, respects uncertainty, and keeps us curious.

• Stop: pause for 60–120 seconds.
• Identify: write denominator and selection rule.
• Count: get |I| and |E| or request counts.
• Bound: compute plausible extremes for M(all).
• Test: schedule a k‑sample pilot (k = 20–100) or match controls.
• Record: log everything in Brali LifeOS.

Mini‑App Nudge (again, short)
Create a Brali micro‑task named “Selection check” with a 3‑question daily check‑in and a 10‑day follow‑up. Use it three times this week.

Check‑in Block (repeat for clarity near the end)
Daily (3 Qs): [sensation/behavior focused]

• What conditional result did we see today? (one sentence)
• Who was included and who was excluded? (counts if available)
• Did we take one small action to test it (yes/no)? If yes, what?

Weekly (3 Qs): [progress/consistency focused]

• How many conditional claims did we inspect this week? (count)
• Which tests started this week? (list with k and expected end date)
• What did we learn that changed a decision? (short, with numbers)

Metrics:

• Count: number of conditional checks logged.
• Minutes: time spent on pilots/checks.

We will practice this today: pick the next conditional claim you see, spend 5–15 minutes on the “Identify → Count → Bound → Test” loop, and record it in Brali LifeOS.