Studying People and Organizations with Scientific Rigor

How to

1. 1Match the method to the question: 'how' and 'why' questions favor case studies and experiments; 'what' and 'how many' favor surveys.
2. 2Where you can manipulate and assign, use random assignment to create comparable groups and manipulate one independent variable while holding others constant.
3. 3Where you can't randomize, build in structural design elements and explicitly identify, operationalize, and test rival explanations for your finding.
4. 4Ensure an adequate subject-to-variable ratio and enough statistical power to detect an effect worth detecting before you commit to a design.
5. 5Screen data for accuracy, missing values, and outliers, and check the assumptions of your intended analysis (linearity, normality where required) before the main analysis.
6. 6For qualitative work, use multiple sources of evidence and maintain a chain of evidence an outsider could trace from question to conclusion.

Watch out for

• —Confounding: your variable of interest quietly co-varies with an unintended one, offering an alternative explanation you never tested.
• —Reactivity and demand characteristics — people behave differently when observed, or read cues about the 'right' answer.
• —Treating a large dataset as a substitute for a sound design; more rows do not remove selection bias.
• —Capitalization on chance: models tuned to one sample that fail to replicate; validate before trusting.

How to

1. 1Report internal consistency (coefficient alpha) and, where feasible, test-retest reliability for any scale you rely on.
2. 2Build enough construct-relevant items — items expressing the same underlying idea in genuinely different ways — to sample the domain adequately.
3. 3Check inter-rater agreement whenever human judgment (e.g., interview ratings, performance appraisals) enters the measure.
4. 4Treat measurement error as real and unavoidable: prefer multiple indicators over single items for important constructs.
5. 5Aim for reliability of roughly .80 or better for scales used to make consequential decisions.

Watch out for

• —Confusing internal consistency with unidimensionality — items can correlate for the wrong reasons; establish that they reflect a single latent variable first.
• —Inflating alpha with near-duplicate item wording rather than genuine content redundancy.
• —Ignoring that low reliability drags down every correlation the measure participates in.

How to

1. 1Begin with a clear, theoretically grounded definition of the construct, specifying what is inside and outside its domain, before writing a single item.
2. 2Establish content validity through expert judging and item generation grounded in the literature.
3. 3Test the construct's expected dimensionality (unidimensional or specified multidimensional) before claiming reliability or validity.
4. 4Gather criterion evidence: does the measure predict the outcome it theoretically should (e.g., does a selection test predict later performance)?
5. 5Demonstrate convergent and discriminant validity by correlating with related and unrelated constructs as theory predicts.
6. 6Re-validate when you move the instrument to a new population, language, or purpose.

Watch out for

• —Skipping construct definition and reverse-engineering meaning from whatever the items happened to capture.
• —Systematic (non-random) error — a measure consistently representing something other than the intended concept (e.g., social desirability), which reliability statistics won't catch.
• —Mistaking factor-analytic structure for substance without theoretical guidance — method artifacts can masquerade as constructs.
• —Assuming a borrowed, published scale is automatically valid in your very different setting.

How to

1. 1State your causal assumptions explicitly (even a simple diagram of what you think causes what) before interpreting relationships.
2. 2Distinguish, in writing, statistical significance from practical significance from causal inference — they are three separate claims.
3. 3For quantitative claims, confirm covariation, correct time order, and the absence of plausible confounders before using causal language.
4. 4For qualitative claims, use constant comparison, write analytic memos, and pursue theoretical sampling until categories saturate.
5. 5Practice reflexivity: examine how your own assumptions shaped what you saw and concluded.
6. 6Generalize deliberately — to theory (analytic generalization) or to a defined population (statistical generalization) — and say which you're doing.

Watch out for

• —Collider and confounding bias: controlling for the wrong variable can manufacture a spurious relationship or hide a real one.
• —Assuming variability across studies is real signal — much of it is artifact from measurement error and sampling error until proven otherwise.
• —Treating causality as a statistical output rather than a theoretical assumption you are responsible for.
• —Overgeneralizing from a convenient sample to the whole workforce.

How to

1. 1Establish a single version of the truth: consolidate, cleanse, standardize, and integrate people data before scaling any analytics.
2. 2Let the business question drive which data you need, rather than analyzing whatever happens to be available.
3. 3Combine data types — internal and external, structured and unstructured — to get a fuller picture of a people question.
4. 4Collect only essential data, anonymize where possible, and be transparent with employees about how their data is used.
5. 5Treat data stewardship as an ongoing management responsibility, not a one-off cleanup.

Watch out for

• —Chasing volume over relevance and drowning in trivial metrics.
• —Building analytics on top of unreconciled source systems that disagree with each other.
• —Eroding employee trust through opaque or intrusive data use — hard to build, easy to destroy.
• —Treating data preparation as a minor step; in practice it dominates the honest work.

How to

1. 1Locate your organization honestly on the descriptive-to-prescriptive maturity continuum and set the next realistic rung, not the last one.
2. 2Start with a well-defined business question tied to a specific outcome (the dependent variable) before touching data.
3. 3Choose the regression or analytic method by outcome type and data structure — and validate model assumptions before declaring results valid.
4. 4Build the translator capability: someone who can move fluently between the analysis and the decision-maker.
5. 5Think big but start small — pursue high-impact, low-effort quick wins to build credibility before attempting enterprise-scale work.
6. 6Prefer inference over raw prediction where decisions are consequential and samples are small.

Watch out for

• —Reinventing the wheel — failing to review existing knowledge before hypothesizing.
• —Correlation-driven insight dressed up as causal explanation.
• —Over-fitting: models tuned to one dataset that fail on the next.
• —Institutionalized metric-oriented behavior — optimizing whatever is measured rather than what matters; balance indicators against each other.

How to

1. 1Frame every analysis to end in a decision — always ask 'so what?' and state the action the finding implies.
2. 2Secure a fact-based senior sponsor early who will model and resource evidence-based decisions.
3. 3Tell stories, not statistics: communicate findings in the decision-maker's terms to drive change.
4. 4Use checklists, structured procedures, and base rates in high-stakes judgments to reduce bias and increase consistency.
5. 5Reduce uncertainty just enough to inform the decision — measure to the point where the information's value justifies the cost, then act.
6. 6Embed analytics into routine management processes so evidence 'teaches rather than tells.'

Watch out for

• —Producing insight nobody adopts — the overhead trap.
• —Letting anchoring and framing shape the decision under the appearance of judgment.
• —Demanding statistical certainty before acting when business intelligence and directional evidence would suffice.
• —Sponsors who endorse analytics rhetorically but override it whenever it contradicts their prior view.

How to

1. 1Start with a rigorous job analysis and a clear, behaviorally defined competency model as your selection criteria.
2. 2Define and prioritize a limited set of essential candidate criteria before launching the search.
3. 3Use structured interviews with consistent questions and rating scales rather than free-form conversations.
4. 4Combine multiple valid methods (ability, structured interview, work sample) to capture incremental validity.
5. 5Front-load hiring rigor and use committee-based, structured decisions to reduce individual bias — and hire people better than current staff.
6. 6Check every tool for fairness and adverse impact, and keep the process legally defensible.

Watch out for

• —Adverse impact and legal exposure from unvalidated or biased tools.
• —Halo in interviewers — one salient trait coloring the whole judgment.
• —Optimizing for 'culture fit' in ways that smuggle in bias rather than job-relevant fit.
• —Assuming a validated test transfers to a very different role or population without re-checking.

How to

1. 1Diagnose which needs are unmet before intervening — safety and belonging must be addressed before esteem and growth appeals land.
2. 2Design work to satisfy autonomy, competence, and relatedness rather than relying on external control alone.
3. 3Distinguish the direction, intensity, and persistence of effort; a motivation problem in one is not the same as in another.
4. 4Where you use extrinsic rewards, be deliberate about incentive intensity — tie strength of reward to measurement precision and the incremental value of effort.
5. 5For skill domains, remember motivation ('ignition') combines with deliberate practice and coaching to build performance over time.

Watch out for

• —Applying a universal 'best' motivator; there is a best way for your specific context, not people in general.
• —Assuming extrinsic rewards are cost-free — they can crowd out intrinsic interest and damage relationships and creativity.
• —Treating conscious stated preferences as the whole story; surface desires can be symptoms of deeper unmet needs.
• —Reading low effort as a disposition when it may be an unmet need or a poorly designed situation.

How to

1. 1Measure engagement with reliable, validated items (apply the reliability and validity discipline from Foundations) and administer confidentially, ideally via a third party.
2. 2Analyze at the workgroup level, not just company-wide, since managers are the driver.
3. 3Address the concrete elements: make expectations clear, supply materials and equipment, cast people into strength-based roles, and deliver specific, timely recognition.
4. 4Show genuine care for people as whole persons and encourage development — the strongest discretionary-effort levers.
5. 5Make it safe to speak up and reduce anxiety-driving conditions (unrealistic workloads, opaque communication) that suppress engagement.

Watch out for

• —Treating engagement as an HR program rather than a daily managerial behavior.
• —Surveying without acting — measuring engagement and doing nothing depresses it further.
• —Using unreliable single-item measures and over-interpreting small movements.
• —Confusing satisfaction (contentment) with engagement (discretionary effort) — they are related but distinct.

How to

1. 1Define clear outcomes and expectations, then give latitude on how people reach them.
2. 2Identify each person's strengths and cast them into roles that use those strengths daily.
3. 3Deliver frequent, specific, timely recognition and demonstrate genuine care for people as persons.
4. 4Use high-leverage activities — the manager's output is the team's output, so invest where a unit of your time multiplies.
5. 5Communicate expectations explicitly (people can't read your mind) and absorb uncertainty for your team rather than passing it down.
6. 6Select and develop managers for the interpersonal and informational roles they actually perform, not for technical prowess alone.

Watch out for

• —Promoting the best individual contributor into management without regard for managerial talent.
• —Delegation without follow-through, which is abdication; monitor without meddling.
• —Spending the most time with the weakest performers rather than the best.
• —Confusing activity and busyness with leverage.

How to

1. 1Define performance in behavioral, observable terms (a competency model) separate from results the person doesn't fully control.
2. 2Recognize the components — task, contextual/citizenship, and counterproductive behavior — rather than a single global score.
3. 3Address the determinants deliberately: build declarative and procedural knowledge through development, and motivation through the levers above.
4. 4Calibrate ratings across raters to reduce leniency and halo, and treat the rating as a measure to be validated.
5. 5Use deliberate, error-focused practice and coaching to grow performance, not just to assess it.

Watch out for

• —Halo in appraisals — a general impression coloring every dimension.
• —Rewarding measurable outputs while ignoring unmeasured but valued behaviors (the equal-compensation problem).
• —Treating performance as a fixed trait rather than partly a product of situation, role fit, and management.
• —Single-source, single-item performance measures with no reliability check.

How to

1. 1Model both why people leave and why they stay, and route distal causes through proximal states (intention to leave, embeddedness).
2. 2Account for base rates, time lag, and labor-market context before interpreting any turnover prediction.
3. 3Use realistic recruitment and structured onboarding to align newcomer expectations and improve early-tenure retention.
4. 4Segment: focus retention effort on high performers and pivotal roles rather than treating all turnover as equally bad.
5. 5Treat retention as everyone's responsibility, especially the direct manager, and use engagement measures as leading indicators.

Watch out for

• —Ignoring functional turnover — some departures improve the workforce; not all attrition is a problem.
• —Attributing quits to controllable internal factors when external opportunity is the driver.
• —Unreliable intention-to-leave measures producing false confidence.
• —Applying a generic turnover model without checking that its conditions hold in your context.

How to

1. 1Connect people investments to business performance through an explicit logic (e.g., which pivotal roles, through what behaviors, affect what strategic outcome) rather than a raw correlation.
2. 2Be skeptical of 'blueprint' studies of successful firms; ask whether the described causes could be after-the-fact attributions.
3. 3Judge performance relative to competitors and over time, not as an absolute number.
4. 4Invest disproportionately where talent performance has nonlinear strategic impact (pivotal roles), not evenly across the workforce.
5. 5Where you claim a people practice drives results, meet the causal standard from the inference section — covariation, time order, ruled-out rivals.

Watch out for

• —The halo effect: attributing culture, leadership, and strategy quality to firms because you already know they succeeded.
• —Reverse causation — successful firms can afford good practices, not only the reverse.
• —Copying the current winners' practices without the strategy and conditions that made them fit.
• —Confusing a single-year result with sustainable advantage.