# Instrument-Fidelity Study — Conversational vs. Standardized Administration of UCLA-3, PHQ-2, PHQ-9, and GAD-7

**DRAFT protocol — for IRB and clinical-director review. Not effective until reviewed, amended, and signed.**

- Program: Boojee Companion Care (AI companion for socially isolated older adults)
- Document: `fidelity-study-protocol.md`
- Version: DRAFT-1 / 2026-07-15
- Study type: Within-subjects, counterbalanced **equivalence / agreement** study (measurement-comparison design)
- Reporting standard: This protocol is structured against the **SPIRIT 2013** 33-item checklist where applicable; the eventual manuscript will follow **STROBE** (observational / measurement-agreement design) and the GRRAS recommendations for reliability & agreement studies.
- Registration: Measurement-agreement studies are not always trial-registrable, but this study **will be pre-registered** (OSF) with a locked analysis plan before data collection.

---

## 0. Why this study exists (the objection it answers)

The single most important methodological objection to an AI companion that "uses validated instruments" is:

> The instruments (UCLA-3, PHQ-2, PHQ-9, GAD-7) were validated under **standardized self-administration** — a fixed-order, fixed-wording, self-report form. The companion administers them **conversationally**, in natural dialogue, sometimes across turns, sometimes reworded for comprehension. **A score obtained that way is not the validated score.** You cannot inherit the instrument's psychometric evidence unless you show your administration mode reproduces it.

This is a real and correct objection. **Score equivalence is not assumed; it must be demonstrated.** This study demonstrates (or fails to demonstrate) it. Until it passes, the program must describe conversational scores as *screening-informed indicators*, **not** validated instrument scores, and must not claim the instruments' published sensitivity/specificity for its conversational output.

---

## 1. Objectives & hypotheses

### 1.1 Primary objective
Determine whether the companion's **conversational administration** of each instrument produces total scores **equivalent** to the same instrument's **standardized self-administration** in older adults.

### 1.2 Primary hypothesis (per instrument, equivalence framing)
H₀ (non-equivalence): |μ_conversational − μ_standard| ≥ Δ
H₁ (equivalence): |μ_conversational − μ_standard| < Δ

where Δ is the pre-specified equivalence margin (Section 5). Equivalence is concluded only if the **two one-sided tests (TOST)** 90% CI for the mean paired difference falls entirely inside (−Δ, +Δ) **and** the agreement criteria (Section 6) are met. We test equivalence, not "no significant difference" — absence of a significant difference is not evidence of agreement.

### 1.3 Secondary objectives
- Item-level agreement (weighted κ per item) to locate *which* items drift under conversation.
- Classification concordance at published cut-points (UCLA-3 ≥6 positive; PHQ-2 ≥3; PHQ-9 bands 5/10/15/20; GAD-7 bands 5/10/15) — a member's **category** (screen positive/negative, severity band) matters more operationally than the raw point.
- Safety-item fidelity: **PHQ-9 item 9** (self-harm ideation) concordance analyzed separately and reported with exact discordant cases. A false-negative on item 9 is a safety event, not a rounding error (Section 8).
- Order/carryover effects (does administration order change scores?).
- Acceptability/comprehension differences between modes (older-adult usability).

---

## 2. Design

- **Within-subjects, two conditions:** (A) Standardized self-administration (validated reference), (B) Conversational administration by the companion.
- **Counterbalanced order:** participants randomized 1:1 to A→B or B→A to control practice/carryover.
- **Washout:** minimum **48 hours to 7 days** between the two administrations of the same instrument — long enough to reduce recall of specific answers, short enough that true state (loneliness/mood/anxiety) is stable. Washout window is a pre-specified sensitivity variable; sessions outside the window are flagged.
- **Same reference period:** both modes use the instruments' native recall windows (PHQ/GAD "over the last 2 weeks"; UCLA-3 present-tense). Scheduling keeps the underlying construct stable across the pair.
- **Blinded scoring:** conversational transcripts are scored by trained human raters blinded to the participant's standardized score, using a pre-written scoring rubric (Section 4.3). The companion's *automated* score is captured separately, so we can compare **three** things: standard self-report, human-rated conversational score, and machine-derived conversational score. Machine-vs-human agreement is itself an endpoint.

Rationale for within-subjects: agreement/equivalence questions require **paired** measurements on the same person; a between-subjects design cannot estimate ICC or limits of agreement.

---

## 3. Population

### 3.1 Target
Community-dwelling **older adults (age ≥60)**, English-speaking, representative of the intended program population (socially isolated, may have mild sensory/cognitive limits).

### 3.2 Inclusion
- Age ≥60.
- Able to complete a self-administered questionnaire (with standard large-print/audio accommodation).
- Capacity to consent (Section 9; brief capacity screen).
- Willing to complete both administration modes.

### 3.3 Exclusion
- Moderate–severe cognitive impairment that precludes valid self-report (e.g., documented dementia; or fails a brief cognitive screen — MMSE/MoCA threshold to be set by clinical director). *Note:* this bounds the claim — fidelity is established for the cognitively-intact-enough population, not for advanced dementia.
- Active suicidal crisis at screening → not enrolled; referred per safety plan (crisis care takes precedence over research).
- Acute psychiatric instability.

### 3.4 Recruitment
Partner Area Agencies on Aging (AAAs), senior centers, and (if a plan pilot is live) enrolled members who separately consent to the sub-study. No cold recruitment of members in crisis.

---

## 4. Procedures

### 4.1 Instruments (all public-domain, no license fee)
- **UCLA-3** — 3-Item Loneliness Scale (Hughes et al., 2004). Score 3–9; ≥6 = positive.
- **PHQ-2** — 2-item depression screen; ≥3 triggers PHQ-9.
- **PHQ-9** — 9-item depression severity (Kroenke, Spitzer, Williams, 2001); item 9 = self-harm.
- **GAD-7** — 7-item anxiety severity (Spitzer et al., 2006).

### 4.2 Standardized condition (reference)
Instruments administered exactly as validated: fixed item order, verbatim wording, standard response options, self-completed on paper or a locked digital form with no conversational reformatting. Large-print / screen-reader accommodations allowed (these are validated accommodations, not rewording).

### 4.3 Conversational condition
The companion administers the same items **in dialogue** per the production script. A locked **scoring rubric** maps each item's response options to conversational equivalents (e.g., "hardly ever / some of the time / often" for UCLA-3; "not at all → nearly every day" 0–3 for PHQ/GAD). Rubric is version-pinned to the exact production prompt build under test; the build hash is recorded so results are attributable to a specific model+prompt version. **Any change to the production administration prompt invalidates a prior fidelity result and requires re-testing** (Section 11).

### 4.4 Session flow
1. Consent + capacity screen + demographics.
2. Randomize to order A→B or B→A.
3. First administration (all four instruments in that mode).
4. Washout window.
5. Second administration (other mode).
6. Post-session acceptability/comprehension items + debrief.
7. Any positive safety trigger in **either** mode handled live per safety plan (Section 8), independent of research flow.

---

## 5. Equivalence margins (pre-specified, per instrument)

Margins are set from the smallest change that is clinically/operationally meaningful, so "equivalent" means "differs by less than we'd care about." Draft values below are **for clinical-director ratification** — they must be signed before data collection.

- **UCLA-3** (range 3–9): Δ = **1.0 point**. One point on a 3-item scale is the smallest movement that can flip screen status; the margin should be tighter than the effect we later want to detect in outcomes.
- **PHQ-2** (range 0–6): Δ = **1.0 point**.
- **PHQ-9** (range 0–27): Δ = **2.0 points**. The literature's *reliable-improvement* threshold on PHQ-9 is ~5–6 points; an administration-mode difference must be materially smaller than that or it would masquerade as/hide real change. Δ=2 is deliberately conservative.
- **GAD-7** (range 0–21): Δ = **2.0 points** (same logic as PHQ-9).

Margins are symmetric. Equivalence requires the 90% CI of the mean paired difference to lie fully within ±Δ (TOST at α=0.05 each side).

---

## 6. Agreement criteria (must pass in addition to TOST)

Equivalence of *means* can hide poor *individual* agreement. We therefore also require:

- **Intraclass Correlation Coefficient (ICC)** — two-way mixed, absolute-agreement, single-measures — with 95% CI. Pre-specified success: **ICC lower 95% CI ≥ 0.75** (good) per total score; ICC ≥0.90 reported as excellent.
- **Bland–Altman** limits of agreement (mean difference ± 1.96·SD_diff), plotted, with the LoA compared against the equivalence margin. Pre-specified: the 95% LoA should not exceed roughly ±Δ (interpretation caveat in analysis plan). Proportional bias assessed by regressing difference on mean.
- **Weighted κ** (quadratic) per item and for severity-band classification, with 95% CI. Pre-specified: κ ≥ 0.60 (substantial) for classification concordance; item-level κ reported for all items.
- **Classification concordance %** at each published cut-point with 95% CI.

A result is reported as "**equivalent / fidelity supported**" only if TOST passes **and** ICC and classification-κ criteria pass. Partial results (e.g., PHQ-9 total equivalent but item 9 discordant) are reported honestly and gate the claim accordingly.

---

## 7. Sample size & power

Justification is provided per statistical target; the final N is the **maximum** across targets, plus attrition allowance.

### 7.1 For the ICC criterion (typically the binding constraint)
To show ICC with adequate precision under a two-way, single-measure absolute-agreement model, targeting an expected ICC ρ₁ = 0.85 tested against a null of ρ₀ = 0.75, with 2 observations per subject, α = 0.05 (one-sided), power = 0.80: required N ≈ **90–110 participants** (exact value computed via the Zou / Bonett ICC sample-size method in the finalized SAP; the range reflects the assumed ICC).

### 7.2 For the TOST equivalence criterion
With Δ = 2.0 on PHQ-9, an assumed true mean difference of 0 and SD of paired differences ≈ 2.5, α = 0.05, power = 0.90: required N ≈ **35–45 paired observations** — smaller than the ICC requirement, so not binding. Tighter-margin instruments (UCLA-3 Δ=1.0) with smaller SD_diff are re-checked; if any instrument's TOST requirement exceeds the ICC-driven N, the larger governs.

### 7.3 For weighted-κ precision
To estimate a κ near 0.70 with a 95% CI half-width ≤ ~0.12 requires roughly **90–120** subjects depending on marginal distributions — consistent with the ICC-driven target.

### 7.4 Planned enrollment
**Target N = 120 completers**, enroll **N = 140** to allow ~15% attrition/exclusion. Final numbers ratified in the Statistical Analysis Plan (SAP) before data lock. Underpowering is a stated risk if recruitment lags; the study will not be reported as "fidelity established" if it fails to reach the pre-specified N — it will be reported as an underpowered pilot.

---

## 8. Safety within the fidelity study

Even though this is a measurement study, participants complete real depression/anxiety/loneliness screens, so **real risk can surface**.

- **PHQ-9 item 9 > 0** or any suicidal statement, in **either** mode, triggers the program's standing crisis SOP immediately: 988 Suicide & Crisis Lifeline provided (call/text 988), live human clinician notified, participant not left alone in the flow. Crisis response overrides research procedure.
- Item-9 discordance between modes (positive in one, missed in the other) is a **reportable analysis outcome** and, if the conversational mode *missed* a positive, a **safety signal** reported to the IRB regardless of statistical results. Fidelity of the safety item is a gating requirement, not an average.
- All crisis events logged and reported to the IRB per the Safety Monitoring Plan (`safety-monitoring-plan.md`).

---

## 9. Consent & capacity (older adults)

- Written informed consent (large-print; verbal walkthrough available).
- Brief **capacity-to-consent** check (teach-back of purpose, voluntariness, risks). Anyone failing is not enrolled and, if appropriate, referred.
- Voluntary; withdrawal any time without affecting their program benefits or care.
- Consent covers transcript scoring by blinded human raters and storage of de-identified transcripts.
- Full elements per Common Rule (45 CFR 46.116). See Safety Monitoring Plan for details.

---

## 10. Analysis plan (locked pre-collection)

1. Descriptives per mode (mean, SD, distribution, floor/ceiling) per instrument.
2. Paired differences (conversational − standard); TOST at each Δ; report 90% CI.
3. ICC (2-way mixed, absolute agreement, single measures) + 95% CI.
4. Bland–Altman: bias, 95% LoA, proportional-bias regression, plots.
5. Weighted (quadratic) κ per item and for severity classification; classification concordance %.
6. Machine-vs-human conversational scoring agreement (ICC/κ) — validates automated scoring separately from the administration mode.
7. Order/carryover: mixed model with mode, order, and mode×order.
8. Subgroup (pre-specified, exploratory, not powered): age band, sensory-impairment, cognitive-screen tier.
9. Missing data: report rates; primary analysis on complete pairs; sensitivity analysis for missingness.

All analyses per the pre-registered SAP; deviations reported.

---

## 11. What each outcome permits the program to claim

- **Full pass (all instruments, incl. item 9):** "Conversational administration of UCLA-3/PHQ-2/PHQ-9/GAD-7 by the companion produces scores equivalent to standardized self-administration in older adults (ICC, TOST, κ reported), for production build `<hash>`." Program may present conversational scores **as** the validated screens for that build.
- **Partial pass:** claim limited to the instruments/uses that passed; failed items/instruments revert to standardized self-administration in production (the `/screening/` flow already exists for this).
- **Fail:** conversational output is labeled a *screening-informed indicator*, standardized self-administration is used for any scored screen, and the instruments' published operating characteristics are **not** claimed for conversational output.
- **Every production prompt/model change** re-opens fidelity — results are pinned to a build hash; a change requires at minimum a bridging re-test.

---

## 12. Honest limitations

- Establishes **measurement agreement**, not clinical outcome — says nothing about whether the companion *helps*. That is the pilot and RCT (`outcomes-pilot-protocol.md`, `rct-design-outline.md`).
- Excludes moderate–severe cognitive impairment; fidelity is not established there.
- Human raters approximate a "true" conversational score; the rubric is a construct, and inter-rater reliability (also reported) bounds it.
- Single-language (English) v1; other languages require their own fidelity study.
- Findings are build-specific and expire on model/prompt change.

---

## 13. Governance

- IRB review + approval before any enrollment.
- Pre-registration (OSF) with locked SAP.
- Clinical-director ratification of equivalence margins (Section 5) and cognitive-screen thresholds (Section 3.3) before data collection.
- Continuing review per IRB determination; safety events per `safety-monitoring-plan.md`.

---

### Sign-off (required before effective)

- Principal Investigator (name / credentials): ______________________
- Licensed Clinical Director (name / license # / state): ______________________
- Biostatistician (SAP author): ______________________
- IRB approval # / date: ______________________
- Pre-registration DOI: ______________________

*Version DRAFT-1 / 2026-07-15. For IRB and clinical-director review. Not effective until signed.*