[FAccT'26] Do User-Aligned Explanations Steer Human Decisions? Context-Dependent Influence and Ethical Implications

FAccT26 presentation slides
Do User-Aligned Explanations Steer Human Decisions? Context-Dependent Influence and Ethical Implications
Paper URL: https://doi.org/10.1145/3805689.3812288

Transcript

1. Do User-Aligned Explanations Steer Human Decisions? Context-Dependent Influence and Ethical Implications Mingzhe Yang (The University of Tokyo) Rina Kagawa (University of Tsukuba) Yukino Baba (The University of Tokyo)
2. Personalized Explanations Can Help Understanding, but May Steer Decisions Standard AI assisted Decision-Making Process 1. User makes an initial decision 2. Review AI prediction & explanation 3. Make the final decision Rise of Personalized Explanations Expected Benefits Personalized explanations tailored to user knowledge, preferences. Aims to improve understandability and foster trust in AI systems. Hidden Risks Understandability does not mean faithfulness. Easy-to-understand explanations may be easier to accept. 2
3. Why User Alignment May Steer Decisions 3 Same AI advice can feel different depending on which criteria the explanation highlights. Explanation highlights certain criteria Criteria match the user’s own criteria AI advice feels more reasonable Decision or trust may change AI prediction unchanged — Only highlighted criteria change. Open question: Does this effect change with alignment strength, and does it depend on task context?
4. Controlling the alignment between explanations and user decision criteria Continuous control via linear combination α = 0: Fully aligned with AI criteria α = 1: Fully aligned with user criteria Alignment Strength α Shifts Explanations from AI-Aligned to User-Aligned 4
5. Procedure: Same AI Predictions, Adjusted Explanations 1. Pre Phase • Users make decisions without AI • Estimate decision criteria from user responses 2. Main Phase (RQ1, 2) • follow the standard AI- assisted decision making process • alignment strength (α) is randmized 3. Post Questions (RQ3) • Survey on ethical acceptability and need for disclosure after revealing the alignment Participants: N = 167 and 50 trials in Main Phase (10 per α) AI predictions were always the original ones; only explanations were modified. Participants were not informed about the adjustment during the main phase. 5
6. Objective Metrics (RQ1) Agreement rate • Percentage of matching the AI's advice Subjective Metrics (RQ2) 5-point Likert scale: Trust, Satisfaction, Understandability, etc. Ethical Acceptability & Disclosure (RQ3) • Ethical Acceptability of explanation adjustment • Need for Disclosure about the adjustment Calculating the Difference Scores The unadjusted condition (α = 0) is used as baseline to measure the impact of explanation adjustment. Measurements: Agreement, Trust, Ethics, and Disclosure 6
7. Task Settings: Objective, High-Stakes, and Subjective Income (objective) Predicting whether a person's annual income exceeds $50K. A task with an objective ground truth. Recidivism (high-stakes) Predicting if a defendant will reoffend within 2 years. A high-stakes task with significant legal and ethical weight. Dating (subjective) Predicting whether someone would like to date again. A subjective task heavily driven by personal preference. 7
8. Above 0 = more agreement with AI α increases → more user-aligned RQ1: User-Aligned Explanations Increase Agreement in Income and Dating User-aligned explanations can steer agreement, depending on task context. 8
9. Above 0 = more agreement with AI α increases → more user-aligned RQ1: User-Aligned Explanations Increase Agreement in Income and Dating User-aligned explanations can steer agreement, depending on task context. 9 Income (Objective) • Significant increase in agreement for α ≥ 0.75 • Even in tasks with ground truth, alignment with intuition increases agreement
10. Above 0 = more agreement with AI α increases → more user-aligned RQ1: User-Aligned Explanations Increase Agreement in Income and Dating User-aligned explanations can steer agreement, depending on task context. Dating (Subjective) • Significant agreement for α≥0.75 • Agreement increases even at α=0.25 • Steering effect achieved with minimal adjustment 10
11. Above 0 = more agreement with AI α increases → more user-aligned RQ1: User-Aligned Explanations Increase Agreement in Income and Dating User-aligned explanations can steer agreement, depending on task context. 11 Recidivism (RQ1) • No significant change in agreement rates • The decisions remain cautious
12. RQ2: Trust and Satisfaction Increased Without Decision Change Impact on Decisions (RQ1) • No significant change in agreement rates • The decisions remain cautious Decisions remain stable for Recidivism, but a risk of unconscious over-reliance on AI emerges. Subjective Evaluation (RQ2) • Significant increase in satisfaction/trust • Notable improvement at α ≥ 0.75 12 Above 0 = increased score α increases → more user-aligned
13. Ethical Acceptability High-stakes Contexts 71.8% say “disagreement ” with adjustment →Significant drop in acceptability. Subjective/Low-Stake Contexts Higher tendency for user acceptance. * Acceptance ≠ Safety. Risk of losing critical perspectives. Need for Disclosure Demands in High-stakes Contexts • 95.2% say "Disclosure is necessary" • 64.7% say "It should be mandatory" Expectation of Consistent Transparency Disclosure was demanded even in tasks where adjustments are accepted. "Acceptance justifies concealment" does not hold. Ethical acceptance varies by context, but disclosure is consistently required regardless of context. RQ3: Acceptance Depends on Context, but Disclosure Is Still Expected 13
14. Takeaway: User-Aligned Explanations Steer Both Decisions and Evaluations RESEARCH FINDINGS Finding (RQ1): User-aligned explanations can steer decision-making; effects depend on task context. Finding (RQ2): In high-stakes contexts, decisions remain unchanged, while trust and satisfaction significantly increase. Finding (RQ3): Ethical acceptability is context-dependent, but disclosure demands are consistent across all contexts. CONCLUSION Tailored explanations impact both decision-making and evaluation. In practice, disclosure of adjustments and ensuring verifiability are essential. 14