Get Started
Home
Topics
Search
Library
Research questionHow reliably do proxy LLM judges capture users’ perceived helpfulness and privacy in privacy-sensitive scenarios?Users may disagree substantially about whether a response is helpful or appropriately privacy-preserving. Proxy LLMs can show strong agreement with one another while still failing to reflect this variation in user perceptions.
AI
Alignment & Safety
Evaluation & Benchmarks
Natural Language Processing
Latest papersRecent research connected to this question, newest first.User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive ScenariosEvidence comes from a user study with 94 participants evaluating LLM responses to 90 PrivacyLens privacy-sensitive scenarios. Five proxy LLMs showed high agreement with one another but low correlation with users’ evaluations; the findings concern perceived helpfulness and privacy in these scenarios rather than objective privacy protection in general.research paper · Sep 3, 2026
Related questions
How reliably do human psychometric questionnaires predict LLM behavior in realistic user interactions?Can black-box LLM judges provide reproducible measurements on shared endpoints?How can we tell whether agreement among LLM judges reflects human alignment or shared blind spots?Can user feedback reliably guide LLM revisions if LLM judges overlook the resulting improvements?