An LLM’s personality score can change when you change the instructions or answer labels. Two papers examine why that matters. In The Unsampled Truth, meaning-preserving changes to the interface move questionnaire response distributions by roughly half to one ordinal step on average. Task instructions and option symbols generally move them more than item paraphrases do. In Human Psychometric Questionnaires Mischaracterize LLM Behavior, demographic personas shift questionnaire scores in directions that often match human survey data. But an independently constructed measure of preferences over fixed candidate responses shows no such alignment. So a questionnaire score has to pass two separate gates before it means what people usually take it to mean: it has to survive equivalent interfaces, and it has to predict something built outside the questionnaire. The papers show failures at each gate. Neither gate alone is the whole test.
What does the questionnaire measure?
The intuitive picture is that the model has traits, and the questionnaire reads them, roughly the way a thermometer reads temperature. Noise exists, so you average over prompts and get a cleaner estimate.
That picture assumes the instrument is a passive channel. It is not. A Likert item reaches the model as tokens: a persona in the system prompt, an instruction about how to answer, an item sentence, and a set of ordinal labels like 1-5 or A-E. The model then puts probability mass somewhere over those label tokens. Every one of those pieces is a design choice made by whoever built the harness, and each is an input the model conditions on. There is no separate channel through which “the trait” arrives.
The artifact study makes that concrete by building a Prompt Variant Matrix. The authors took five personas from the Nemotron dataset as the only construct-relevant component, then created five semantically equivalent variants of each of four other components — persona wording, task instruction, item wording drawn from a validated reformulation set, and option symbols (Arabic numerals, Roman numerals, uppercase and lowercase letters, and an arbitrary x1-x5 set). That gives 5⁴ combinations per persona-item cell, evaluated on 13 instruction-tuned open-weight models from the Llama, Qwen, Ministral, and Gemma families, with reasoning disabled. Responses were read as the probability mass on option tokens at the answer position, and distances between response distributions were measured with a 1-Wasserstein metric, which — unlike a divergence that treats the five options as unordered categories — charges more for moving mass three steps than one.
The headline number is APWD (Average Pairwise Wasserstein Displacement) between 0.5 and 1.0 on the BFI: averaged over pairs of prompts that are supposed to mean the same thing, the answer distribution moves by roughly half to one full ordinal step. And the movement is not evenly spread across components. Task instructions and option symbols generally displaced BFI distributions more than wording did — that is, the pieces carrying no construct content at all moved the score more than the paraphrases of the actual questionnaire items.
A useful sanity check on interpretation: near-zero displacement is not automatically good news. The authors argue that in the smallest models, such as Llama-3.2-1B, flat marginal displacement reflects failure to process the component rather than robustness to it. A model that ignores your instruction is stable for the wrong reason.
When the interface outvotes the persona
Displacement tells you the score moves. The next question is whether it moves more than the thing you meant to measure. The authors partition variation across the matrix into the intended persona signal, the artifact components, and interactions. In sub-billion models the intended psychological signal sits close to zero. As models scale toward 14B, the persona effect strengthens on average — but the average hides item-level failures. Even at 14B, some BFI items show more than 50% of variation attributed to prompt artifacts rather than the baseline personas. The authors’ conclusion for their setup is blunt: none of the evaluated models demonstrated sufficient robustness for the psychometric configurations tested.
Some caveats matter for how far you carry this. The variant set is targeted, not exhaustive; structural placement and reversed option order were not tested. Semantic equivalence is a premise the design cannot fully verify, though the authors note that where equivalence is most contestable — the paraphrases — displacement was smallest, while the components whose equivalence is closest to definitional drove most of the variation. The Nemotron personas may carry weak latent trait density, which would inflate the apparent artifact share. Large parts of variation stayed unexplained because components interact. And only open-weight models were tested, because the readout needs access to response distributions; nothing here settles whether proprietary systems behave the same way. Even the readout itself is a choice: option-token probabilities are the current standard, but whether artifact dominance persists under repeated sampling is explicitly left open.
Stability would not be enough anyway
Suppose you fixed all of that. You lock the harness, run a sensitivity panel, find your configuration sits in a flat region, and now your persona-conditioned trait scores are reproducible. Have you measured personality?
The behavior paper says: not yet, and it shows why with an unusually clean contrast. The setup asks whether persona-induced shifts on questionnaires carry over to generation behavior. Seven models received eight demographic system-prompt personas (two contrasting conditions each for gender, age, political orientation, education). For each condition the authors computed the shift from a vanilla baseline in Schwartz value profiles, then compared that shift to the corresponding human subgroup difference in the European Social Survey. Two readouts were compared against the same human reference: the established questionnaires PVQ-40 and PVQ-21, and a generation-probability profile.
That second measure is the pivot, so it is worth understanding before its numbers. Instead of asking the model to rate itself, it uses the Value Portrait dataset: 104 real-world queries drawn from human-LLM conversation logs and human-human advisory archives, each paired with five candidate responses evaluated for Schwartz-value and Big Five associations by 681 human raters; response–construct associations were tagged when their validation correlation reached ρ ≥ .30. For each scenario the model’s log-probability for each candidate response is computed, then averaged per construct across scenarios. So the question shifts from “how do you rate yourself on benevolence” to “how much probability do you assign to the benevolent reply in this actual advice-column dilemma.” To check the candidates were not off-distribution, the authors sampled ten responses per scenario at temperature 1.0 and ranked all fifteen; the highest-ranked dataset candidate had a median rank of 4.
The questionnaires looked good. PVQ-40 cosine similarity with human subgroup differences was positive in all eight conditions, peaking at Age (Young 0.95, Old 0.89) and Political orientation (Left-wing 0.86, Right-wing 0.70), with direction match 62/80 (p < 0.001). PVQ-21 followed the same pattern more weakly. Female and University+ were the notable exceptions, with weak or negative alignment.
The scenario measure showed nothing of the kind. Mean cosine was −0.03; the aggregate cosine over the concatenated 80-dimensional delta was 0.007 with a bootstrap 95% CI of [−0.22, 0.24]; direction match was 40/80, indistinguishable from chance at p = 0.54. Importantly, this is not the model failing to respond to the persona. Contrasting conditions produced opposite-sign shifts (Male −0.59 vs. Female +0.62; Right-wing −0.69 vs. Left-wing +0.65), which is directional incoherence rather than attenuation. On the normalized magnitude scale, the questionnaires shifted about 0.67–0.71 against a human value of 0.20 — three to three and a half times as much — while the scenario measure shifted 0.37. The personas moved the model; they just did not move it along the human demographic directions the questionnaires advertised.
One boundary the authors draw themselves: this is a behavioral measure over fixed, construct-validated candidate responses, not a measure of unconstrained real-world behavior. It sits between a Likert scale and free generation. It also covers Schwartz values only in this comparison, since the ESS has no Big Five items, and it depends on token-level log-probabilities — available to developers with internal logits for open and proprietary models alike, but a real obstacle for external evaluators whose APIs do not expose likelihoods.
Why the two failures are not the same failure
It is tempting to file both papers under “questionnaires are unreliable.” That merges two problems with different fixes.
The first is repeatability across a bounded family of interfaces. If your score changes when 1-5 becomes A-E, you cannot compare across harnesses, or say the number describes the model rather than your template. The remedy is a sensitivity panel, and the artifact study’s matrix is a workable template for one.
The second is whether the number refers to anything outside its own instrument. Here is the uncomfortable part: passing the first gate does nothing for the second. The persona experiment shows exactly this shape — the questionnaires showed roughly twice as frequent cross-model agreement as the scenario measure and often matched human survey directions. Those are reassuring observations. Yet the separately constructed scenario measure moved in directions unrelated to those patterns. A self-report readout and a response-preference readout can both be well-behaved and still disagree about what the persona did.
Neither paper establishes one as the true personality measure. Both are interface-bound measurements. The scenario measure’s advantage is that it was constructed independently of the questionnaire and scores responses of the kind a user would actually receive — not that it is ground truth. Agreement between two independently constructed measurements would be evidence that the score tracks something beyond one instrument; disagreement, as observed here, means at least one of them does not generalize the way its users assume.
I would use the two findings to guide the reporting, starting with these checks.
First, a sensitivity panel. Before quoting a persona profile, vary persona wording, task instruction, item wording, and option symbols within the range a competent researcher might plausibly have chosen, and report how much the score moves. If the movement is comparable to the persona effect you are claiming, you have a harness reading, not a model reading. Note also whether flat results might mean the model is ignoring a component rather than resisting it. If your access is sampling-only, an empirical distribution over the ordinal options substitutes for the logit readout; the artifact framework only requires a distribution per item, though whether artifact dominance persists under that readout is untested.
Second, a held-out interaction set built independently of the questionnaire — realistic prompts, candidate responses labeled by a process that does not reuse your Likert items, and a check that the persona moves that measure in the direction the questionnaire implies. Describe it as what it is. A fixed-candidate probability measure is not deployment behavior, and calling it that reintroduces the overclaim you were trying to avoid.
Third, keep using questionnaire scores where they earn their keep. A locked harness with a documented variant panel gives you a cheap regression signal: run it after a fine-tune, see if the profile moves, investigate if it does. That makes it useful for catching changes that need investigation. The trait labels still do not establish how the deployed system will behave.
Nothing here proves that any particular validation set predicts production behavior; that question stays open, and the behavior paper’s authors name analysis of open-ended outputs as the natural next step. What the two papers jointly establish is narrower and still consequential: a precise-looking personality profile can fail because the interface outvoted the persona, or because the persona moved the questionnaire along human-looking directions while pushing a separately built behavioral measure somewhere unrelated. Those are different diagnoses. A single stability check finds only the first.