MatrAIx is an evaluation infrastructure that swaps human user studies for cohorts of persona-conditioned LLM agents, drawn from a Persona 8B dataset of 8.3B records over 1,290 attributes, hitting 91.5% adherence to declared persona behavior across four interaction environments.
You’ve shipped a chatbot or a checkout flow, and you want to know how a rural retiree with basic English reacts versus a bay-area engineer. Today you either recruit a small human panel (slow, expensive, tiny) or run an offline benchmark that only measures task success on an averaged user. Neither tells you which subgroup abandons the flow when the price jumps $2 or when the bot hallucinates.
Prior persona work like Generative Agents and OASIS focused on multi-agent social simulations. MatrAIx instead treats the product as the system under test and varies the simulated user population against it.
Three pieces plug together. First, Persona 8B: 8.3 billion records, each a value assignment over 1,290 categorical attributes (age bracket, English proficiency, risk tolerance, coding-tool use, and so on). Synthetic records are sampled from a Directed Acyclic Graph (DAG) where each attribute’s distribution depends on its parent attributes, plus hard compatibility masks that forbid impossible combinations (English-primary but English proficiency = None). Human-grounded records come from Wikipedia, Amazon Reviews, the Stack Overflow survey, the General Social Survey, PRISM Alignment, and a small consented volunteer survey, all extracted into the same 1,290-field schema by an LLM with regex-anchored evidence.
Second, the Playground: four environment adapters (Survey, AI Chatbot, Web, App) that run one persona through one task and record the trajectory. Third, Applications: 1,010 versioned task specs, each declaring the target product, the cohort query, the scenario, and a task-owned verifier.
cohort = sample(persona_8b, query, seed) # e.g. cost-sensitive shoppers
for persona in cohort: # trials are independent
trial = agent(persona, task, model, seed)
artifact = trial.run() # transcript, actions, final state
findings = task.verifier(artifact) # programmatic + optional judge
store(persona, task, model, findings, artifact)
report = aggregate(findings, by=["cohort", "subgroup"])
A compatibility factor upweights values that are more common given the parent context; a separate binary mask removes forbidden combinations. Keeping statistical rarity separate from logical invalidity means a rare-but-valid profile survives sampling.
The prevailing move in agent evaluation is to score a single generic user against a task and read off a leaderboard number. This paper takes the opposite stance. Hold the product fixed and vary the simulated user population under a declared sampling design; the interesting signal lives in the subgroup differences, not the aggregate score. The load-bearing evidence is not the headline task scores but the controlled adherence probe, which isolates whether declared persona attributes actually drive agent behavior.
The controlled adherence probe is the finding that makes the thesis operational. Ten behavioral attributes were each tested in Survey, Chatbot, Web, and App, with five personas at one pole and five at the opposite, and an LLM judge checked whether the declared behavior showed up in the trajectory. Agents expressed or correctly suppressed the assigned behavior in 366 of 400 trials (91.5%). Survey, Chatbot, and Web each hit at least 4-of-5 on both arms for 9 of 10 attributes; App got there for 6 of 10, with politeness and storytelling suppression the weakest cells (agents don’t reliably act rude on command).
Secondary evidence: across 18,189 trials on eight representative tasks, product conclusions swing sharply with the persona-agent model on identical cohorts. A paid-plan choice on one page ranged from 23.2% to 93.9% across GPT 5.5, Claude Opus 4.8, and Claude Haiku 4.5. On an OpenBB honesty task, all three models ordered the four trust-level subgroups identically (Cramér’s V 0.228-0.363), showing that when the task provides a clean channel for a persona attribute, the ordering is model-stable even when levels are not. For extraction quality, six human raters gave the 100-persona subset a 4.135 / 5 mean, and Claude Opus 4.8 was within one point of the human mean in 93.8% of comparisons.
Reach for this when you’re about to ship a chatbot, checkout flow, or onboarding step and you want to see how it plays for subgroups you can’t easily recruit. Attach your product as a REST sidecar or a browser target, write a task spec naming the cohort query (say, cost-sensitive shoppers in low-income bands), and run a few hundred trials before the human study to catch obvious subgroup friction. The output isn’t a satisfaction number to ship on: it’s a set of transcripts and structured verifier findings you can inspect per persona.
Code and website live at GitHub and matraix.ai. The Persona 1M coreset (599,847 human-grounded + 400,000 synthetic records under a shared 1,290-field schema) is on Hugging Face. The task library ships 1,010 specs across Commerce, Software, Finance, Healthcare, and 20+ other domains, though only eight were actually executed in the paper.
Simulated users are hypothesis generators, not human replacements. Report the persona-agent model as part of every result, and never let it share a backbone with the system under test unchecked.
•
Cross-model agreement is thin. Median Cohen’s kappa across 88 comparable fields between model pairs is near zero, and one product’s paid-plan share moved 70+ points across three persona models. A finding that isn’t replicated across at least two backbones isn’t a finding.
•
Persona model and system model are never crossed in the experiments. If the same backbone plays both the user and the product, favorable results may just be self-preference. The paper flags this but doesn’t measure it.
•
The volunteer cohort is 355 people, opt-in, skewed toward South Asia, Sub-Saharan Africa, and lower income bands. The 1M coreset calibrates to four marginals (age, region, gender, urbanicity) only, not joint coverage. Do not treat cohort proportions as population estimates.