Get Started
Home
Topics
Search
Library
Research questionHow stable are recommender-system evaluation and model-selection conclusions across training seeds?Stochastic training can change model parameters, evaluation scores, and recommendation lists even when the data split is unchanged. This makes it difficult to tell whether model-selection outcomes reflect robust differences or a particular training run.
Evaluation & Benchmarks
Information Retrieval
Machine Learning
Statistical Machine Learning
Latest papersRecent research connected to this question, newest first.Training seeds and model-selection stability in recommender-system evaluationThe study varies training seeds across hyperparameter configurations while holding the data partition fixed. It examines user-level metric sensitivity, validation-based model selection, validation-to-test transfer, and agreement between top-k recommendation lists; the evidence indicates that seed effects are often detectable and can make single-seed conclusions appear more stable than they are.research paper · Sep 2, 2026
Related questions
How can recommenders adapt to changing preferences without trusting unreliable inputs?How can we measure saturation of repeated LLM recommendations without fixed rosters masking new brands and sources?How can production RAG teams maintain reliable comparisons as new retrieval candidates arrive without rejudging overlapping documents?How can small-scale pretraining mixture experiments stay reliable when scarce high-quality data is repeated at target scale?