Proper Dataset Valuation by Pointwise Mutual Information
Chen, Qi, Chen, Kwon, Zou, Zheng · cs.LG,cs.GT · 2026-09-05 · 原文
Data plays a central role in advancements in modern artificial intelligence, with high-quality data emerging as a key driver of model performance. This has prompted the development of principled and effective data curation methods in recent years. However, existing methods largely rely on heuristics, and whether they are truly effective remains unclear. For instance, standard evaluation methods that assess a trained model's performance on specific benchmarks may incentivize assigning high scores to data that merely resembles the test set. This issue exemplifies Goodhart's law: when a measure becomes a target, it ceases to be a good measure. To address this issue, we propose an information-theoretic framework for evaluating data curation methods. We define dataset quality in terms of its informativeness about the true model parameters, formalized using the Blackwell ordering of informativeness. Under this ordering, Blackwell's theorem ensures that more informative data yields optimal models with lower expected loss on the true underlying distribution. To measure informativeness, we show that the Blackwell order can be determined by the Shannon mutual information between the curated
讲义
讲义·推断 依据「原文」自动生成的结构化摘要（推断），非原文表述；以原文为准。
1. 人话版
Data plays a central role in advancements in modern artificial intelligence, with high-quality data emerging as a key driver of model performance.
This has prompted the development of principled and effective data curation methods in recent years.
2. 领域脉络
本文类目：cs.LG、cs.GT，属于其所在研究脉络的最新进展。
3. 机制拆解
For instance, standard evaluation methods that assess a trained model's performance on specific benchmarks may incentivize assigning high scores to data that merely resembles the test set.
This issue exemplifies Goodhart's law: when a measure becomes a target, it ceases to be a good measure.
4. 证据与数字
摘要未给出量化结果——留意原文的实验与数据。
5. 反例与边界
However, existing methods largely rely on heuristics, and whether they are truly effective remains unclear.
Under this ordering, Blackwell's theorem ensures that more informative data yields optimal models with lower expected loss on the true underlying distribution.
6. 跨领域连接与意外收获
横跨 2 个类目（cs.LG、cs.GT），关注其在你兴趣板块间的迁移面。
7. 可复用方法
把本文机制与你手头项目对照，找一个两周内能验证的最小实验。
8. 术语表
精读时把不熟的术语记入此处，作为下次回忆的锚点。