杰恩斯（Jaynes）的《概率论：科学的逻辑》（Probability Theory: The Logic of Science，https://books.google.com/books/about/Probability_Theory.html...）和格伦瓦尔德（Grunwald）的《最小描述长度原则》（The Minimum Description Length Principle）。
Jaynes, Probability Theory: The Logic of Science (https://books.google.com/books/about/Probability_Theory.html...) Grunwald, The Minimum Description Length Principle.
…exasperation about the Darwinist analogy, and for my claims about the flabbiness of 10-year-old climate predictions, which also happens to be interesting reading for hackers interested in machine learning: _The Minimum Description Length Principle_. Lots of math to help you think precisely about questions like "by how many bits did this hypothesis reduce the surprisingness of the world compared to alternative hypotheses?" For sources more specific…
…0% probability of being "by chance". Then, you say "that's a complicated rule: no one can memorize the list", and that's fine. But finding the simplest rule that has the best fit can be shown, via the Minimum Description Length principle, to be equivalent to models with deep connections to statistical learning. The gist of the matter is: even if you have a single dataset, statistical considerations are very important to make sure that…
Some ideas like the Minimum Description Length principle derived from information theory turn out to be just rephrasings of already-known statistical ideas. This can occasionally provide more insight, but also can lead to ridiculously irrelevant things like…
…so-called "universal" coding. It's ok* to depart from that starting point in creating subtheories but if you don't start there you'll end up with garbage like the last 50 years of confusion over what "The Minimum Description Length Principle" really means. *It is, however, _not_ "ok" if what you are trying to do is come up with causal models. You can't get away from Turing complete codes if you're trying to model dynamical systems even…
……https://arxiv.org/abs/1811.06965 《通过最小化权重描述长度保持神经网络简洁》（Keeping Neural Networks Simple by Minimizing the Description Length of the Weights）- https://www.cs.toronto.edu/~hinton/absps/colt93.pdf 《最小描述长度原则入门教程》（A Tutorial Introduction to the Minimum Description Length Principle）- https://arxiv.org/abs/math/0406077 《复杂动力学第一定律》（The First Law of Complexodynamics）- https://scottaaronson.blog/?p=762 《量化封闭系统中复杂性的消长：咖啡自动机》（Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton）- https://arxiv.org/abs/1405.6903……
…https://arxiv.org/abs/1811.06965 Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077 The First Law of Complexodynamics - https://scottaaronson.blog/?p=762 Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903…
杰恩斯（Jaynes）的《概率论：科学的逻辑》（Probability Theory: The Logic of Science，https://books.google.com/books/about/Probability_Theory.html...）和格伦瓦尔德（Grunwald）的《最小描述长度原则》（The Minimum Description Length Principle）。
Jaynes, Probability Theory: The Logic of Science (https://books.google.com/books/about/Probability_Theory.html...) Grunwald, The Minimum Description Length Principle.
……我对达尔文类比的恼火，以及我关于十年前的气候预测有多立不住脚的论断，巧的是，对机器学习感兴趣的黑客来说，这恰好也是一本很有意思的读物：《最小描述长度原则》（The Minimum Description Length Principle）。里面有大量的数学知识，能帮你精准思考这类问题：“与备选假说相比，该假说使世界的令人惊异程度减少了多少个 bit？”至于更具体的文献来源……
…exasperation about the Darwinist analogy, and for my claims about the flabbiness of 10-year-old climate predictions, which also happens to be interesting reading for hackers interested in machine learning: _The Minimum Description Length Principle_. Lots of math to help you think precisely about questions like "by how many bits did this hypothesis reduce the surprisingness of the world compared to alternative hypotheses?" For sources more specific…
……0% 的概率是“纯属巧合”。然后你就会说“这规则太复杂了：根本没人能记住这张列表”，这很合理。但通过最小描述长度原则（Minimum Description Length principle）可以证明，找到拟合最好且最简短的规则，等价于那些与统计学习有深厚渊源的模型。核心要点在于：即便你手里只有一个数据集，统计学层面的考量也至关重要，它可以确保……
…0% probability of being "by chance". Then, you say "that's a complicated rule: no one can memorize the list", and that's fine. But finding the simplest rule that has the best fit can be shown, via the Minimum Description Length principle, to be equivalent to models with deep connections to statistical learning. The gist of the matter is: even if you have a single dataset, statistical considerations are very important to make sure that…
一些源于信息论的想法（比如最小描述长度原则），到头来只不过是对既有统计学思想的重新表述。这偶尔能提供更深层的洞见，但也能导致极其脱靶扯淡的事情，比如……
Some ideas like the Minimum Description Length principle derived from information theory turn out to be just rephrasings of already-known statistical ideas. This can occasionally provide more insight, but also can lead to ridiculously irrelevant things like…
……所谓的“通用”编码。在构建子理论时脱离这个起点没问题*，但如果不从这里出发，最终就会搞出一堆垃圾，比如过去50年里关于“最小描述长度原则”究竟意味着什么的混乱认知。*然而，如果你试图构建的是因果模型，那这样做就“不行”了。即便你建模的是动力系统，也绕不开图灵完备编码……
…so-called "universal" coding. It's ok* to depart from that starting point in creating subtheories but if you don't start there you'll end up with garbage like the last 50 years of confusion over what "The Minimum Description Length Principle" really means. *It is, however, _not_ "ok" if what you are trying to do is come up with causal models. You can't get away from Turing complete codes if you're trying to model dynamical systems even…
……https://arxiv.org/abs/1811.06965 《通过最小化权重描述长度保持神经网络简洁》（Keeping Neural Networks Simple by Minimizing the Description Length of the Weights）- https://www.cs.toronto.edu/~hinton/absps/colt93.pdf 《最小描述长度原则入门教程》（A Tutorial Introduction to the Minimum Description Length Principle）- https://arxiv.org/abs/math/0406077 《复杂动力学第一定律》（The First Law of Complexodynamics）- https://scottaaronson.blog/?p=762 《量化封闭系统中复杂性的消长：咖啡自动机》（Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton）- https://arxiv.org/abs/1405.6903……
…https://arxiv.org/abs/1811.06965 Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077 The First Law of Complexodynamics - https://scottaaronson.blog/?p=762 Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903…