※ 本文轉寄自 ptt.cc, 文章原始頁面
Re: [討論] 語文效率的偽科學
https://machinelearningmastery.com/what-is-information-entropy/
A Gentle Introduction to Information Entropy
「A foundational concept from information is the quantification of the amount of information in things like events, random variables, and distributions.」
Calculate the Information for an Event
Quantifying information is the foundation of the field of information theory.
The intuition behind quantifying information is the idea of measuring how much surprise there is in an event. Those events that are rare (low probability) are more surprising and therefore have more information than those events that are common (high probability).
Low Probability Event: High Information (surprising).
High Probability Event: Low Information (unsurprising).
The basic intuition behind information theory is that learning that an unlikely event has occurred is more informative than learning that a likely event has occurred.
Rare events are more uncertain or more surprising and require more information to represent them than common events.
We can calculate the amount of information there is in an event using the probability of the event. This is called “Shannon information,” “self-information,” or simply the “information,” and can be calculated for a discrete event x as follows:
information(x) = -log( p(x) )
Where log() is the base-2 logarithm and p(x) is the probability of the event x.
The choice of the base-2 logarithm means that the units of the information measu
re is in bits (binary digits). This can be directly interpreted in the informati
on processing sense as the number of bits required to represent the event.
h(x) = -log( p(x) )
The negative sign ensures that the result is always positive or zero.
Information will be zero when the probability of an event is 1.0 or a certainty,
e.g. there is no surprise.
Calculate the Entropy for a Random Variable
We can also quantify how much information there is in a random variable.
For example, if we wanted to calculate the information for a random variable X with probability distribution p, this might be written as a function H(); for example:
H(X)
In effect, calculating the information for a random variable is the same as calculating the information for the probability distribution of the events for the random variable.
Calculating the information for a random variable is called “information entropy,” “Shannon entropy,” or simply “entropy“. It is related to the idea of entropy from physics by analogy, in that both are concerned with uncertainty.
The intuition for entropy is that it is the average number of bits required to represent or transmit an event drawn from the probability distribution for the random variable.
… the Shannon entropy of a distribution is the expected amount of information in an event drawn from that distribution. It gives a lower bound on the number of bits […] needed on average to encode symbols drawn from a distribution P.
— Page 74, Deep Learning, 2016.
最後一次回槓精了
你們要繼續跳針請便
-----
Sent from JPTT on my iPhone
--
※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 111.235.192.33 (臺灣)
※ 文章網址: https://www.ptt.cc/bbs/DummyHistory/M.1695701266.A.F18.html
Re: 回文串
11
[討論] 語文效率的偽科學
DummyHistory09/22 05:27
410
Re: [討論] 語文效率的偽科學
DummyHistory09/22 06:01
311
Re: [討論] 語文效率的偽科學
DummyHistory09/22 09:41
015
Re: [討論] 語文效率的偽科學
DummyHistory09/22 11:22
1580
Re: [討論] 語文效率的偽科學
DummyHistory09/22 12:22
01
Re: [討論] 語文效率的偽科學
DummyHistory09/22 13:56
110
Re: [討論] 語文效率的偽科學
DummyHistory09/22 16:38
00
Re: [討論] 語文效率的偽科學
DummyHistory09/22 18:23
319
Re: [討論] 語文效率的偽科學
DummyHistory09/22 18:32
11
Re: [討論] 語文效率的偽科學
DummyHistory09/23 11:00
00
Re: [討論] 語文效率的偽科學
DummyHistory09/23 19:10
06
Re: [討論] 語文效率的偽科學
DummyHistory09/24 09:22
322
Re: [討論] 語文效率的偽科學
DummyHistory09/24 15:47
215
Re: [討論] 語文效率的偽科學
DummyHistory09/24 16:12
00
Re: [討論] 語文效率的偽科學
DummyHistory09/24 16:21
00
Re: [討論] 語文效率的偽科學
DummyHistory09/24 16:38
430
Re: [討論] 語文效率的偽科學
DummyHistory09/24 18:14
123
Re: [討論] 語文效率的偽科學
DummyHistory09/24 18:39
12
Re: [討論] 語文效率的偽科學
DummyHistory09/24 22:26
05
Re: [討論] 語文效率的偽科學
DummyHistory09/24 22:52
010
Re: [討論] 語文效率的偽科學
DummyHistory09/25 00:18
24
Re: [討論] 語文效率的偽科學
DummyHistory09/25 00:24
00
Re: [討論] 語文效率的偽科學
DummyHistory09/25 01:04
00
Re: [討論] 語文效率的偽科學
DummyHistory09/25 01:07
00
Re: [討論] 語文效率的偽科學
DummyHistory09/25 09:35
00
Re: [討論] 語文效率的偽科學
DummyHistory09/25 11:25
676
Re: [討論] 語文效率的偽科學
DummyHistory09/25 16:47
22
> Re: [討論] 語文效率的偽科學
DummyHistory09/26 12:07
02
Re: [討論] 語文效率的偽科學
DummyHistory09/26 19:58
00
Re: [討論] 語文效率的偽科學
DummyHistory09/26 22:47
00
Re: [討論] 語文效率的偽科學
DummyHistory09/26 22:56
00
Re: [討論] 語文效率的偽科學
DummyHistory09/26 23:23
110
Re: [討論] 語文效率的偽科學
DummyHistory09/27 00:25
00
Re: [討論] 語文效率的偽科學
DummyHistory09/27 00:48
00
Re: [討論] 語文效率的偽科學
DummyHistory09/27 09:56
00
Re: [討論] 語文效率的偽科學
DummyHistory09/27 12:18
110
Re: [討論] 語文效率的偽科學
DummyHistory09/27 13:17
00
Re: [討論] 語文效率的偽科學
DummyHistory09/29 22:12
09
Re: [討論] 語文效率的偽科學
DummyHistory09/29 22:35
2 則留言
innominate 作者的近期文章
Re: [討論] 國民黨執政中華民國時期對臺灣半導體產業的貢獻
5.10 Parallelism and Memory Hierarchy: Cache Coherence Given that a multicore multiprocessor means multiple processors o
Re: [討論] 語文效率的偽科學
※ 引述《fw190a (las)》之銘言: : ※ 引述《ZMittermeyer (ZM)》之銘言: : : 1 bit的定義是:如果一個黑盒子中有A與B兩種可能性,他們出現的機率相等 : : 要確定會是A還是B,我們需要的訊息量就是
推
推