Figure 2.3. Ambiguity on the symbol at the input when yj is received
The average value of this uncertainty, or the entropy associated with the receipt of the symbol yj, is:
[2.26]
The mean value of this entropy for all the possible symbols yj received is:
[2.27]
Which can be written as:
[2.28]
or:
[2.29]
The entropy H(X/Y) is called equivocation (ambiguity) and corresponds to the loss of information due to disturbances (as I(X, Y) = H(X)− H(X/Y)). This will be specified a little further.
Because of disturbances, if the symbol xi is issued, there is uncertainty about the received symbol yj, j = 1, ... , m.
Figure 2.4. Uncertainty on the output when we know the input
The entropy of the random variable Y at the output knowing the X at the input is:
[2.30]
This entropy is a measure of the uncertainty on the output variable when that of the input is known.
The matrix P(Y/X) is called the channel noise matrix:
[2.31]
A fundamental property of this matrix is:
[2.32]
Where: p(yj/xi) is the probability of receiving the symbol yj when the symbol xi has been emitted.
In addition, one has:
[2.33]
with:
[2.34]
p(yj) is the probability of receiving the symbol yjwhatever the symbol xi emitted, and:
[2.35]
p(xi/yj) is the probability that the symbol xi was issued when the symbol yj is received.
2.5.2. Relations between the various entropies
We can write:
In the same way, as one has: H(Y, X) = H(X, Y), therefore:
In addition, one has the following inequalities:
[2.38]
and similarly:
[2.39]
SPECIAL CASES.–
– Noiseless channel: in this case, on receipt of yj, there is certainty about the symbol actually transmitted, called xi (one-to-one correspondence), therefore:
[2.40]
Consequently:
[2.41]
and:
[2.42]
– Channel with maximum power noise: in this case, the variable at the input is independent of that of the output, i.e.:
[2.43]
We then have:
[2.44]
[2.45]
[2.46]
Note.– In information security, if xi is the plaintext, and yj is the corresponding ciphertext, then p(xi/yi) = p(xi) is the condition of the perfect secret of a cryptosystem.
2.6. Mutual information
The mutual information obtained on the symbol xi when the symbol yj is received is given by:
[2.47]
where i(xi, yj) represents the decrease in uncertainty on xi due to the receipt of yj.
The average value of the mutual information, or the amount of information I(X, Y) transmitted through the channel is:
[2.48]
or:
[2.49]
Hence:
[2.50]
Given relationships [2.36] and [2.37], we get:
[2.51]