back to arrowtheory.com

## On the concept of entropy of a finite probabilistic scheme

*D. K. Faddeev, 1956.*

*English translation by Arina Zinovyeva.*

*Section 1.*
A. Ya. Khinchin's article [1] established that the entropy
of a finite probabilistic scheme
can be characterised by the following system of axioms.

*1*.
is a continuous function of
in the domain
.

*2*. is a symmetric function of .

*3*. .

*4*.

where (this axiom is formulated in the article [1] in other terms).

*5*.

Entropy is defined by these axioms uniquely up to a positive multiplier. The aim of this note is a simplification of this system of axioms. Namely, we offer the following axioms:

1'. is continuous, with and positive at least at one point.

2'. is a symmetric function of .

3'. With Here .

The difference between these two systems of axioms is, first of all, axiom 5 (extremality) is replaced with the requirement of positivity of the entropy at one point, and second of all, axioms 3 and 4 are replaced with axiom 3' that is a very natural one if we consider entropy as a measure of the "indefiniteness" of a scheme. Truly, the "indefiniteness" of a scheme is different from the "indefiniteness" of the scheme occurring from dividing events into two sub-events , with conditional probabilities . This "indefiniteness" is overcome only when the event is realised, the probability of which equals .

*Section 2.*
First of all, we establish that axioms 2, 3, 4 are equivalent to 2', 3'.

**Lemma 1.** From 2, 3, 4, we conclude

**Proof**. We calculate
with two methods.
On one hand, by the
consequence of 3, we have
On the other hand,
and by the consequence of 4, we get Therefore,

**Lemma 2.** From 2, 3, 4 we conclude 3'.

**Proof**.
equals, by the consequence of 2 and 3, , which equals, by the consequence of 4,

**Lemma 3.** From 2', 3', we conclude that

**Proof**. We calculate with two methods On one hand, by the consequence of 3', we have On the other hand, Therefore,

**Lemma 4.** From 2', 3' we conclude 3.

**Proof**.

**Lemma 5.** From 3', we conclude that:
Here ,

**Proof**.
With the statement of lemma is the same as axiom 3',
with the lemma is easily proven by induction on .

**Lemma 6.** From 2', 3', we conclude that:
Here

**Proof**. We need to apply lemma 5, times to each group of arguments, which is
possible by the consequence of symmetry.
Setting
we get that from 2', 3' we can conclude 4.

So, we have established the equivalence of axiom groups 2, 3, 4 and 2', 3'.

*Section 3.*
Assume, following the article [1],
with and Applying lemma 6 to the case we get with With or equation (1) is trivial. Further, applying lemma 5 to , we get
from that we conclude:

**Lemma 7.** With

**Proof**. By the consequence of the continuity of with

Furthermore, where and But is an arithmetic average of the first members of the sequence which tends to zero. Therefore, and Furthermore, The lemma is proven.

By the consequence of equation (1), will be defined for all natural , as soon as we set the values of the function on prime numbers. Precisely, if where are prime, then Assuming Then

**Lemma 8.** There is a greatest number among .

**Proof**. We assume the opposite and show that this assumption will contradict
the assumption about the continuity of at It's true, if there is no greatest number in the sequence then we can build an infinite sequence of prime numbers so that
is the smallest prime number such that It is clear from the method of building this sequence that when a prime number is smaller than , then

Let and be a canonical decomposition of the number . Consider It is clear that because and by the consequence of the parity of among there is a number 2 with a nonzero index. Then with and

Therefore, which is impossible.

Using exactly the same method, we establish that there is a smallest among

**Lemma 9.** where c is a constant.

**Proof**. It's enough to establish that all are equal.

Assume that such a prime number is found, that We set a prime number for which takes the greatest value. Then and

Let be a natural number and a canonical decomposition of . We consider Under the transition to the limit we establish: which is impossible. Therefore, takes place for all

Using exactly the same method, it's established that for all . The lemma is proven.

**Theorem**.

**Proof**. We first consider the case
Let with r and s whole.
We apply lemma 6 to
by combining arguments into two groups with
and elements. We get:
Where

By the consequence of the continuity of , the formula that we get can be also applied for irrational values of by the consequence of positivity of at least at one point.

The transition to the general case is realised by induction, based on axiom 3'.

*Citations:*

[1] A. Ya. Khinchin, On the concept of entropy in the theory of probabilities, Uspekhi Mat. Nauk, VIII, issue 3 (55), (1953).

Copyright (c) 2018 - 2019.