Graph entropy 3: Changing the rules

After showing that the standard Gibbs cross-entropy was flawed and tried to fix it with a also flawed initial formulation of "free-of-logs" entropy, we faced the problem of finding a way to substitute a summary by a product without breaking anything important. Here we go...

When you define an entropy as a summary, each of the terms is supposed to be a "a little above zero": small and positive 𝛆 ≥0 so, when you add it to the entropy it can only slightly increase the entropy. Also, when you add a new probability term having (p=0) or (p=1), you need this new term to be 0 so it doesn't change the resulting entropy at all.

Conversely, when you want to define an entropy as a product of terms, they need to be "a little above 1" in the form (1+𝛆), and the terms associated with the extreme probabilities (p=0) and (p=1) can not change the resulting entropy, so they need to be exactly 1.

In the previous entropy this 𝛆 term was defined as (1-p_i^p_i), and now we need something like (1+𝛆) so why not just try with (2-p_i^p_i)?

Let us be naive again an propose the following formulas for entropy and cross-entropy:

H₂(P) = ∏(2-p_i^p_i)

H₂(Q|P) = ∏(2-q_i^p_i)

Once again it looks too easy to be worth researching, but once again I did, and it proved (well, my friend José María Amigó actually did) to be a perfectly defined generalised entropy of a really weird class, with Hanel-Thurner exponents being (0, 0), something never seen in the literature.

As you can see, this new cross-entropy formula is perfectly well defined for any combination of p_i and q_i (in this context, we are assuming 0⁰ = 1) and, if you graphically compare both cross-entropy terms, you find that, for the Gibbs version, this term is unbounded (when q=0 the term value goes up to infinity):

𝛟G(p, q) = -(p × log(q))

In the new multiplicative form of entropy, this term is 'smoothed out' and nicely bounded between 1 and 2:

𝛟2(p, q) = (2-qp)

Just to make sure it was what it looked like, I directly -and naively- sent an email to Stefan Thurner asking for his expert opinion: surprisingly, he kindly answered me! It was a big surprise for him too, so it was official: I had quite an extrange entropy on my hands.

The next question was whether or not this entropy had a nice replacement for the 4th axiom or, in other words, if the entropy of the Cartesian product PxQ of two independent distributions, H(P, Q), was bounded by some elegant combination of the entropies of P and Q, H(P) and H(Q).

It turned out that I needed to modify it a little bit in order to get a decent separability property, so I defined a third generalised entropy (this is the last one, promised!):

H₃(P) = 1 + log(H₂(P)) = 1 + log(∏(2 - p_i^p_i))

H₃(Q|P) = 1 + log(H₂(Q|P)) = 1 + log(∏(2 - q_i^p_i))

If now you compare the four generator functions, one for Gibbs entropy and three more for the new multiplicative ones, you can see how related they are:

It really didn't change much at a first glance, except that now exponents were again (1, 1) and it started to show some interesting separability properties/inequations, but the most interesting property was hidden: you could combine different distributions in a tree-like structure and still get a coherent value of entropy... it seemed to be a graph entropy instead of a distribution entropy.

This was far from evident. You needed to define some strange combination of those two multiplicative forms of entropy, H₂ and H₃, before you were able to see the "graph side" of it.

In the next post I will be showing how to define this entropy over any directed -and acyclic- graph, so we will be basically extending the concept of entropy of a distributions to a much broader audience.