tag:blogger.com,1999:blog-6923947282926324208.post7020695640073914982..comments2024-03-08T09:15:52.903+01:00Comments on Entropic and Fractal Intelligence: Graph entropy 1: The problemSergio Hernandezhttp://www.blogger.com/profile/18108694861191833007noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-6923947282926324208.post-61647552702868095932019-08-29T18:19:03.027+02:002019-08-29T18:19:03.027+02:00Hi Artem. Defining log(0) as -inf does not solve a...Hi Artem. Defining log(0) as -inf does not solve any problem. You can not operate with that and, when p tends to 0, you still blow up.<br /><br />And about KL-div, being forced to only apply it to probabilities that "support" each other is just crazy if you think on it: almost all interesting examples will have some elements with probability 0 in some places, saying Pi and Qi need to be non-cero all the time, in all the cases, is highly restrictive for a formula that is so basic to everything like entropy, cross-entropy, and divergence. Somehow we have been said it was natural, and we just accepted.Sergio Hernandezhttps://www.blogger.com/profile/18108694861191833007noreply@blogger.comtag:blogger.com,1999:blog-6923947282926324208.post-87260511475949653222019-08-29T18:18:34.441+02:002019-08-29T18:18:34.441+02:00This comment has been removed by the author.Sergio Hernandezhttps://www.blogger.com/profile/18108694861191833007noreply@blogger.comtag:blogger.com,1999:blog-6923947282926324208.post-26908424884145335892019-08-10T17:10:33.144+02:002019-08-10T17:10:33.144+02:00I'd argue the log(0) is not a problem if you m...I'd argue the log(0) is not a problem if you move to the space of extended reals and assume log(0) = -∞.<br /><br />As to where it comes from: this is inherent behavior of KL divergence, which forces supports of two distributions to agree is a certain sense, otherwise it'd blow up to infinity. There exist many other divergences, many of which instead "saturate" on some finite value when supports disagree.Artem Sobolevhttp://artem.sobolev.namenoreply@blogger.com