Dancing with the danger

As I mentioned in the last post, negative scoring was meant to be mandatory in the AI, so level 7 brought the posibility to use those negativeness for good.

Now we have here the first example of how good negative scoring can be: A new tendency to not going outside the track, even if a big drop is in the edge of the track attracting you to the dissaster.

Before explanations, a first video with 3 karts:

The new intelligence level 7

For the 3rd or 4rd time, I think the new improvement in the base intelligence algorithm can be the final one, so let be cautious on this: may be the new AI level 7 is the perfect one.

But let's start with the origin: I really need negative scorings on the futures. I previously thought it was an abomination, and in a philosophical point of view still is, but it is mandatory if you want to have an real AI, one that can compite to acomplish a goal, to beat an oponent or to get as many score as possible: to have an intelligence that is usefull on optimizing, on game theory, etc.

Why it rendered mandatory to have negative scoring will be covered on a next post, now I will just introduce you the level 7, and we will start with a visual comparasion with some previous intelligence level on my old and good "test circuit".

No more suicides

Last video was quite impresive, a rocket flying inside a cave at full speed, but for me it was really disapointing: it showed up quite clearly a suicide tendency in the AI.

As you can see, the rocket leaves the track 2 or 3 times as it gets too fast to stop before crashing, but if you look closely, it happends that the crash WAS not impossible to avoid, not at all, but somehow the rocket put hands down, stop fighting, give up and let itself crash without even trying to scape. Why?

As I commented on the last post, I considered usign a negative scoring tendency to "keep alive" the player: if you die before 2 seconds while imagining a future, then give it a negative scoring so the player will actively avoid it. It was a really desperate try, as score is internally an "entropy gain", and allowing negative values is like allowing entropy to lower with time... it is a physic abomination, and I am really happy it didn't work out in my tests (I didn't finally use negative scoring but something as ugly as this).

Driving a rocket inside a cave!

Version 0.7 of the software came with a great generalization of all the parts that conform this AI (at the cost of a noticeable drop in perfomance) so it is now possible to use a base class of  "Player2D" to create a brand new kind of vehicle quite easily, with a minimun amount of code: just define the vehicle params, its simulation code, the drawing stuff, and thats all.

It was then time to try with a brand new creature and compare it with the old known kart. I decided to code a classical rocket that travels the circuit as if it were a vertical cave, with a gravity force downwards that only apply to rockets, and see how the AI deal with it.

I have to admint I made it way too powerful, and given that the AI will try to maximize entropy, it is short of natural that it will drive the rocket as fast as it can. Notice how much it likes to spin around as a way to hover around a place before it decides where to go: the quickiest it rotates, the more future options it has, as it can leave the spinning to any needed direction in a short time. That is way spinning is a nice option to its eyes.

So here you have a rocket and a kart trying to get sweet drops from the circuit, look carefully at the rocket as it enter the narrowest parts of the circuit, it is amazing how good the AI is managing such a difficult to master kind of ship:

Mixing goals

Next video shows 36 karts driving around with 3 goals each one:

1) Move fast.
2) Pick up the drops.
3) Store the drops.

The internals are easy:

1) When you drive N meters, you score N.
2) When you drive over a drop of size 5, if you have internal storage left (a kart has a capacity of 100) you "upload" the drop and get 5 points into your scoring.
3) When you drive over a "storage" rectangle, if it still has free space, your internal storage is "downloaded" into the storage and you receive scoring for it.

The result is something like a group of ants collecting food and accumulating on some spots:

Garbage collectors

In the last post I showed you "motivations", after it I renamed it, on the code and on my mind, to "goals", much shorter and general!

But in this previous video there were only "personal goals".

Each player (or kart) had its own set of motivations, and that is why only orange kart was able to eat the orange sweet drops on the track: only this kart was able to "see" them.

This time I have added "team goals", so a goal is shared by all of the players in the team. Now, if I add a "sweet drops" goal, all the karts will fight to get the sweet drops on the track:

Introducing "motivations"

The intelligence at level 5 is quite nice, almost perfect, or may be perfect, but there is something we can't control: the goals.

We never knows what the AI will decide to do to solve the puzzle we present to it, nor we know witch goal, if any, will it follow.

This is quite nice to have something like this, it could react to unespected scenarios as if it were used to them, but it would also be lovely to be able to "drive" the curse of action towards some more mundane goals: domesticate the intelligence and make it follow our likings.

Redefining goals will make this AI suitable for optimizing -in any sense- any system you could define and simulate, in a "intelligent way". Anything. Amazing.

Beyond entropy

Level 5 of intelligence seems to be reflecting the actual definition of entropy on the original paper, so before going any further, we will write it in pseudo-code and embed on it the example of the kart seen in video 1 entry:

AI level 4 and 5: Entropy and energy

In this video, yellown kart is using "level 3" intelligence so it score an option with N different futures with Ln(N), and it would be ok if all futures were equiprobable, but they aren't.

In the case not all futures are equipropable you have to switch to another, more complex, way of calculating entropy.

When you have N microstates but each one has a different probability of happening, call it P(i), in the instantaneous or "clasical" entropy, we use:

S = Sum. on all possible microstates(P(i)*Ln(P(i)))

AI level 3: Using real entropy

Intelligence "Level 3"
Looking back at level 1 and 2 of our AI, you can notice that in both cases we are scoring each option using just a count of the different futures we were able to find.

But this is not a real entropy definition!  If a macrostate has N possible and equiprobable microstates, then its entropy is not just N, it is:

S = k*Ln(N)

As k is constant, we can forget about it, and so instead of using N to score each option as in the first videos, we now use Ln(N) for the orange kart:

Intelligence Level 2

Video 4: Intelligence level 2

In video 1 we commented on the simpliest way to implement the entropic intelligence on a kart: count how many different end points you have in the futures that start by chosing "+5", and compare with the number you get for choice "-5", then average and take this as your decision. We will call this "intelligence level 1".

But as simple as it seems, almost every aspect of the AI explained on video 1 can be redefined so the driving of the AI "looks" more natural, as if the driver were a real driver doing his best.

By the way, chosing a kart simulation as a test-bed for the algortihm as proven to be a really good choice, as it is very easy to just observe two kart driving side by side, each one with a different version of the AI, and tell witch one of them was doing a better job. It wouldn't have been that easy with another simulation.

So, steeping over video 2 and 3, that just show intelligence level 1 solving different circuits -you can watch them on the "YouTube" link avobe- we jump to video 4, the first one to really level up intelligence to level 2:

 

The entropy

Before going any further on the algorithm itself, we will stop for a moment on the real meaning of those "causal entropic forces" the algorithm is based on.

This is a little technical -but quite interesting- and I will try my best on being easy to follow, but feel free to pass on this and focus on the algortihmic-only articles if you want. You will get as much knowledge of the AI as you will need to apply it, but be warned: when it comes to defining your own system, adjusting the params of the AI and polishing the way you measure how better a new situation is compared to a previous one, the understanding of the underlaying physics will give you an extra insight on the proccess and will help you pin-point the weak points on your implementation.

Disclaimer: I am not a physicist, just an oxidized mathematician and a programmer who loves reading about the subject, so please be benevolent when commenting! I just pretended anyone could have a clear picture of the concept itself and the extreme power under the nice sounding word "entropy".

Entropy

Entropy is a very powerful phisical concept, it is behind almost all laws of the classic phisics. It is quite simple in its definition, but almost imposible to directly use in any real world calculation.