Let’s imagine we have an (x,y) plane where a robot can move. Now we define the middle of our world as the goal state, which means that we are going to give a reward of 100 to our robot once it reaches that state.
Now, let’s say that there are 4 states(which I will call A,B,C,D) that can lead to the goal state.
The first time we are in A and go to the goal state, we will update our QValues table as following:
Q(state = A, action = going to goal state) = 100 + 0
One of 2 things can happen. I can end the episode here, and start a different one where the robot has to find again the goal state, or I can continue exploring the world even after I found the goal state. If I try to do this, I see a problem though. If I am in the goal state and go back to state A, it’s Qvalue will be the following:
Q(state = goalState, action = going to A) = 0 + gamma * 100
Now, if I try to go again to the goal state from A:
Q(state = A, action = going to goal state) = 100 + gamma * (gamma * 100)
Which means that if I keep doing this, as 0 <= gamma <= 0, both qValues are going to rise forever.
Is this the expected behavior of QLearning? Am I doing something wrong? If this is the expected behavior, can’t this lead to problems? I know that probabilistically, all the 4 states(A,B,C and D), will grow at the same rate, but even so it kinda bugs me having them growing forever.
The ideia of allowing the agent to continue exploring even after finding the goal has to do with that the nearer he is from the goal state, the more likely it is to being in states that can be updated at the moment.
This is as expected, since the Q estimate isn’t the expected reward, it’s the expected return, which is the (possibly discounted via gamma) amount of reward I’d expect to reap from that state/action if I started there and followed my policy until the end of the episode or forever.
If you give me some buttons, and one of those buttons always produces $1 when pressed, then the true expected reward for pressing that button is $1. But the true expected return for pressing the button is infinity dollars, assuming I get infinite number of chances to push a button.