br.unicamp.cst.learning

Class QLearning

java.lang.Object

br.unicamp.cst.learning.QLearning

public class QLearning
extends java.lang.Object

The update equation for TD Q-learning is: Q(s,a)= Q(s,a) + alpha * (r(s) + gamma * Max(s', all actions) - Q(s,a)) which is calculated whenever action a is executed in state s leading to state s'. acting randomly on some fraction of steps, where the fraction decreases over time, we can dispense with keeping statistics about taken actions. [1] Ganapathy 2009 "Utilization of Webots and the Khepera II as a Platform for Neural Q-Learning Controllers" [2] http://people.revoledu.com/kardi/tutorial/ReinforcementLearning/Q-Learning-Matlab.htm [3] Norvig

Constructor Summary

Constructors

Constructor and Description
QLearning() Default Constructor.

Method Summary

Modifier and Type	Method and Description
void	clearQ() Clear Q values.
java.lang.String	getA() Gets A value.
java.lang.String	getAction(java.lang.String state) Selects the best action for this state with probability "e", and a random one with probability (1-e) If a given state has no record of one or more actions, it will consider them as valued 0.
java.util.ArrayList<java.lang.String>	getActionsList() Gets the action list.
java.lang.String	getAl() Gets the AL value.
java.lang.String	getAllActionsFromState(java.lang.String state) Gets all actions from state.
java.util.HashMap	getAllQ() Gets all Q values.
double	getAlpha() Gets alpha value.
double	getB() Gets B value.
double	getE() Gets E value.
double	getGamma() Gets gamma value.
double	getQ(java.lang.String state, java.lang.String action) Returns the utility value Q related to the given state/action pair
java.lang.String	getS() Gets S value.
java.lang.String	getSl() Gets SL value.
java.util.ArrayList<java.lang.String>	getStatesList() Gets states list.
double	maxQsl(java.lang.String sl) Returns the maximum Q value for sl.
double	maxwellBoltzmann() This method "maxwellBoltzmann()" returns a pseudo-random value from a Maxwell-Boltzmann distribution with parameter sigma.
void	printQ() This print Q values.
void	recoverQ() Recover Q values from file in JSON structure.
void	setA(java.lang.String a) Sets A value.
void	setActionsList(java.util.ArrayList<java.lang.String> actionsList) This sets action list.
void	setAl(java.lang.String al) Sets AL value.
void	setAlpha(double alpha) Sets the learning rate parameter alpha.
void	setB(double b) Sets B value.
void	setE(double e) Sets the chances of getting the best possible action.
void	setGamma(double gamma) Sets the discount factor.
void	setQ(double Qval, java.lang.String state, java.lang.String action) This method set Q value with parameters Qval, state and action.
void	setS(java.lang.String s) Sets S value.
void	setSl(java.lang.String sl) Sets SL value.
void	setStatesList(java.util.ArrayList<java.lang.String> statesList) Sets action list.
void	storeQ() Store Q values to file using JSON structure.
void	update(java.lang.String stateIWas, java.lang.String actionIDid, double rewardIGot) This methods is responsible for update the state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

QLearning

public QLearning()

Default Constructor.

Method Detail

setQ

public void setQ(double Qval,
                 java.lang.String state,
                 java.lang.String action)

This method set Q value with parameters Qval, state and action.

Parameters:

Qval -

state -

action -

getQ

public double getQ(java.lang.String state,
                   java.lang.String action)

Returns the utility value Q related to the given state/action pair

Parameters:

state -

action -

Returns:

maxQsl

public double maxQsl(java.lang.String sl)

Returns the maximum Q value for sl.

Parameters:

sl -

Returns:

Q Value

update

public void update(java.lang.String stateIWas,
                   java.lang.String actionIDid,
                   double rewardIGot)

This methods is responsible for update the state.

Parameters:

stateIWas - state I was previously

actionIDid - action I did while at the previous state

rewardIGot - reward I got after moving from previous state to the present one

printQ

public void printQ()

This print Q values.

storeQ

public void storeQ()

Store Q values to file using JSON structure.

recoverQ

public void recoverQ()

Recover Q values from file in JSON structure.

clearQ

public void clearQ()

Clear Q values.

getAlpha

public double getAlpha()

Gets alpha value.

Returns:

setAlpha

public void setAlpha(double alpha)

Sets the learning rate parameter alpha. Should be between 0 and 1

Parameters:

alpha -

getGamma

public double getGamma()

Gets gamma value.

Returns:

setGamma

public void setGamma(double gamma)

Sets the discount factor. Should be between 0 and 1.

Parameters:

gamma -

getAction

public java.lang.String getAction(java.lang.String state)

Selects the best action for this state with probability "e", and a random one with probability (1-e) If a given state has no record of one or more actions, it will consider them as valued 0.

Parameters:

state -

Returns:

selectedAction

getStatesList

public java.util.ArrayList<java.lang.String> getStatesList()

Gets states list.

Returns:

the statesList

setStatesList

public void setStatesList(java.util.ArrayList<java.lang.String> statesList)

Sets action list.

Parameters:

statesList - the statesList to set

getActionsList

public java.util.ArrayList<java.lang.String> getActionsList()

Gets the action list.

Returns:

the actionsList

setActionsList

public void setActionsList(java.util.ArrayList<java.lang.String> actionsList)

This sets action list.

Parameters:

actionsList - the actionsList to set

getE

public double getE()

Gets E value.

Returns:

e

setE

public void setE(double e)

Sets the chances of getting the best possible action. With e=0.9 for instance, there is a .9 chance of getting the best action for the given state, and .1 probability of getting a random action.

Parameters:

e - the e to set

getAllActionsFromState

public java.lang.String getAllActionsFromState(java.lang.String state)

Gets all actions from state.

Parameters:

state -

Returns:

actons

maxwellBoltzmann

public double maxwellBoltzmann()

This method "maxwellBoltzmann()" returns a pseudo-random value from a Maxwell-Boltzmann distribution with parameter sigma. Take the sum of the squares of three gaussian random variables with mean 0, and standard deviation sigma, and return the square root. double e = random.nextGaussian(); // Gaussian with mean 0 and stddev = 1

Returns:

sum

getB

public double getB()

Gets B value.

Returns:

the b

setB

public void setB(double b)

Sets B value.

Parameters:

b - the b to set

getS

public java.lang.String getS()

Gets S value.

Returns:

the s

setS

public void setS(java.lang.String s)

Sets S value.

Parameters:

s - the s to set

getA

public java.lang.String getA()

Gets A value.

Returns:

the a

setA

public void setA(java.lang.String a)

Sets A value.

Parameters:

a - the a to set

getSl

public java.lang.String getSl()

Gets SL value.

Returns:

the sl

setSl

public void setSl(java.lang.String sl)

Sets SL value.

Parameters:

sl - the sl to set

getAl

public java.lang.String getAl()

Gets the AL value.

Returns:

the al

setAl

public void setAl(java.lang.String al)

Sets AL value.

Parameters:

al - the al to set

getAllQ

public java.util.HashMap getAllQ()

Gets all Q values.

Returns: