mlpack::rl::PrioritizedReplay< EnvironmentType > Class Template Reference

Implementation of prioritized experience replay. More...

#include <prioritized_replay.hpp>

Classes
struct	Transition

Public Types
using	ActionType = typename EnvironmentType::Action
	Convenient typedef for action.
using	StateType = typename EnvironmentType::State
	Convenient typedef for state.

Public Member Functions
	PrioritizedReplay ()
	Default constructor.
	PrioritizedReplay (const size_t batchSize, const size_t capacity, const double alpha, const size_t nSteps=1, const size_t dimension=StateType::dimension)
	Construct an instance of prioritized experience replay class. More...
void	Store (StateType state, ActionType action, double reward, StateType nextState, bool isEnd, const double &discount)
	Store the given experience and set the priorities for the given experience. More...
void	GetNStepInfo (double &reward, StateType &nextState, bool &isEnd, const double &discount)
	Get the reward, next state and terminal boolean for nth step. More...
arma::ucolvec	SampleProportional ()
	Sample some experience according to their priorities. More...
void	Sample (arma::mat &sampledStates, std::vector< ActionType > &sampledActions, arma::rowvec &sampledRewards, arma::mat &sampledNextStates, arma::irowvec &isTerminal)
	Sample some experience according to their priorities. More...
void	UpdatePriorities (arma::ucolvec &indices, arma::colvec &priorities)
	Update priorities of sampled transitions. More...
const size_t &	Size ()
	Get the number of transitions in the memory. More...
void	BetaAnneal ()
	Annealing the beta.
void	Update (arma::mat target, std::vector< ActionType > sampledActions, arma::mat nextActionValues, arma::mat &gradients)
	Update the priorities of transitions and Update the gradients. More...
const size_t &	NSteps () const
	Get the number of steps for n-step agent.

Detailed Description

template<typename EnvironmentType>
class mlpack::rl::PrioritizedReplay< EnvironmentType >

Implementation of prioritized experience replay.

Prioritized experience replay can replay important transitions more frequently by prioritizing transitions, and make agent learn more efficiently.

@article{schaul2015prioritized,
 title   = {Prioritized experience replay},
 author  = {Schaul, Tom and Quan, John and Antonoglou,
            Ioannis and Silver, David},
 journal = {arXiv preprint arXiv:1511.05952},
 year    = {2015}
 }

Template Parameters

EnvironmentType

Desired task.

Constructor & Destructor Documentation

PrioritizedReplay()

template<typename EnvironmentType >

mlpack::rl::PrioritizedReplay< EnvironmentType >::PrioritizedReplay ( const size_t  batchSize, const size_t  capacity, const double  alpha, const size_t  nSteps = 1, const size_t  dimension = StateType::dimension  )

inline

Construct an instance of prioritized experience replay class.

Parameters

batchSize	Number of examples returned at each sample.
capacity	Total memory size in terms of number of examples.
alpha	How much prioritization is used.
nSteps	Number of steps to look in the future.
dimension	The dimension of an encoded state.

Member Function Documentation

GetNStepInfo()

template<typename EnvironmentType >

void mlpack::rl::PrioritizedReplay< EnvironmentType >::GetNStepInfo ( double &  reward, StateType &  nextState, bool &  isEnd, const double &  discount  )

inline

Get the reward, next state and terminal boolean for nth step.

Parameters

reward	Given reward.
nextState	Given next state.
isEnd	Whether next state is terminal state.
discount	The discount parameter.

Sample()

template<typename EnvironmentType >

void mlpack::rl::PrioritizedReplay< EnvironmentType >::Sample ( arma::mat &  sampledStates, std::vector< ActionType > &  sampledActions, arma::rowvec &  sampledRewards, arma::mat &  sampledNextStates, arma::irowvec &  isTerminal  )

inline

Sample some experience according to their priorities.

Parameters

sampledStates	Sampled encoded states.
sampledActions	Sampled actions.
sampledRewards	Sampled rewards.
sampledNextStates	Sampled encoded next states.
isTerminal	Indicate whether corresponding next state is terminal state.

SampleProportional()

template<typename EnvironmentType >

arma::ucolvec mlpack::rl::PrioritizedReplay< EnvironmentType >::SampleProportional ( )

inline

Sample some experience according to their priorities.

Returns

The indices to be chosen.

Size()

template<typename EnvironmentType >

const size_t& mlpack::rl::PrioritizedReplay< EnvironmentType >::Size ( )

inline

Get the number of transitions in the memory.

Returns

Actual used memory size.

Store()

template<typename EnvironmentType >

void mlpack::rl::PrioritizedReplay< EnvironmentType >::Store ( StateType  state, ActionType  action, double  reward, StateType  nextState, bool  isEnd, const double &  discount  )

inline

Store the given experience and set the priorities for the given experience.

Parameters

state	Given state.
action	Given action.
reward	Given reward.
nextState	Given next state.
isEnd	Whether next state is terminal state.
discount	The discount parameter.

Update()

template<typename EnvironmentType >

void mlpack::rl::PrioritizedReplay< EnvironmentType >::Update ( arma::mat  target, std::vector< ActionType >  sampledActions, arma::mat  nextActionValues, arma::mat &  gradients  )

inline

Update the priorities of transitions and Update the gradients.

Parameters

target	The learned value.
sampledActions	Agent's sampled action.
nextActionValues	Agent's next action.
gradients	The model's gradients.

UpdatePriorities()

template<typename EnvironmentType >

void mlpack::rl::PrioritizedReplay< EnvironmentType >::UpdatePriorities ( arma::ucolvec &  indices, arma::colvec &  priorities  )

inline

Update priorities of sampled transitions.

Parameters

indices	The indices of sample to be updated.
priorities	Their corresponding priorities.

---

The documentation for this class was generated from the following file:

• src/mlpack/methods/reinforcement_learning/replay/prioritized_replay.hpp