Deterministic Policy Gradient Algorithms

Background

優化目標

隨機策略梯度理論

這個公式使得隨機策略梯度變為簡單的計算一個期望

Off-Policy Actor-Critic

Gradients of Deterministic Policies

Action-Value Gradients

對於連續的情況，使策略引數的移動方向正比於

所以

However, the theory below shows that, like the stochastic policy gradient theorem, there is no need to compute the gradient of the state distribution; and that the intuitive update outlined above is following precisely the gradient of the performance objective

隨機性策略取極限

Deterministic Policy Gradient Theorem

Deterministic Actor-Critic Algorithms

On-Policy Deterministic Actor-Critic

Off-Policy Deterministic Actor-Critic

目標函式變為target policy

We note that stochastic off-policy actor-critic algorithms typically use importance sampling for both actor and critic(Degris et al., 2012b). However, because the deterministic policy gradient removes the integral over actions, we can avoid importance sampling in the actor; and by using Q-learning, we can avoid importance sampling in the critic

Compatible Function Approximation