The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning
V. S. Borkar, S. P. Meyn

abstract
     new results
          simpler derivation of known RL results
          convergence proof for a class of asynchronous stochastic approximation algorithms
               without a priori stability assumption
          convergence proof for asynchronous adaptive critic and Q learning algorithms
introduction
     d-dimensional recursion
          X(n+1) = X(n) + a(n)[h(X(n)) + M(n+1)] | n≥0
          X(n) = [X₁(n), …, X_d(n)]ᵀ є R^d
          h : R^d → R^d
          {a(n)} a positive number sequence
          {M(n)} an uncorrelated sequence with zero mean
     recurrence assumption
          iterates remain bounded with probability 1
          or
          visits bounded set infinitely often with probability 1
     o.d.e.
          x(t) = h(x(t))
          x(0) = X(0)
     main idea
          Lyapunov functions require noise time-averaging
          requires infeasable calculation
          combine time averaging with a limiting operation
          replace stochastic system with deterministic