Author Archives: czxttkl

Why TD($latex lambda$)? Why actor-critic? Why eligibility trace? Why contextual regret minimization?

Importance sampling is a way to reduce variance of your estimation on integration over a region for an integrand. Let’s first see how traditional Monte Carlo method is used to estimate integration [2]. To estimate $latex \int_a^b f(x) dx$, one can think of reshaping the area to be integrated as a rectangle, whose width is …

Continue reading “Importance sampling”

In my rough understanding, inverse reinforcement learning is a branch of RL research in which people try to perform state-action sequences resembling given tutor sequences. There are two famous works on inverse reinforcement learning. One is Apprenticeship Learning via Inverse Reinforcement Learning [1], and the other is Maximum Margin Planning [2]. Maximum Margin Planning In …

Continue reading “Inverse Reinforcement Learning”

Here are some materials I found useful to learn Reinforcement Learning (RL). Let’s first look at Markov Decision Process (MDP), in which you know a transition function $latex T(s,a,s’)$ and a reward function $latex R(s,a,s’)$. In the diagram below, the green state is called “q state”.  Some notations that need to be clarified: Dynamic programming …

Continue reading “Reinforcement learning overview”

I am introducing some basic definitions of abstract algebra, structures like monoid, groups, rings, fields and vector spaces and homomorphism/isomorphism. I find the clear definitions of structures from [1]: Also, the tables below show a clear comparisons between several structures [2,3]:   All these structures are defined with both a set and operation(s). Based on [4], …

Continue reading “Abstract Algebra”

Dijkstra algorithm is a well known algorithm for finding exact distance from a source to a destination. In order to improve the path finding speed, A* algorithm combines heuristics and known distances to find the heuristically best path towards a goal. A common A* implementation maintains an open set for discovered yet not evaluated nodes and a closed …

Continue reading “When A* algorithm returns optimal solution”