State-Action-Reward-State-Action (SARSA) and Q-learning are two forms of reinforcement learning. The difference of the two methods are discussed in: https://studywolf.wordpress.com/2013/07/01/reinforcement-learning-sarsa-vs-q-learning/ http://stackoverflow.com/questions/6848828/reinforcement-learning-differences-between-qlearning-and-sarsatd http://stats.stackexchange.com/questions/184657/difference-between-off-policy-and-on-policy-learning Let’s explain why Q-learning is called off-policy learning and SARSA is called on-policy learning. Suppose at state $latex s_t$, a method takes action $latex a_t$ which results to land in a new state …
Category Archives: Algorithm
Why the greedy algorithm of maximum weighted matching is a 2-approximation?
This post explains my understanding in a proposed greedy algorithm for the maximum weighted matching problem. The greedy algorithm goes as follows (listed by this paper in Introduction section): It is claimed that the greedy algorithm is a 2 approximation, i.e., greedy result >= 1/2 optimal result. The document where the greedy algorithm is proposed is …
Continue reading “Why the greedy algorithm of maximum weighted matching is a 2-approximation?”
Theano LSTM Code Walk Through
In this post, I am going to explain the code (as much as I can) from theano LSTM tutorial: http://deeplearning.net/tutorial/lstm.html You need to first understand LSTM. Here is an online recommended material: http://colah.github.io/posts/2015-08-Understanding-LSTMs/, in which many beautiful figures are provided to illustrate LSTM step by step. The tutorial aims to predict positive/negative sentiment based on movie reviews …
Overview for Sequential Data Learning
Hidden Markov Model You should bear in mind clearly the three questions people usually ask for Hidden Markov Model: 1. what is the probability of an observed sequence? 2. what is the most likely series of states given a specific observed observation? 3. Given a set of observations, what are the values of the state …
Adaptive Regularization of Weight Vectors — Mathematical derivation
In this post I am showing my understanding about the paper Adaptive Regularization of Weight Vectors: http://papers.nips.cc/paper/3848-adaptive-regularization-of-weight-vectors.pdf The paper aims to address the negatives of a previous algorithm called confidence weighted (CW) learning by introducing the algorithm Adaptive Regularization Of Weights (AGOW). CW and AGOW are both online learning algorithms, meaning updates happen after observing each …
Continue reading “Adaptive Regularization of Weight Vectors — Mathematical derivation”
Math Derivation for Bayesian Probabilistic Matrix Factorization Model
In this paper I am trying to derive the mathematical formulas that appear in the paper Bayesian Probabilistic Matrix Factorization using Markov Chain Monte Carlo. We will use exactly the same notations as in the paper. The part that I am interested in is the Inference part (Section 3.3). Sample $latex U_i$ In Gibbs sampling, …
Continue reading “Math Derivation for Bayesian Probabilistic Matrix Factorization Model”
Estimate best K for K-Means in parallel
Gap statistic is often used to determine the best number of clusters. Please see a local version implementation for gap statistic here: https://github.com/echen/gap-statistic. It is often desired to parallelize such tedious job to boost the speed. I implement a parallelized version basd on the source code: library(plyr) library(ggplot2) # Calculate log(sum_i(within-cluster_i sum of squares around cluster_i …
How to make match prediction using TrueSkill?
TrueSkill [1] is a widely accepted algorithm for matchmaking, in which player skills are initialized with a random variable following Gaussian distribution and then get updated through team-based match outcomes. Eventually, with sufficient amount of matches, player skill random variables will converge to some means with very small variance. There are multiple open sourced libraries …
Continue reading “How to make match prediction using TrueSkill?”
Optimization Overview
In this post, I am summarizing all I know for optimization. It is a hard, primary problem once you define your model with some kind of objective function integrating a bunch of parameters and you start to wonder how you could learn the values of those parameters to minimize (or maximize) the objective function. Unconstrained Optimization …
How to understand Eigenvector/Eigenvalue?
This topic has been in my wish list for a long time. I will try to articulate my understanding of eigenvector and eigenvalue after digestion informative materials from different sources. First of all, we can consider multiplication between a square matrix and a vector as a transformation of such that is rotated and scaled. By …
Continue reading “How to understand Eigenvector/Eigenvalue?”