I’ve talked about the vanishing gradient problem in one old post in normal multiple layer neural networks. Pascanur et al. (the first in References below) particularly discussed the vanishing gradient problem as well as another type of gradient instable issue, the exploding gradient problem in the scope of recurrent neural network. Let’s recap the …
Category Archives: Algorithm
Sparse AutoEncoder
Andrew Ng Tutorial: https://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf UFLDL Exercise: http://ufldl.stanford.edu/wiki/index.php/Exercise:Sparse_Autoencoder A Chinese blog: http://www.cnblogs.com/tornadomeet/archive/2013/03/20/2970724.html Stacked Denoising Autoencoder (paper): http://jmlr.csail.mit.edu/papers/volume11/vincent10a/vincent10a.pdf
Convolutional Neural Network Simple Tutorial
I am going to demonstrate a simple version of convolutional neural network, a type of deep learning structure in this post. Motivation Knowing normal multi-layer neural network (probably the one with input layer, output layer and one hidden layer) is helpful before you proceed reading this post. A good tutorial of MLN can be found …
Continue reading “Convolutional Neural Network Simple Tutorial”
How does gradient vanish in Multi-Layer Neural Network?
Background This post reviews how we update weights using the back propagation approach in a neural network. The goal of the review is to illustrate a notorious phenomenon in training MLNN, called “gradient vanish”. Start Let’s suppose that we have a very simple NN structure, with only one unit in each hidden layer, input layer …
Continue reading “How does gradient vanish in Multi-Layer Neural Network?”
Logical deduction in NP-completeness proof
Maybe it sounds simple to computer scientists, I just want to backup some logical deduction of myself in NP-completeness proofs in case I will deal with such proofs in the future. In NP-completeness proof, we always want to find a polynomial-time reduction (transformation) from problem A to problem B, denoted as: We say that there …
Continue reading “Logical deduction in NP-completeness proof”
2-SAT is in P
Proof that BFS finds shortest distance
Intuitively, Breadth-First Search (BFS) can find shortest distances from a source $latex s&s=4$ to any other reachable vertex. However, proving this intuition needs a bit hard work. After some searches, I found that there are two ideas to prove it. The two ideas both use induction to prove it. Both ideas require a bit long proof. …
How is Logistic Regression designed？
背景 Logistic Regression是ML中再熟悉不过的Model了，它能基于数据X，得出生成binary label的概率： （在上式中，X仅有一个feature） 假设你出生在Logistic Regression被发明之前且在Normal Linear Regression被发明之后，现在让你设计一个Model来预测Binary的label——Y，使得这个Model能够基于观测数据X得出Y。你会怎么设计呢？ (注：接下来我们都假设X仅有一个feature）