In a previous post, we discussed an earlier generative modeling called Normalizing Flows [1]. However, Normalizing Flows has its own limitations: (1) it requires the flow mapping function to be invertible. This limits choices of potential neural network architectures to instantiate , because being invertible means that hidden layers must have the exact same dimensionality …
Continue reading “New progress of generative modeling – flow matching”
It has been a while since I learned GPU knowledge. I am going to keep updating more recent materials for ramping up my GPU knowledge. FlashAttention [1] We start from recapping the standard Self-Attention mechanism, which is computed in 3-passes: Notes: The shape and represent the sequence length and internal dimension, respectively. , , , …
We are entering the second half of AI [1]. Environments and evals are becoming as important as algorithms. In my vision, a real useful consumer AI will be a general agent supporting both GUI and bash environments with real-time voice support. In this post, I’m going to list relevant pointers for building such an agent. …
In this post, let’s visit how modern LLMs encode positional information. We start from the most famous paper in this domain [1] and dive into some key details. Why we need positional encoding LLMs need positional encodings to differentiate different semantic meanings of the same word. We use the motivational example from [2]: The two …