In a previous post, we discussed an earlier generative modeling called Normalizing Flows [1]. However, Normalizing Flows has its own limitations: (1) it requires the flow mapping function to be invertible. This limits choices of potential neural network architectures to instantiate , because being invertible means that hidden layers must have the exact same dimensionality …
Continue reading “New progress of generative modeling – flow matching”
We are entering the second half of AI [1]. Environments and evals are becoming as important as algorithms. In my vision, a real useful consumer AI will be a general agent supporting both GUI and bash environments with real-time voice support. In this post, I’m going to list relevant pointers for building such an agent. …
In this post, let’s visit how modern LLMs encode positional information. We start from the most famous paper in this domain [1] and dive into some key details. Why we need positional encoding LLMs need positional encodings to differentiate different semantic meanings of the same word. We use the motivational example from [2]: The two …