It has been a while since I learned GPU knowledge. I am going to keep updating more recent materials for ramping up my GPU knowledge. FlashAttention [1] We start from recapping the standard Self-Attention mechanism, which is computed in 3-passes: Notes: The shape and represent the sequence length and internal dimension, respectively. , , , …
Category Archives: Network Technology
Minimal examples of HuggingFace LLM training
Chrome extension for saving web pages as pdfs as exactly they look
Terms you need to know when working for Ads
Data Parallelism and Model Parallelism
Someone just saved this website: WordPress backup and Crayon Syntax Highlighter
Cross entropy with logits
I keep forgetting the exact formulation of `binary_cross_entropy_with_logits` in pytorch. So write this down for future reference. The function binary_cross_entropy_with_logits takes as two kinds of inputs: (1) the value right before the probability transformation (softmax) layer, whose range is (-infinity, +infinity); (2) the target, whose values are binary binary_cross_entropy_with_logits calculates the following loss (i.e., negative …
mujoco only works with gcc8
Make PDFs search able
I just found a useful library that converts scanned, image-based pdfs into searchable pdfs. The library is named OCRmyPDF and can be found here: https://ocrmypdf.readthedocs.io/en/latest/installation.html#
Download and process Chinese songs from Youtube
This posts introduces the way to download Chinese songs from a playlist on youtube and process titles of songs. I use youtube-dl to download all songs from a playlist (replace the youtube link with your own, make sure the playlist is public): youtube-dl -i –yes-playlist -x –audio-format mp3 -o “%(title)s.%(ext)s” –audio-quality 0 “https://www.youtube.com/watch?v=4V3hxNyiwaA&index=1&list=PL-VzXmWCFX7iz_hxy6Xb-JXZFs4GGKMdG” Update 2024-1-26: …
Continue reading “Download and process Chinese songs from Youtube”