Found in the Wild (4)

Interesting snippets I stumble upon while browsing the internet. A quote, a take, a revelation, a throwaway line that turns out to be gold. Something that made me pause and think "huh, that's worth sharing."

Under this section, you will find my collection of raw quotes - presented as-is, with just enough context to point you toward the source, and a nudge to look closer.

Local LLMs

Understanding LLM Size, Weights, Parameters, Quantization, KV Cache & Inference Memory

How much RAM do you need to run a 30 billion parameter model? Why are there multiple versions of the same model at different file sizes? What does "8-bit quantization" actually mean, and how does it affect performance and/or precision? If you're running language models locally or planning to, understanding the relationship between parameters, weights, quantization, and memory is essential.