Quick reference untuk konsep LLM.
Topics
Scaling Laws
- Kaplan scaling: loss proportional to (N,D) where N=params, D=data
- Chinchilla scaling: optimal compute = 20 tokens/param
Attention Mechanisms
- Multi-head, Grouped-query, Multi-query, Flash attention
- KV cache optimization
Fine-tuning
- Full fine-tune, LoRA, QLoRA, Adapter, Prefix tuning
- RLHF vs DPO
Infrastructure
- vLLM, TensorRT-LLM, TGI
- Quantization: AWQ, GPTQ, GGUF
See also: llm-finetuning-toolchain, test-time-compute-system2