LLM Engineering

Everything starts at the tokenizer: byte-level BPE token counts, pair frequencies, vocabulary growth and merge savings, WordPiece pieces, tokenizer fertility, characters per token, vocabulary coverage, special-token budgets, and chunk lengths.

Tokens become vectors: skip-gram context pairs, CBOW averaging, negative-sampling probabilities, word2vec analogies, top-k cosine ranking, sentence mean pooling, whitening transforms, and rotary position application.

Inside the block: scaled attention scores, softmax weights, attention scaling factors, entropy and row sums, temperature, cross-attention, RMSNorm, and SwiGLU gating.

Generate and serve: greedy decoding, two-step beam search, top-p and top-k filtering, temperature-adjusted entropy, repetition penalties, KV-cache appends and sizing, and continuous batching rounds.

Retrieve, rank, and judge: RAG top-k retrieval, dense passage scores, hybrid score fusion, reciprocal rank fusion, precision and recall at k, MRR, nDCG, expected calibration error, and DPO loss.