A field reference for how these systems actually work: attention, training dynamics that fail quietly, inference kernels, the interpretability toolkit, and the attack surface that follows.https://aiwiki.thehackersbrain.dev/enhttps://aiwiki.thehackersbrain.dev/entries/multi-head-attention/https://aiwiki.thehackersbrain.dev/entries/multi-head-attention/h attention operations in d/h-dimensional subspaces, summed back into the residual stream.Tue, 01 Sep 2026 00:00:00 GMTArchitecturesattentiontransformercorehttps://aiwiki.thehackersbrain.dev/notes/building-the-attention-weights/https://aiwiki.thehackersbrain.dev/notes/building-the-attention-weights/From "which positions matter" to softmax over scaled dot products, by elimination.Tue, 01 Sep 2026 00:00:00 GMTNotesattentionfoundationsmathshttps://aiwiki.thehackersbrain.dev/notes/the-quadratic-problem/https://aiwiki.thehackersbrain.dev/notes/the-quadratic-problem/Where the n² really comes from, and why the escape routes divide into exactly three kinds.Tue, 01 Sep 2026 00:00:00 GMTNotesattentionefficiencyfoundationshttps://aiwiki.thehackersbrain.dev/notes/why-attention-exists/https://aiwiki.thehackersbrain.dev/notes/why-attention-exists/The fixed-size bottleneck that broke sequence-to-sequence models, and the obvious fix nobody could take.Tue, 01 Sep 2026 00:00:00 GMTNotesattentionhistoryfoundationshttps://aiwiki.thehackersbrain.dev/entries/attention-sinks/https://aiwiki.thehackersbrain.dev/entries/attention-sinks/Softmax must sum to one, so heads need somewhere to dump mass; evict it and generation collapses.Mon, 31 Aug 2026 00:00:00 GMTArchitecturesattentioninferencelong-contexthttps://aiwiki.thehackersbrain.dev/notes/reading-an-attention-matrix/https://aiwiki.thehackersbrain.dev/notes/reading-an-attention-matrix/What the n×n matrix contains, what the pretty heatmaps show, and how much of it is an artefact.Mon, 31 Aug 2026 00:00:00 GMTNotesattentioninterpretabilityfoundationshttps://aiwiki.thehackersbrain.dev/entries/additive-attention/https://aiwiki.thehackersbrain.dev/entries/additive-attention/The original mechanism: a learned MLP scores each query–key pair.Sun, 30 Aug 2026 00:00:00 GMTArchitecturesattentionhistoryfoundationshttps://aiwiki.thehackersbrain.dev/entries/llm-as-judge/https://aiwiki.thehackersbrain.dev/entries/llm-as-judge/Using a model to score model outputs, and the biases that come with it.Sun, 30 Aug 2026 00:00:00 GMTEvaluationmetricpreferencehttps://aiwiki.thehackersbrain.dev/entries/linear-attention/https://aiwiki.thehackersbrain.dev/entries/linear-attention/Drop the softmax, reassociate the matmuls, and decode in constant memory.Sat, 29 Aug 2026 00:00:00 GMTArchitecturesattentionefficiencyrecurrencehttps://aiwiki.thehackersbrain.dev/entries/chinchilla-scaling-laws/https://aiwiki.thehackersbrain.dev/entries/chinchilla-scaling-laws/Compute-optimal allocation between parameters and training tokens.Fri, 28 Aug 2026 00:00:00 GMTTrainingscalingtraininghttps://aiwiki.thehackersbrain.dev/entries/scaled-dot-product-attention/https://aiwiki.thehackersbrain.dev/entries/scaled-dot-product-attention/Content-based retrieval over key–value pairs with 1/√d_k logit scaling.Fri, 28 Aug 2026 00:00:00 GMTArchitecturesattentiontransformercorehttps://aiwiki.thehackersbrain.dev/entries/sparse-autoencoders/https://aiwiki.thehackersbrain.dev/entries/sparse-autoencoders/Overcomplete dictionary learning on activations for monosemantic features.Fri, 28 Aug 2026 00:00:00 GMTInterpretabilitysparsefeatureshttps://aiwiki.thehackersbrain.dev/entries/multi-head-latent-attention/https://aiwiki.thehackersbrain.dev/entries/multi-head-latent-attention/Cache a shared low-rank latent instead of per-head keys and values.Thu, 27 Aug 2026 00:00:00 GMTArchitecturesattentionkv-cacheinferencehttps://aiwiki.thehackersbrain.dev/entries/flash-attention/https://aiwiki.thehackersbrain.dev/entries/flash-attention/Tiled, IO-aware attention kernel; recomputes softmax statistics online.Wed, 26 Aug 2026 00:00:00 GMTInferenceattentionkernelsmemoryhttps://aiwiki.thehackersbrain.dev/entries/grouped-query-attention/https://aiwiki.thehackersbrain.dev/entries/grouped-query-attention/g key–value heads shared across h query heads; the dial between MHA and MQA.Wed, 26 Aug 2026 00:00:00 GMTArchitecturesattentionkv-cacheinferencehttps://aiwiki.thehackersbrain.dev/entries/speculative-decoding/https://aiwiki.thehackersbrain.dev/entries/speculative-decoding/Draft model proposes tokens; target model verifies in one pass.Wed, 26 Aug 2026 00:00:00 GMTInferencedecodinglatencyhttps://aiwiki.thehackersbrain.dev/entries/block-sparse-attention/https://aiwiki.thehackersbrain.dev/entries/block-sparse-attention/Local windows plus a few global tokens, chosen before the scores are computed.Tue, 25 Aug 2026 00:00:00 GMTArchitecturesattentionsparselong-contexthttps://aiwiki.thehackersbrain.dev/entries/direct-preference-optimisation/https://aiwiki.thehackersbrain.dev/entries/direct-preference-optimisation/Closed-form preference objective that removes the reward model.Tue, 25 Aug 2026 00:00:00 GMTTrainingRLalignmenthttps://aiwiki.thehackersbrain.dev/entries/grpo/https://aiwiki.thehackersbrain.dev/entries/grpo/Group-relative advantages from sampled completions; no value network.Tue, 25 Aug 2026 00:00:00 GMTTrainingRLalignmenthttps://aiwiki.thehackersbrain.dev/entries/lora/https://aiwiki.thehackersbrain.dev/entries/lora/Low-rank additive update to frozen weight matrices during adaptation.Tue, 25 Aug 2026 00:00:00 GMTFine-tuningfine-tuningpefthttps://aiwiki.thehackersbrain.dev/entries/causal-masking/https://aiwiki.thehackersbrain.dev/entries/causal-masking/The triangular mask that turns one forward pass into n training examples.Mon, 24 Aug 2026 00:00:00 GMTArchitecturesattentiontrainingcorehttps://aiwiki.thehackersbrain.dev/entries/induction-heads/https://aiwiki.thehackersbrain.dev/entries/induction-heads/A two-head circuit that completes [A][B]…[A] with [B]; the substrate of in-context learning.Sun, 23 Aug 2026 00:00:00 GMTInterpretabilitycircuitscausalhttps://aiwiki.thehackersbrain.dev/entries/swiglu/https://aiwiki.thehackersbrain.dev/entries/swiglu/Gated feed-forward block; a Swish gate multiplied into a linear branch.Fri, 21 Aug 2026 00:00:00 GMTArchitecturesffncorehttps://aiwiki.thehackersbrain.dev/entries/cross-attention/https://aiwiki.thehackersbrain.dev/entries/cross-attention/Queries from one sequence, keys and values from another.Thu, 20 Aug 2026 00:00:00 GMTArchitecturesattentionconditioningmultimodalhttps://aiwiki.thehackersbrain.dev/entries/state-space-models/https://aiwiki.thehackersbrain.dev/entries/state-space-models/Linear recurrence with input-dependent dynamics; Mamba and the S4 line.Wed, 19 Aug 2026 00:00:00 GMTArchitecturessequencerecurrencesparsehttps://aiwiki.thehackersbrain.dev/entries/activation-patching/https://aiwiki.thehackersbrain.dev/entries/activation-patching/Causal intervention swapping activations between clean and corrupt runs.Tue, 18 Aug 2026 00:00:00 GMTInterpretabilitycausalcircuitshttps://aiwiki.thehackersbrain.dev/entries/proximal-policy-optimisation/https://aiwiki.thehackersbrain.dev/entries/proximal-policy-optimisation/Clipped surrogate advantage objective with KL control to a reference.Tue, 18 Aug 2026 00:00:00 GMTTrainingRLalignmenthttps://aiwiki.thehackersbrain.dev/entries/qlora/https://aiwiki.thehackersbrain.dev/entries/qlora/4-bit NF4 base weights with paged optimiser state and LoRA adapters.Tue, 18 Aug 2026 00:00:00 GMTFine-tuningfine-tuningquantisationhttps://aiwiki.thehackersbrain.dev/entries/rotary-position-embedding/https://aiwiki.thehackersbrain.dev/entries/rotary-position-embedding/Relative position injected as a per-pair rotation in complex subspaces.Tue, 18 Aug 2026 00:00:00 GMTArchitecturespositionattentionhttps://aiwiki.thehackersbrain.dev/entries/statistical-power/https://aiwiki.thehackersbrain.dev/entries/statistical-power/Error bars, paired tests and how many items an eval actually needs.Tue, 18 Aug 2026 00:00:00 GMTEvaluationmetricdata