Recursive Language Models and Advanced LLM Architectures [10/2026, ENG]
movies, software, shows...
Files
| 1 - Section 1 The Architecture Landscape and Your Bench | |||
| 1 - Course Documents and How to Use Them.mp4 | 28.96 MB | ||
| 1 - D01_Reading_an_Architecture_Papers_Claim_Checklist_and_Worked_Examples.docx | 26.37 KB | ||
| 1 - D02_Fair_Architecture_Comparison_Protocol_Rules_bench_py_Harness_and_Pre_Run_Checklist.docx | 25.51 KB | ||
| 1 - D03_CPU_Lab_Bench_Setup_and_Reproducibility_Checklist.docx | 25.11 KB | ||
| 1 - D04_Recursion_in_Neural_Networks_Field_Guide_Tree_Networks_Looped_Depth_and_RLMs.docx | 24.18 KB | ||
| 1 - D05_Recursive_Language_Models_in_Practice_Minimal_Loop_Trace_Reading_Safe_REPL.docx | 24.29 KB | ||
| 1 - D06_Attention_Cost_and_KV_Cache_Sizing_Worksheet.docx | 21.79 KB | ||
| 1 - D07_State_Space_Linear_Recurrent_and_Hybrid_Models_Reference_Card.docx | 22.57 KB | ||
| 1 - D08_Mixture_of_Experts_Design_and_Router_Health_Checklist.docx | 23.75 KB | ||
| 1 - D09_RAG_Pipeline_Evaluation_Template_Labelled_Set_Retrieval_Metrics_Faithfulness.docx | 22.93 KB | ||
| 1 - D10_Results_Table_and_Experiment_Log_Template.docx | 21.31 KB | ||
| 1 - D11_Architecture_Selection_Report_Template_with_the_Capstone_Worked_Example.docx | 22.58 KB | ||
| 10 - Section 1 Wrap The Map, the Bench and the Rules of Fair Measurement.mp4 | 11.67 MB | ||
| 2 - Section 1 Intro Why Look Past the Plain Transformer.mp4 | 7.36 MB | ||
| 3 - What Advanced Architecture Means in 2026 Attention, Recurrence, Sparsity.mp4 | 33.28 MB | ||
| 4 - Three Meanings of Recursive Tree Networks, Looped Depth and RLMs.mp4 | 14.55 MB | ||
| 5 - The Quadratic Wall Where Attention's Cost and the KV Cache Actually Bite.mp4 | 22.52 MB | ||
| 6 - Reading a Paper's Claim Correctly Hardware, Scale, Setup and Version.mp4 | 25.06 MB | ||
| 7 - [LAB] The Bench 8 CPUs, No GPU — Measuring What This Machine Can Do.mp4 | 17.53 MB | ||
| 8 - [LAB] Building the Lab Environment CPU PyTorch, Transformers, Pinned Versions.mp4 | 18.73 MB | ||
| 9 - [LAB] Opening archlab The Course Repository and Its Empty Results Table.mp4 | 15.45 MB | ||
| 10 - Section 10 RAG Patterns That Work in Production | |||
| 87 - Section 10 Intro Retrieval as External Memory You Can Update.mp4 | 14.24 MB | ||
| 88 - RAG From Lewis et al. (2020) to Today The Moving Parts and Their Failure Points.mp4 | 26.9 MB | ||
| 89 - [LAB] Chunking Strategies on One Corpus Fixed, Recursive, Sentence-Aware.mp4 | 32.27 MB | ||
| 90 - [LAB] Embeddings on CPU Choosing a Small Model, Encoding, Checking Its Licence.mp4 | 26.65 MB | ||
| 91 - [LAB] Vector Indexes With FAISS Flat vs HNSW vs IVF, Recall and Latency.mp4 | 28.94 MB | ||
| 92 - [LAB] BM25 Plus Dense Hybrid Search With Reciprocal Rank Fusion.mp4 | 27.21 MB | ||
| 93 - [LAB] Re-Ranking With a Cross-Encoder What It Adds, What It Costs.mp4 | 25.43 MB | ||
| 94 - [LAB] Retrieval Evaluation A Labelled Set Scored With Recall@k, MRR and nDCG.mp4 | 30.95 MB | ||
| 95 - [LAB] End-to-End Local RAG Answers With Citations and a Faithfulness Check.mp4 | 36.25 MB | ||
| 96 - Contextual Retrieval, GraphRAG, Self-RAG and CRAG When the Extra Machinery Pays.mp4 | 25 MB | ||
| 97 - [LAB] RAG vs RLM vs Long Prompt on the Same Question Set.mp4 | 37.21 MB | ||
| 98 - Section 10 Wrap A RAG Pipeline You Can Defend With Numbers.mp4 | 6.79 MB | ||
| 11 - Section 11 Benchmarking and Evaluating New Architectures | |||
| 100 - Fair Comparison Rules Same Data, Tokens, Parameters, Seeds — and Bits per Byte.mp4 | 32.91 MB | ||
| 101 - Perplexity Pitfalls Tokenizers, Context Windows and Strided Evaluation.mp4 | 26.7 MB | ||
| 102 - [LAB] Throughput, Time-to-First-Token and Memory on CPU Warm-Up and Spread.mp4 | 27.97 MB | ||
| 103 - [LAB] Long-Context Tests Needle Sweeps and Position Effects on a Small Model.mp4 | 27.17 MB | ||
| 104 - [LAB] Synthetic Probes Selective Copying, Induction Heads, Multi-Query Recall.mp4 | 29.58 MB | ||
| 105 - [LAB] lm-evaluation-harness on CPU One Small Task, Two Architectures.mp4 | 26.96 MB | ||
| 106 - Scaling Laws and What a Toy Experiment Can and Cannot Tell You.mp4 | 28.45 MB | ||
| 107 - Section 11 Wrap The Results Table, Complete and Honest.mp4 | 6.67 MB | ||
| 99 - Section 11 Intro Numbers You Would Bet Your Architecture On.mp4 | 7.11 MB | ||
| 12 - Section 12 Decisions, Trade-Offs and the Capstone | |||
| 108 - Section 12 Intro From Experiments to an Architecture Decision.mp4 | 6.93 MB | ||
| 109 - The Trade-Off Axes Training Cost, Latency, Memory, Context Length, Maturity.mp4 | 28.4 MB | ||
| 110 - Ecosystem Reality Check What vLLM, llama.cpp and transformers Support Today.mp4 | 13.91 MB | ||
| 111 - [LAB] Capstone Part 1 One Task, Three Architectures, One Harness.mp4 | 33.83 MB | ||
| 112 - [LAB] Capstone Part 2 Running the Comparison and Reading the Spread.mp4 | 32.02 MB | ||
| 113 - [LAB] Capstone Part 3 Writing the Architecture Selection Report From Results.mp4 | 27.65 MB | ||
| 114 - Where the Field Is Moving Hybrids, Recursive Inference and Test-Time Memory.mp4 | 27.46 MB | ||
| 115 - Section 12 Wrap What You Can Now Build, Measure and Decide.mp4 | 15.99 MB | ||
| 2 - Section 2 A Transformer Baseline You Built Yourself | |||
| 11 - Section 2 Intro Every Comparison Needs a Baseline.mp4 | 14.59 MB | ||
| 12 - [LAB] Data and Tokenisation A Character-Level Corpus and Its Train Val Split.mp4 | 25.79 MB | ||
| 13 - [LAB] Self-Attention From Scratch in PyTorch, Checked Against the Built-In SDPA.mp4 | 27.79 MB | ||
| 14 - [LAB] A Tiny Decoder-Only Transformer Blocks, Norms, Residuals, Causal Mask.mp4 | 26.94 MB | ||
| 15 - [LAB] Training It on CPU Loss Curves, Steps per Second and Where the Time Goes.mp4 | 34.46 MB | ||
| 16 - [LAB] Loss, Perplexity and Bits per Character — Computed, Not Quoted.mp4 | 20.38 MB | ||
| 17 - [LAB] The Bench Harness One Tool for Speed, Memory and Quality Across Models.mp4 | 23.17 MB | ||
| 18 - [LAB] Sampling From the Baseline and Recording Its Row in the Results Table.mp4 | 22.33 MB | ||
| 19 - Section 2 Wrap A Measured Baseline and the Three Measurement Mistakes.mp4 | 12.57 MB | ||
| 3 - Section 3 Recursive Neural Networks — From Trees to Looped Depth | |||
| 20 - Section 3 Intro Reusing the Same Weights, Again and Again.mp4 | 15.16 MB | ||
| 21 - Recursive vs Recurrent From RAAM (1990) to the Recursive Neural Tensor Network.mp4 | 29.45 MB | ||
| 22 - Tree-LSTM (2015) Gates Over a Parse Tree Instead of a Chain.mp4 | 13.28 MB | ||
| 23 - [LAB] A Recursive Network Over Binary Trees in PyTorch, Step by Step.mp4 | 26.64 MB | ||
| 24 - [LAB] Tree-LSTM on Tiny Arithmetic Expressions Training and Checking Answers.mp4 | 19.78 MB | ||
| 25 - Universal Transformers and Adaptive Depth The Halting Idea.mp4 | 25.56 MB | ||
| 26 - Looped Transformers 2024-2025 Relaxed Recursion and Mixture-of-Recursions.mp4 | 30.27 MB | ||
| 27 - HRM and the Tiny Recursive Model (TRM) Small Networks That Iterate.mp4 | 27.6 MB | ||
| 28 - [LAB] A Weight-Tied Looped Transformer vs the Baseline at Equal Parameters.mp4 | 31.09 MB | ||
| 29 - Section 3 Wrap What Recursion in Depth Buys You, and What It Costs.mp4 | 14.03 MB | ||
| 4 - Section 4 Recursive Language Models (RLM) | |||
| 30 - Section 4 Intro When the Prompt Is Too Long to Read.mp4 | 13.31 MB | ||
| 31 - Context Rot What NIAH, RULER, Lost in the Middle and OOLONG Show.mp4 | 16.28 MB | ||
| 32 - The RLM Idea The Prompt as a Variable in a REPL, Sub-Calls as a Function.mp4 | 27.75 MB | ||
| 33 - RLM vs RAG vs ReAct vs CodeAct Where the Long Input Lives, Who Decides Access.mp4 | 26.12 MB | ||
| 34 - What the RLM Paper Reports Benchmarks, Depth Ablations, Costs and Versions.mp4 | 29.68 MB | ||
| 35 - [LAB] A Minimal RLM Loop by Hand Root Model, Python REPL, llm_query and FINAL.mp4 | 40.53 MB | ||
| 36 - [LAB] Serving a Local Model for the Loop Ollama's OpenAI-Compatible Endpoint.mp4 | 14.56 MB | ||
| 37 - [LAB] Running the Hand-Built RLM on a Needle Task and Reading the Full Trace.mp4 | 19.29 MB | ||
| 38 - [LAB] The Official rlms Library on a Local Model, and Why a Small Root Fails.mp4 | 31.08 MB | ||
| 39 - [LAB] Aggregating a Long Document Chunked Sub-Calls vs One Long Prompt.mp4 | 28.96 MB | ||
| 40 - Stated Limits Cost Blow-Ups, Sequential Calls, Brittle FINAL, REPL Sandboxing.mp4 | 25.72 MB | ||
| 41 - Section 4 Wrap When an RLM Is the Right Tool.mp4 | 12.03 MB | ||
| 5 - Section 5 Efficient Attention and Linear Transformers | |||
| 42 - Section 5 Intro Making Attention Cheaper Without Losing the Plot.mp4 | 7.21 MB | ||
| 43 - Sparse and Local Patterns Longformer, BigBird and Sliding-Window Attention.mp4 | 27.11 MB | ||
| 44 - Low-Rank and Kernel Tricks Linformer, Performer and Transformers Are RNNs.mp4 | 28.78 MB | ||
| 45 - FlashAttention 1 to 4 Exact Attention, IO-Aware — and Why It Is a GPU Story.mp4 | 27.16 MB | ||
| 46 - [LAB] Sliding-Window Attention on the Baseline Memory and Quality vs Window.mp4 | 31.31 MB | ||
| 47 - [LAB] A Linear-Attention Layer From Scratch and Its Recurrent Form.mp4 | 30.85 MB | ||
| 48 - [LAB] Timing Attention on CPU Length Sweeps, Math vs Flash Backend in PyTorch.mp4 | 21.13 MB | ||
| 49 - MQA, GQA and Multi-Head Latent Attention Shrinking the Cache, Not the Compute.mp4 | 27.81 MB | ||
| 50 - Section 5 Wrap Which Attention Trick for Which Bottleneck.mp4 | 11.81 MB | ||
| 6 - Section 6 State Space Models — S4 and Mamba | |||
| 51 - Section 6 Intro A Recurrent Model That Trains Like a Convolution.mp4 | 13.41 MB | ||
| 52 - From Continuous Systems to Discrete Recurrence The SSM Equations Made Concrete.mp4 | 26.65 MB | ||
| 53 - S4, H3 and Hyena Long Convolutions and the Long Range Arena.mp4 | 26.66 MB | ||
| 54 - Mamba's Selective Scan Input-Dependent Parameters, Hardware-Aware Algorithm.mp4 | 26.74 MB | ||
| 55 - [LAB] A Selective SSM Layer From Scratch Discretise, Scan, Gate.mp4 | 34.4 MB | ||
| 56 - [LAB] Sequential vs Chunked Scan on CPU Speed, and Keeping the Numbers Stable.mp4 | 34.47 MB | ||
| 57 - [LAB] Training a Tiny Mamba-Style Model and Adding Its Row to the Results Table.mp4 | 26.42 MB | ||
| 58 - [LAB] Pretrained Mamba-130m vs Pythia-160m on CPU Same Data and Tokenizer.mp4 | 33.64 MB | ||
| 59 - Mamba-2 and SSD, Mamba-3 What Changed and What Needs a GPU.mp4 | 26.28 MB | ||
| 60 - Section 6 Wrap Where SSMs Win, and Where Your CPU Disagrees.mp4 | 13.78 MB | ||
| 7 - Section 7 Hybrids and Modern Linear Recurrences | |||
| 61 - Section 7 Intro Why the 2025 Models Mix Layer Types.mp4 | 14.11 MB | ||
| 62 - RWKV and RetNet RNN Inference, Parallel Training.mp4 | 24.48 MB | ||
| 63 - Gated DeltaNet and Kimi Delta Attention The Delta Rule Returns.mp4 | 24.23 MB | ||
| 64 - Hybrid Stacks in Production Jamba, Zamba, Falcon-H1, Nemotron-H, Qwen3-Next.mp4 | 28.73 MB | ||
| 65 - [LAB] Associative Recall The Task That Separates Attention From Pure SSMs.mp4 | 36.44 MB | ||
| 66 - [LAB] A Hybrid Model SSM Layers Plus One Attention Layer, Trained and Compared.mp4 | 36.78 MB | ||
| 67 - Section 7 Wrap Choosing a Mix, Not a Religion.mp4 | 11.9 MB | ||
| 8 - Section 8 Mixture of Experts | |||
| 68 - Section 8 Intro More Parameters, Same Compute per Token.mp4 | 12.56 MB | ||
| 69 - Sparsely-Gated MoE to Switch Transformer Top-k Routing, Capacity, Token Drops.mp4 | 31.89 MB | ||
| 70 - Load Balancing Auxiliary Loss, Router z-Loss, Expert Choice, Aux-Loss-Free Bias.mp4 | 28.62 MB | ||
| 71 - Fine-Grained and Shared Experts DeepSeekMoE, Mixtral, OLMoE and Qwen MoE.mp4 | 28.9 MB | ||
| 72 - [LAB] An MoE Feed-Forward Layer From Scratch Router, Top-2 and Weighted Combine.mp4 | 31.16 MB | ||
| 73 - [LAB] Training the MoE and Watching Expert Load With and Without Balance Loss.mp4 | 41.88 MB | ||
| 74 - [LAB] Collapse on Purpose What an Unbalanced Router Looks Like in the Logs.mp4 | 32 MB | ||
| 75 - [LAB] A Real MoE on CPU Granite 3.1 1B-A400M Router Choices Token by Token.mp4 | 37.6 MB | ||
| 76 - MoE in Practice Memory, Serving, Expert Parallelism and When Dense Is Better.mp4 | 25.75 MB | ||
| 77 - Section 8 Wrap Sparse Compute, Full Memory.mp4 | 12.26 MB | ||
| 9 - Section 9 Memory-Augmented Networks and the KV Cache | |||
| 78 - Section 9 Intro Where Does a Model Keep What It Has Read.mp4 | 6.85 MB | ||
| 79 - External Memory Neural Turing Machine, Memory Networks and the DNC.mp4 | 32.42 MB | ||
| 80 - Memory for Transformers Transformer-XL, Memorizing Transformers, RMT and Titans.mp4 | 29.75 MB | ||
| 81 - [LAB] A Content-Addressed Memory Module Trained on Associative Recall.mp4 | 32.54 MB | ||
| 82 - [LAB] The KV Cache Measured With and Without It, and Its Size Formula.mp4 | 28.28 MB | ||
| 83 - [LAB] Cache Management in transformers Dynamic, Static, Quantized, Cropping.mp4 | 28.1 MB | ||
| 84 - PagedAttention, Attention Sinks and KV Quantization The Serving View.mp4 | 29.05 MB | ||
| 85 - [LAB] Sliding Window Plus Sinks Keeping Generation Stable Past the Window.mp4 | 31.11 MB | ||
| 86 - Section 9 Wrap Memory Inside, Beside and Outside the Model.mp4 | 13.21 MB |
Description
Обучающие видео » Компьютерные видеоуроки и обучающие интерактивные DVD » Machine/Deep Learning, Neural Networks
Recursive Language Models and Advanced LLM Architectures
Год выпуска: 10/2026
Производитель: Udemy
Сайт производителя: https://www.udemy.com/course/recursive-language-models-and-advanced-llm-architectures/
Автор: NEXUS ACADEMY
Продолжительность: 11h 13m
Тип раздаваемого материала: Видеоурок
Язык: Английский
Описание: This course teaches you to understand modern architecture ideas beyond the plain transformer, implement small versions yourself, and measure them fairly on an ordinary CPU, so you can decide what is worth using despite new papers arriving weekly with claims made on hardware and scales you will never reproduce. You start by building your own bench and lab environment with CPU PyTorch, Hugging Face transformers, and a pinned dependency set, then build a tiny decoder-only transformer from scratch, train it, and record its speed, memory, and quality in a results table—one benchmark harness reused by every architecture that follows so every row is comparable.
Recursion is covered in all three meanings: recursive networks over trees and a Tree-LSTM, weight-tied looped transformers, and architectures like Universal Transformers, Mixture-of-Recursions, HRM, and the Tiny Recursive Model. The Recursive Language Model section builds an RLM loop by hand, where a long prompt lives as a variable in a Python REPL and the root model calls sub-models on pieces of it, running against a local Ollama model, reading the full trace, trying the official library, and seeing where a small root model fails. The middle covers efficiency ideas: sparse, sliding-window, and linear attention implemented and timed; FlashAttention, MQA, GQA, and latent attention explained with honest notes on GPU requirements; a selective state space layer implemented and a tiny Mamba-style model trained, comparing pretrained Mamba and Pythia on the same data; and RWKV, RetNet, Gated DeltaNet, and hybrid stacks explained, with an associative recall task showing why hybrids keep an attention layer.
Mixture of experts gets its own section where you build a router, watch expert load with and without balancing loss, cause router collapse on purpose, and inspect routing choices of a real small MoE model token by token. Memory follows from external memory networks to the KV cache, which you measure, size from model configuration, and manage with transformers cache classes and attention sinks. Retrieval is treated as external memory you can update: comparing chunking strategies, choosing a small embedding model and checking its licence, building FAISS indexes, combining BM25 and dense search, adding a cross-encoder re-ranker, and scoring retrieval with recall, MRR, and nDCG on a labelled set, with a complete local RAG pipeline answering with citations and a faithfulness check compared against an RLM and a long prompt on the same questions. The last sections cover decisions: the rules of fair comparison, the pitfalls of perplexity, how to measure throughput and time to first token properly, and what a toy experiment can and cannot say about scaling. The capstone compares three architectures on one task with one harness and ends with a written architecture selection report. Eleven downloadable documents include a paper-claim checklist, fair comparison protocol, CPU bench setup guide, KV cache sizing and MoE checklists, a RAG evaluation template, and the selection report template, with all numbers measured on the course bench and presented as such.
[spoiler="Содержание"]
1 - Section 1 The Architecture Landscape and Your Bench
1 - Course Documents and How to Use Them (06:50)
2 - Section 1 Intro Why Look Past the Plain Transformer (03:02)
3 - What Advanced Architecture Means in 2026 Attention, Recurrence, Sparsity (07:18)
4 - Three Meanings of Recursive Tree Networks, Looped Depth and RLMs (06:09)
5 - The Quadratic Wall Where Attention's Cost and the KV Cache Actually Bite (05:31)
6 - Reading a Paper's Claim Correctly Hardware, Scale, Setup and Version (05:42)
7 - [LAB] The Bench 8 CPUs, No GPU — Measuring What This Machine Can Do (04:52)
8 - [LAB] Building the Lab Environment CPU PyTorch, Transformers, Pinned Versions (05:39)
9 - [LAB] Opening archlab The Course Repository and Its Empty Results Table (04:22)
10 - Section 1 Wrap The Map, the Bench and the Rules of Fair Measurement (02:52)
2 - Section 2 A Transformer Baseline You Built Yourself
11 - Section 2 Intro Every Comparison Needs a Baseline (03:01)
12 - [LAB] Data and Tokenisation A Character-Level Corpus and Its Train Val Split (05:55)
13 - [LAB] Self-Attention From Scratch in PyTorch, Checked Against the Built-In SDPA (06:45)
14 - [LAB] A Tiny Decoder-Only Transformer Blocks, Norms, Residuals, Causal Mask (06:34)
15 - [LAB] Training It on CPU Loss Curves, Steps per Second and Where the Time Goes (07:17)
16 - [LAB] Loss, Perplexity and Bits per Character — Computed, Not Quoted (06:07)
17 - [LAB] The Bench Harness One Tool for Speed, Memory and Quality Across Models (05:49)
18 - [LAB] Sampling From the Baseline and Recording Its Row in the Results Table (05:45)
19 - Section 2 Wrap A Measured Baseline and the Three Measurement Mistakes (02:58)
3 - Section 3 Recursive Neural Networks — From Trees to Looped Depth
20 - Section 3 Intro Reusing the Same Weights, Again and Again (02:58)
21 - Recursive vs Recurrent From RAAM (1990) to the Recursive Neural Tensor Network (06:44)
22 - Tree-LSTM (2015) Gates Over a Parse Tree Instead of a Chain (05:33)
23 - [LAB] A Recursive Network Over Binary Trees in PyTorch, Step by Step (06:28)
24 - [LAB] Tree-LSTM on Tiny Arithmetic Expressions Training and Checking Answers (05:00)
25 - Universal Transformers and Adaptive Depth The Halting Idea (05:53)
26 - Looped Transformers 2024-2025 Relaxed Recursion and Mixture-of-Recursions (06:49)
27 - HRM and the Tiny Recursive Model (TRM) Small Networks That Iterate (05:49)
28 - [LAB] A Weight-Tied Looped Transformer vs the Baseline at Equal Parameters (08:37)
29 - Section 3 Wrap What Recursion in Depth Buys You, and What It Costs (03:02)
4 - Section 4 Recursive Language Models (RLM)
30 - Section 4 Intro When the Prompt Is Too Long to Read (02:54)
31 - Context Rot What NIAH, RULER, Lost in the Middle and OOLONG Show (06:41)
32 - The RLM Idea The Prompt as a Variable in a REPL, Sub-Calls as a Function (06:29)
33 - RLM vs RAG vs ReAct vs CodeAct Where the Long Input Lives, Who Decides Access (05:57)
34 - What the RLM Paper Reports Benchmarks, Depth Ablations, Costs and Versions (06:39)
35 - [LAB] A Minimal RLM Loop by Hand Root Model, Python REPL, llm_query and FINAL (07:22)
36 - [LAB] Serving a Local Model for the Loop Ollama's OpenAI-Compatible Endpoint (04:28)
37 - [LAB] Running the Hand-Built RLM on a Needle Task and Reading the Full Trace (04:54)
38 - [LAB] The Official rlms Library on a Local Model, and Why a Small Root Fails (08:17)
39 - [LAB] Aggregating a Long Document Chunked Sub-Calls vs One Long Prompt (08:56)
40 - Stated Limits Cost Blow-Ups, Sequential Calls, Brittle FINAL, REPL Sandboxing (05:39)
41 - Section 4 Wrap When an RLM Is the Right Tool (02:45)
5 - Section 5 Efficient Attention and Linear Transformers
42 - Section 5 Intro Making Attention Cheaper Without Losing the Plot (02:58)
43 - Sparse and Local Patterns Longformer, BigBird and Sliding-Window Attention (06:03)
44 - Low-Rank and Kernel Tricks Linformer, Performer and Transformers Are RNNs (06:58)
45 - FlashAttention 1 to 4 Exact Attention, IO-Aware — and Why It Is a GPU Story (05:42)
46 - [LAB] Sliding-Window Attention on the Baseline Memory and Quality vs Window (08:30)
47 - [LAB] A Linear-Attention Layer From Scratch and Its Recurrent Form (07:38)
48 - [LAB] Timing Attention on CPU Length Sweeps, Math vs Flash Backend in PyTorch (06:49)
49 - MQA, GQA and Multi-Head Latent Attention Shrinking the Cache, Not the Compute (05:37)
50 - Section 5 Wrap Which Attention Trick for Which Bottleneck (02:45)
6 - Section 6 State Space Models — S4 and Mamba
51 - Section 6 Intro A Recurrent Model That Trains Like a Convolution (02:49)
52 - From Continuous Systems to Discrete Recurrence The SSM Equations Made Concrete (06:27)
53 - S4, H3 and Hyena Long Convolutions and the Long Range Arena (06:17)
54 - Mamba's Selective Scan Input-Dependent Parameters, Hardware-Aware Algorithm (06:42)
55 - [LAB] A Selective SSM Layer From Scratch Discretise, Scan, Gate (08:23)
56 - [LAB] Sequential vs Chunked Scan on CPU Speed, and Keeping the Numbers Stable (07:29)
57 - [LAB] Training a Tiny Mamba-Style Model and Adding Its Row to the Results Table (07:25)
58 - [LAB] Pretrained Mamba-130m vs Pythia-160m on CPU Same Data and Tokenizer (07:51)
59 - Mamba-2 and SSD, Mamba-3 What Changed and What Needs a GPU (05:56)
60 - Section 6 Wrap Where SSMs Win, and Where Your CPU Disagrees (02:53)
7 - Section 7 Hybrids and Modern Linear Recurrences
61 - Section 7 Intro Why the 2025 Models Mix Layer Types (03:00)
62 - RWKV and RetNet RNN Inference, Parallel Training (06:03)
63 - Gated DeltaNet and Kimi Delta Attention The Delta Rule Returns (05:50)
64 - Hybrid Stacks in Production Jamba, Zamba, Falcon-H1, Nemotron-H, Qwen3-Next (06:52)
65 - [LAB] Associative Recall The Task That Separates Attention From Pure SSMs (07:27)
66 - [LAB] A Hybrid Model SSM Layers Plus One Attention Layer, Trained and Compared (07:38)
67 - Section 7 Wrap Choosing a Mix, Not a Religion (02:48)
8 - Section 8 Mixture of Experts
68 - Section 8 Intro More Parameters, Same Compute per Token (02:46)
69 - Sparsely-Gated MoE to Switch Transformer Top-k Routing, Capacity, Token Drops (06:35)
70 - Load Balancing Auxiliary Loss, Router z-Loss, Expert Choice, Aux-Loss-Free Bias (06:32)
71 - Fine-Grained and Shared Experts DeepSeekMoE, Mixtral, OLMoE and Qwen MoE (05:48)
72 - [LAB] An MoE Feed-Forward Layer From Scratch Router, Top-2 and Weighted Combine (07:38)
73 - [LAB] Training the MoE and Watching Expert Load With and Without Balance Loss (07:53)
74 - [LAB] Collapse on Purpose What an Unbalanced Router Looks Like in the Logs (05:36)
75 - [LAB] A Real MoE on CPU Granite 3.1 1B-A400M Router Choices Token by Token (07:29)
76 - MoE in Practice Memory, Serving, Expert Parallelism and When Dense Is Better (05:41)
77 - Section 8 Wrap Sparse Compute, Full Memory (02:43)
9 - Section 9 Memory-Augmented Networks and the KV Cache
78 - Section 9 Intro Where Does a Model Keep What It Has Read (02:46)
79 - External Memory Neural Turing Machine, Memory Networks and the DNC (06:33)
80 - Memory for Transformers Transformer-XL, Memorizing Transformers, RMT and Titans (06:36)
81 - [LAB] A Content-Addressed Memory Module Trained on Associative Recall (07:25)
82 - [LAB] The KV Cache Measured With and Without It, and Its Size Formula (07:35)
83 - [LAB] Cache Management in transformers Dynamic, Static, Quantized, Cropping (06:51)
84 - PagedAttention, Attention Sinks and KV Quantization The Serving View (05:30)
85 - [LAB] Sliding Window Plus Sinks Keeping Generation Stable Past the Window (06:44)
86 - Section 9 Wrap Memory Inside, Beside and Outside the Model (02:45)
10 - Section 10 RAG Patterns That Work in Production
87 - Section 10 Intro Retrieval as External Memory You Can Update (02:45)
88 - RAG From Lewis et al. (2020) to Today The Moving Parts and Their Failure Points (05:34)
89 - [LAB] Chunking Strategies on One Corpus Fixed, Recursive, Sentence-Aware (07:30)
90 - [LAB] Embeddings on CPU Choosing a Small Model, Encoding, Checking Its Licence (07:03)
91 - [LAB] Vector Indexes With FAISS Flat vs HNSW vs IVF, Recall and Latency (07:18)
92 - [LAB] BM25 Plus Dense Hybrid Search With Reciprocal Rank Fusion (06:53)
93 - [LAB] Re-Ranking With a Cross-Encoder What It Adds, What It Costs (07:01)
94 - [LAB] Retrieval Evaluation A Labelled Set Scored With Recall@k, MRR and nDCG (07:45)
95 - [LAB] End-to-End Local RAG Answers With Citations and a Faithfulness Check (08:39)
96 - Contextual Retrieval, GraphRAG, Self-RAG and CRAG When the Extra Machinery Pays (05:37)
97 - [LAB] RAG vs RLM vs Long Prompt on the Same Question Set (07:36)
98 - Section 10 Wrap A RAG Pipeline You Can Defend With Numbers (02:45)
11 - Section 11 Benchmarking and Evaluating New Architectures
99 - Section 11 Intro Numbers You Would Bet Your Architecture On (02:49)
100 - Fair Comparison Rules Same Data, Tokens, Parameters, Seeds — and Bits per Byte (06:41)
101 - Perplexity Pitfalls Tokenizers, Context Windows and Strided Evaluation (05:25)
102 - [LAB] Throughput, Time-to-First-Token and Memory on CPU Warm-Up and Spread (07:33)
103 - [LAB] Long-Context Tests Needle Sweeps and Position Effects on a Small Model (07:29)
104 - [LAB] Synthetic Probes Selective Copying, Induction Heads, Multi-Query Recall (07:10)
105 - [LAB] lm-evaluation-harness on CPU One Small Task, Two Architectures (07:12)
106 - Scaling Laws and What a Toy Experiment Can and Cannot Tell You (05:41)
107 - Section 11 Wrap The Results Table, Complete and Honest (02:43)
12 - Section 12 Decisions, Trade-Offs and the Capstone
108 - Section 12 Intro From Experiments to an Architecture Decision (02:46)
109 - The Trade-Off Axes Training Cost, Latency, Memory, Context Length, Maturity (06:26)
110 - Ecosystem Reality Check What vLLM, llama.cpp and transformers Support Today (05:33)
111 - [LAB] Capstone Part 1 One Task, Three Architectures, One Harness (08:01)
112 - [LAB] Capstone Part 2 Running the Comparison and Reading the Spread (08:16)
113 - [LAB] Capstone Part 3 Writing the Architecture Selection Report From Results (06:38)
114 - Where the Field Is Moving Hybrids, Recursive Inference and Test-Time Memory (05:28)
115 - Section 12 Wrap What You Can Now Build, Measure and Decide (03:43)
[/spoiler]
Файлы примеров: не предусмотрены
Формат видео: MP4
Видео: AVC, 1280x720, 16:9, 30fps, 527kbps
Аудио: AAC, 44.1Hz, 128kbps, stereo
[spoiler="Скриншоты"]
[/spoiler]
All Comments