# Wiki: Speedrunning the LLM from scratch Curriculum in 72 hours (2024-03-10) **URL:** \ **Category:** projects\ **Tags:** gpt, llm, generative-ai\ **Created:** [March 10, 2024, 5:00am UTC](https://pluralist.net/t/wiki-speedrunning-the-llm-from-scratch-curriculum-in-72-hours-2024-03-10/104 "2024-03-10T05:00:00Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![delyan](https://pluralist.net/user_avatar/pluralist.net/delyan/32/245_2.png) [@delyan](https://pluralist.net/u/delyan)\ **Post date:** [March 10, 2024, 5:00am UTC](https://pluralist.net/t/wiki-speedrunning-the-llm-from-scratch-curriculum-in-72-hours-2024-03-10/104/1 "2024-03-10T05:00:00Z")
Note: This wiki will overtime grow to become an efficient way to learn Generative AI (GPT) in a contiguous period of 72 hours (optionally including sleep). This is a collection of notes and links at this point. The goal is for this to become an [Alexey Guzey](https://guzey.com/) type of a [intense](https://guzey.com/2022-lessons/) [minute by minute curriculum](https://www.youtube.com/watch?v=CDzb20mBL5Y). # Goal Hacker Retreats are fun. We’ve done music themed ones, ham radio ones etc. The goal of this web page is to compose a [**curriculum**](https://en.wikipedia.org/wiki/Curriculum), or simply an intense schedule, which would guide a group of hackers from point A to point Z in developing a ChatGPT-like system from scratch. * * * # What - Books - [Deep Learning and the Game of Go - Max Pumperla and Kevin Ferguson](https://www.manning.com/books/deep-learning-and-the-game-of-go) - [Build a Large Language Model (From Scratch) - Sebastian Raschka](https://www.manning.com/books/build-a-large-language-model-from-scratch) - the repo: [GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by step](https://github.com/rasbt/LLMs-from-scratch) - [Grokking Deep Learning - Andrew W. Trask](https://www.manning.com/books/grokking-deep-learning) - Videos - [Let’s build GPT: from scratch, in code, spelled out. YouTube · Andrej Karpathy](https://www.youtube.com/watch?v=kCc8FmEb1nY) - [The spelled-out intro to neural networks and backpropagation … YouTube · Andrej Karpathy](https://www.youtube.com/watch?v=VMj-3S1tku0) - [Let’s build the GPT Tokenizer YouTube · Andrej Karpathy](https://www.youtube.com/watch?v=zduSFxRajkE) - [https://youtu.be/zjkBMFhNj\_g?si=w2NAb8AJePbFYsMM](https://youtu.be/zjkBMFhNj_g?si=w2NAb8AJePbFYsMM) - HN - [Show HN: Train a GPT to write like Shakespeare-from scratch, in one Python file | Hacker News](https://news.ycombinator.com/item?id=38902719) - [Show HN: Train a GPT to write like Shakespeare-from scratch, in one Python file | Hacker News](https://news.ycombinator.com/item?id=38902719) - [Show HN: GPT-2 from scratch in PyTorch – Video tutorial | Hacker News](https://news.ycombinator.com/item?id=30177968) - [WyGPT: C++ GPT Language Model from Scratch | Hacker News](https://news.ycombinator.com/item?id=27551849) - [https://news.ycombinator.com/item?id=34726115](https://news.ycombinator.com/item?id=34726115) - articles - [How to Build an LLM from Scratch | Towards Data Science](https://towardsdatascience.com/how-to-build-an-llm-from-scratch-8c477768f1f9) - [https://youtu.be/ZLbVdvOoTKM?si=GeB\_emuZoAYdVCO3](https://youtu.be/ZLbVdvOoTKM?si=GeB_emuZoAYdVCO3) - [https://youtu.be/UU1WVnMk4E8?si=lpT0ajBZC2CbjD2p](https://youtu.be/UU1WVnMk4E8?si=lpT0ajBZC2CbjD2p) - [Fine-Tuning Overview — The GenAI Guidebook](https://ravinkumar.com/GenAiGuidebook/language_models/finetuning.html) - [Neural Networks: Zero To Hero](https://karpathy.ai/zero-to-hero.html) - [https://nandgame.com/](https://nandgame.com/) - [https://www.nand2tetris.org/](https://www.nand2tetris.org/) - [GPT in 60 Lines of NumPy | Jay Mody](https://jaykmody.com/blog/gpt-from-scratch/) - [Speeding up the GPT - KV cache | Becoming The Unbeatable against AGI](https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-kvcache/) - [https://course.fast.ai/](https://course.fast.ai/) - [Box2D - Gymnasium Documentation](https://gymnasium.farama.org/environments/box2d/) - [learn-pytorch/SAC.ipynb at main · DevJac/learn-pytorch · GitHub](https://github.com/DevJac/learn-pytorch/blob/main/SAC.ipynb) - [Grokking Deep Reinforcement Learning - Miguel Morales](https://www.manning.com/books/grokking-deep-reinforcement-learning) - [Welcome to Spinning Up in Deep RL! — Spinning Up documentation](https://spinningup.openai.com/en/latest/) - [TDDC17 \> Lab 5: Reinforcement Learning](https://www.ida.liu.se/~TDDC17/info/labs/rl.en.shtml) - [Sutton & Barto Book: Reinforcement Learning: An Introduction](http://incompleteideas.net/book/the-book-2nd.html) - [https://youtu.be/UU1WVnMk4E8?si=38fRZhPQDvaoEr\_N](https://youtu.be/UU1WVnMk4E8?si=38fRZhPQDvaoEr_N) - [Mosaic LLMs: GPT-3 quality for](https://www.databricks.com/blog/gpt-3-quality-for-500k) - [How to build a large language model from scratch | Online Courses, Learning Paths, and Certifications - Pluralsight](https://www.pluralsight.com/resources/blog/data/how-build-large-language-model) - [Training a causal language model from scratch - Hugging Face LLM Course](https://huggingface.co/learn/nlp-course/en/chapter7/6) * * * # How (time management) - apply Alexey’s lessons: [My 2022 self (I don't know them) was very wrong about meditation, huge monitors, and... sleep. - Alexey Guzey](https://guzey.com/2022-lessons/) - use Gerganov’s local LLM to avoid distractions: [GitHub - ggml-org/llama.cpp: LLM inference in C/C++](https://github.com/ggerganov/llama.cpp) - Ultralearning - Scott Young’s [MIT challenge](https://www.scotthyoung.com/blog/myprojects/mit-challenge-2/) - [Memorizing a programming language using spaced repetition software | Derek Sivers](https://sive.rs/srs) - [https://tracingwoodgrains.medium.com/speedrunning-college-my-plan-to-get-a-bachelors-degree-within-a-year-abe741c4c8bb](https://tracingwoodgrains.medium.com/speedrunning-college-my-plan-to-get-a-bachelors-degree-within-a-year-abe741c4c8bb) - [How To Remember Anything Forever-ish](https://ncase.me/remember/) - [Effective learning: Twenty rules of formulating knowledge - SuperMemo](https://www.supermemo.com/en/blog/twenty-rules-of-formulating-knowledge) More Tools: - [My 4-monitor computer setup (16-inches + 49-inches + 34-inches + 24-inches) - Alexey Guzey](https://guzey.com/my-computer-setup/) - Worker League app by [https://twitter.com/mishayagudin](https://twitter.com/mishayagudin) * * * # Who (can help) - [https://guzey.com/](https://guzey.com/) - [https://karpathy.ai/](https://karpathy.ai/) * * * # Is this even a good idea - not sure - [https://twitter.com/ID\_AA\_Carmack/status/1200111228460376064](https://twitter.com/ID_AA_Carmack/status/1200111228460376064) - maybe yes - [https://twitter.com/hardmaru/status/971297318665322498](https://twitter.com/hardmaru/status/971297318665322498) - definitely yes - [https://twitter.com/ID\_AA\_Carmack/status/970512819107090432](https://twitter.com/ID_AA_Carmack/status/970512819107090432) - the post - [https://www.facebook.com/permalink.php?story\_fbid=2110408722526967&id=100006735798590](https://www.facebook.com/permalink.php?story_fbid=2110408722526967&id=100006735798590) - _No!_ - Cal Newport may disagree with Speedrunning this (maybe not since this is more like cramming vs creating something new) - [My new book: Slow Productivity - Cal Newport](https://calnewport.com/my-new-book-slow-productivity/) * * * # Prereqs - Focus - Food - Machines ready to go - Slow Internet? (Distraction-free internet) - Pen & Paper - Felt-tip pen like Bill Gates - [Memex: Today's Interesting Links (Bill Gates writing with felt-tip pens; competing with ed) (2024-01-15)](https://pluralist.net/t/memex-todays-interesting-links-bill-gates-writing-with-felt-tip-pens-competing-with-ed-2024-01-15/43) - source: [Dawn of a revolution — Harvard Gazette](https://news.harvard.edu/gazette/story/2013/09/dawn-of-a-revolution/) ---
**Author:** ![delyan](https://pluralist.net/user_avatar/pluralist.net/delyan/32/245_2.png) [@delyan](https://pluralist.net/u/delyan)\ **Post date:** [March 10, 2024, 1:36pm UTC](https://pluralist.net/t/wiki-speedrunning-the-llm-from-scratch-curriculum-in-72-hours-2024-03-10/104/2 "2024-03-10T13:36:11Z")
Here is a reply on a wiki post - email sent?? ---
**Author:** ![delyan](https://pluralist.net/user_avatar/pluralist.net/delyan/32/245_2.png) [@delyan](https://pluralist.net/u/delyan)\ **Post date:** [March 10, 2024, 1:37pm UTC](https://pluralist.net/t/wiki-speedrunning-the-llm-from-scratch-curriculum-in-72-hours-2024-03-10/104/3 "2024-03-10T13:37:01Z")
And here is another one!