https://therahulbhati.github.io/tags/machine-learning/ Recent content in Machine Learning on Rahul Bhati Hugo -- 0.151.0 en-us Tue, 30 Jun 2026 10:00:00 +0530 https://therahulbhati.github.io/posts/four-nn-ideas-tested/ Tue, 30 Jun 2026 10:00:00 +0530 https://therahulbhati.github.io/posts/four-nn-ideas-tested/ <p>Most neural-network explanations start with math. That&rsquo;s honest. But the ideas stick when you&rsquo;ve actually broken something. Each section below is a live demo: click Run, watch it train, change a control, run it again.</p> <ol> <li><a href="#1-activations-exist-for-a-reason">Activations exist for a reason</a></li> <li><a href="#2-depth-without-nonlinearity-is-a-lie">Depth without nonlinearity is a lie</a></li> <li><a href="#3-embeddings-learn-similarity-from-next-token-alone">Embeddings learn similarity from next-token alone</a></li> <li><a href="#4-memorization-vs-generalization">Memorization vs generalization</a></li> </ol> <script> window.addEventListener('message', function(e) { if (!e.data || e.data.type !== 'resize') return; var frames = document.querySelectorAll('iframe[data-autoresize]'); for (var i = 0; i < frames.length; i++) { if (frames[i].contentWindow === e.source) { frames[i].style.height = (e.data.h) + 'px'; break; } } }); </script> <hr> <h2 id="1-activations-exist-for-a-reason">1. Activations exist for a reason</h2> <p>A linear model and a ReLU model, side by side, trying to separate a red ring from a blue one. The linear model can&rsquo;t draw a curved boundary no matter how long it trains. Without a nonlinear activation, a stack of layers collapses to one matrix multiply. The curved shape you need is impossible to express.</p>