# Machine Learning: The Complete Architecture > What machine learning is, how each type works, how models are trained and deployed, and how developers use ML inside agents and bot systems. - Canonical: https://thearchitecturedeepdive.blog/articles/machine-learning-deepdeepest.html - Series: The Architecture DeepDive, No. 003 - Author: Barnabas Waweru - Published: 2026-04-01 - Tags: AI Systems - Cite as: Barnabas Waweru, "Machine Learning: The Complete Architecture", The Architecture DeepDive, https://thearchitecturedeepdive.blog/articles/machine-learning-deepdeepest.html --- [← Back to all articles](https://thearchitecturedeepdive.blog/) DEEPDEEPEST No. 003 · AI SYSTEMS What machine learning actually is, how every type works under the hood, how models are trained and deployed, and how developers use ML inside Claude Code, AI agents, and bot systems. Barnabas Waweru · April 2026 · ~30 min read 01 // FOUNDATIONS ## What Machine Learning Actually Is 02 // TAXONOMY ## The Four Major Types of Machine Learning 03 // ARCHITECTURE ## Neural Networks: How Machines Think 04 // TRAINING LIFECYCLE ## How a Model Actually Gets Trained 05 // PRODUCTION ## ML in Production: Serving, Monitoring, and Drift 06 // THE FULL MAP ## ML, AI, Deep Learning, and LLMs: How They Relate 07 // DEVELOPER TOOLS ## ML in Claude Code, AI Agents, and Bot Training 08 // THE STACK ## The Agentic ML Development Loop 09 // THE FRONTIER ## Bot Training: What It Actually Involves ## The Complete ML Developer Stack ## What Comes Next: Edge ML, Multimodal, and Agentic Futures Traditional software is a set of explicit instructions. A developer writes rules. The computer follows them. If the rule says "if input is greater than 10, return true," then the machine obeys that rule forever, unchanged, whether it is wrong or right. Machine learning inverts this completely. Instead of writing the rules, you show the machine examples. Thousands. Millions. The machine finds the pattern. It writes its own rules in the form of a mathematical function, a function so complex it would take a human years to express, but which the machine assembles in hours. At its core, ML is optimization. You have a model, a function with millions of internal parameters. You have data. You have a loss function that measures how wrong the model currently is. Training is the process of repeatedly adjusting those parameters in the direction that makes the model less wrong. Over thousands of iterations, the model converges on behavior that generalizes beyond the training examples. This is why machine learning looks like magic from the outside. The model was never told what a cat looks like. It was shown 10 million cats and non-cats, and it discovered what a cat looks like on its own, in a mathematical space no human can visualize. Not all ML is the same. The four main paradigms differ in what kind of data they consume, what kind of signal teaches the model, and what kind of problem they are designed to solve. Neural networks are the dominant architecture behind modern ML. The name is biological but the mechanics are mathematical. A neural network is a directed graph of computation: layers of nodes connected by weighted edges, where each node applies a small mathematical operation on its inputs and passes the result forward. The input layer receives raw data. Pixel values. Token embeddings. Sensor readings. Structured tabular features. Hidden layers transform that data through matrix multiplications and non-linear activation functions, building progressively more abstract representations at each layer. The output layer produces the final prediction: a class, a number, a probability distribution over tokens. The key innovation is backpropagation. During training, the network computes how wrong its output was using the loss function. It then traces that error backwards through every layer using the chain rule of calculus, computing the exact contribution of each weight to the error. Gradient descent then nudges each weight by a tiny amount in the direction that reduces the error. Repeat this millions of times and the network converges on something that works. Training a model is a software engineering and infrastructure problem as much as it is a mathematical one. The lifecycle moves through distinct phases, each with its own tooling, failure modes, and architecture decisions. Getting a model to work on a validation set is the easy part. Deploying it in a system that serves real users under real load, with real data drift, is the engineering challenge that consumes most of an ML team's time. Model serving exposes the model as an API endpoint. ONNX Runtime, TorchServe, TensorFlow Serving, and Triton Inference Server are common serving frameworks. Each handles batching, hardware acceleration, and concurrency. Quantization reduces model weights from 32-bit floats to 8-bit or 4-bit integers, cutting memory usage and inference latency at the cost of minor accuracy degradation. Hardware-aware optimization using TensorRT or torch.compile compiles models to low-level GPU kernels tuned for the specific accelerator. Data drift is the silent killer of production ML. The statistical distribution of incoming data changes over time. A fraud detection model trained on 2024 transaction patterns starts degrading in 2026 as fraud tactics evolve. Monitoring systems track input feature distributions, output confidence distributions, and downstream business metrics. When drift exceeds a threshold, retraining triggers automatically or via alert. These terms are used interchangeably in the press and imprecisely in most technical conversations. The actual relationship is a set of nested circles, each a subset of the one around it. Artificial intelligence is the broadest category: any computational system that performs tasks we associate with intelligent behavior. Search algorithms, rule-based expert systems, and game trees are all AI but involve no learning. Machine learning is the subset of AI where systems improve through data exposure rather than explicit programming. Deep learning is the subset of ML that uses neural networks with many layers. Large language models are deep learning systems pre-trained on text corpora at internet scale using the transformer architecture. Generative AI sits within deep learning but crosses paradigms. Diffusion models, GANs, VAEs, and autoregressive transformers all generate new data that resembles training data. They are applied to text, images, audio, video, code, and protein structures. The generative AI explosion of 2023 through 2026 is not a new category. It is the maturation of ML tooling to a point where training and inference became cheap enough for production use. The practical question for a developer in 2026 is not what ML is theoretically. It is how to use ML systems inside real products. Claude Code, LangChain, LlamaIndex, AutoGen, and CrewAI represent a new layer in the ML stack: the agentic orchestration layer. AI agents are ML systems that plan sequences of actions to complete goals. They combine an LLM backbone with tool access (web search, code execution, database queries, API calls) and a reasoning loop (ReAct, Chain of Thought, Tree of Thought) that decides what to do next based on observations. The pattern above is a simplified version of what production ML agent systems do. Claude receives a goal, calls tools to gather information or take actions, interprets the results, and plans the next action. The ML model is no longer just the product. It becomes the engineer deciding how to build the product. When people say they are training a bot they typically mean one of three distinct technical processes, each with different compute requirements, dataset requirements, and outcomes. The choice between RAG, fine-tuning, and prompting depends on the nature of the task. Prompting alone works for general reasoning. RAG is necessary when the model needs access to proprietary or rapidly changing information. Fine-tuning is necessary when you need to change the model's tone, persona, or response format at a level that prompting cannot achieve reliably. The ML ecosystem is fragmented by design. Different layers of the stack are owned by different tools, and the right combination depends on what you are building. ML in 2026 is not a settled field. Three structural shifts are reshaping what it means to build with machine intelligence. Every developer who builds with ML today is working at the orchestration layer. The frontier labs handle pre-training. The platform companies handle serving infrastructure. The developer's job is to design the loop: what information the model sees, what tools it can reach, how it plans, and how its outputs connect back to the systems that depend on them. This is not a diminishment of the developer's role. It is an expansion. The developer who understands ML architecture at the level this article describes can make principled decisions about every layer of that loop. Which model for which sub-task. When to fine-tune versus when to retrieve. When an agent needs memory and when statelessness is a feature. When to trust the model's judgment and when to impose hard constraints. Machine learning is not a specialty anymore. It is the material that software is increasingly built from. Understanding it end to end is no longer optional for anyone building systems that intend to matter. > You do not program the answer. You program the ability to find answers. > A neural network learns by repeatedly asking: in which direction should each weight move to make the output slightly less wrong? > 04 // TRAINING LIFECYCLE > All LLMs are deep learning. All deep learning is ML. All ML is AI. Not all AI learns. > 07 // DEVELOPER TOOLS > The model is no longer the product. The loop the model runs inside is the product. ## Training Lifecycle ### Data Collection and Curation Data is the fuel. Bad data is poison. The collection phase involves gathering raw data at scale: web crawls, proprietary databases, human-annotated datasets, synthetic generation. Curation removes duplicates, corrupted samples, mislabeled examples, and toxic or biased content. Data pipelines move this material through validation, transformation, and storage. In production ML, 80% of engineering time is spent on data, not models. ### Feature Engineering and Preprocessing Raw data rarely enters a model directly. Numerical features must be normalized or standardized so no single column dominates by scale. Categorical variables are encoded as integers or one-hot vectors. Text is tokenized. Images are resized and normalized to a fixed input shape. For deep learning, feature engineering is partially replaced by the model learning its own internal representations, but preprocessing is still mandatory. ### Model Selection and Architecture Design Choosing the right model class for the problem. A linear model may be sufficient for a structured tabular task with clear feature relationships. A transformer is necessary for long-range sequence dependencies. A CNN is efficient for spatial locality in images. Architecture design involves choosing depth (number of layers), width (neurons per layer), attention heads, embedding dimensions, and how components connect. These choices determine capacity, compute cost, and sample efficiency. ### Training Infrastructure Training large models requires distributed infrastructure. Data parallelism splits a batch across multiple GPUs and aggregates gradients. Model parallelism splits the model itself across devices when it is too large for a single GPU's memory. Tensor parallelism further splits individual matrix operations across accelerators. Gradient checkpointing trades compute for memory by recomputing activations during backprop rather than storing them. Mixed precision training uses FP16 or BF16 for compute and FP32 for parameter updates, roughly halving memory usage. ### Loss Functions and Optimization The loss function defines what "wrong" means. Cross-entropy loss is standard for classification. Mean squared error for regression. For language models, the loss is the average negative log-likelihood of the next token given all previous tokens. Optimizers like Adam maintain running estimates of the gradient mean and variance to adaptively scale learning rates per parameter. Learning rate schedules warm up from zero, then decay by cosine annealing or step functions. Gradient clipping prevents explosive updates. ### Evaluation, Validation, and Metrics A model that memorizes its training data is useless. The validation set measures generalization to unseen data. Metrics must match the business objective: accuracy fails on imbalanced datasets so you need precision, recall, F1, or AUC-ROC. For generation tasks, BLEU measures n-gram overlap with reference outputs. Perplexity measures how surprised the model is by held-out text. Human evaluation is often the only reliable metric for open-ended generation quality. ### Fine-Tuning and Alignment A base model trained on general data is rarely production-ready for a specific task. Supervised fine-tuning (SFT) continues training on a smaller, high-quality domain-specific dataset. RLHF collects human preference comparisons between model outputs, trains a reward model to predict human preference, and then uses PPO to optimize the base model against that reward. PEFT methods like LoRA and QLoRA inject small adapter matrices that are trained while the base model is frozen, dramatically reducing the compute and memory cost of fine-tuning. ## Key Components CNNs Convolutional Neural Networks process spatial data. Shared filter weights slide across an image, detecting edges, textures, and eventually objects. Foundational for computer vision. RNNs / LSTMs Recurrent networks maintain hidden state across sequential inputs. LSTMs solve vanishing gradients using gated memory cells. Essential for speech, time series, and early NLP. Transformers Self-attention mechanisms allow every token to directly attend to every other token. Scale-efficient and parallelizable. The architecture behind GPT, Claude, Gemini, and DALL-E. Diffusion Models Learn to reverse a noise-adding process. Starting from Gaussian noise, the model iteratively denoises toward a coherent output. Powers Stable Diffusion, DALL-E 3, and Sora. Graph Neural Networks Operate on graph-structured data where nodes and edges carry meaning. Used in drug discovery, molecular property prediction, fraud detection, and social network analysis. Mixture of Experts A routing mechanism activates only a subset of model parameters for each input. Enables massive model scale without proportional compute cost. Used in GPT-4, Gemini, and Mistral. MLflow Open source experiment tracking, model registry, and deployment tooling. Track hyperparameters, metrics, and artifacts across training runs. Weights and Biases Collaborative experiment tracking with interactive dashboards, sweep hyperparameter optimization, and model lineage tracking. Kubeflow Kubernetes-native ML pipeline orchestration. Runs training jobs, serves models, and manages pipelines at scale on any cloud. Evidently AI Open source ML monitoring. Detects data drift, target drift, and data quality issues across production model inputs and outputs. BentoML Framework-agnostic model serving. Package any model with dependencies and deploy as REST, gRPC, or streaming endpoints. Ray Distributed computing framework for ML. Scales training, hyperparameter tuning (Ray Tune), and reinforcement learning across clusters. Pre-Training from Scratch Training a model on a massive raw corpus with self-supervised objectives. Compute-intensive. Requires billions of tokens and thousands of GPU hours. Produces a foundation model. Almost no organization outside frontier labs does this today. Fine-Tuning Taking a pre-trained model and continuing training on a smaller, curated dataset for a specific domain or behavior. Supervised fine-tuning for instruction following. LoRA and QLoRA for parameter-efficient adaptation on consumer hardware. Most teams' primary training method. RAG (Retrieval Augmented Generation) Not training in the traditional sense. Embedding your documents into a vector store and retrieving relevant chunks at inference time to ground model responses. Cheaper than fine-tuning, more current, and easier to update. Often the right first step before deciding to fine-tune at all. Edge Inference Models are shrinking. Quantization, pruning, and knowledge distillation compress billion-parameter models into forms that run on mobile chips, embedded hardware, and browser WebGPU. On-device inference removes latency and privacy concerns. Apple's Core ML, MediaPipe, and GGUF formats drive this shift. Multimodal Architecture A single model processes text, images, audio, and video together. Cross-attention bridges modalities. The same model that reads a PDF also analyzes the chart within it and explains it in natural language. Claude 3 Opus, GPT-4o, and Gemini Ultra are all multimodal. The boundary between modalities is collapsing. Agentic Networks Networks of specialized agents, each with different capabilities and memory systems, collaborate on long-horizon tasks. One agent plans, another researches, another writes code, another evaluates output. The orchestration of these agents is becoming the primary engineering challenge in applied ML. ## ML Developer Stack DATA LAYER Pandas · Polars · DVC · Great Expectations · Airflow · dbt · Neon PostgreSQL · Pinecone · Weaviate Ingest, validate, transform, version, and store data. Vector databases like Pinecone and Weaviate serve embeddings at low latency for RAG systems. TRAINING LAYER PyTorch · JAX · HuggingFace Transformers · Axolotl · LitGPT · DeepSpeed · FSDP · Ray Train Define architectures, run forward and backward passes, manage distributed training across multiple GPUs and nodes. HuggingFace Transformers provides pre-trained weights for virtually any architecture. EXPERIMENT LAYER MLflow · Weights and Biases · Optuna · Neptune · Comet ML Track hyperparameters, log metrics, compare runs, and manage model checkpoints. Hyperparameter search (Optuna, W&B Sweeps) automates the process of finding optimal training configurations. SERVING LAYER vLLM · TGI · BentoML · Triton · ONNX Runtime · TorchServe · Modal · Replicate Expose model inference as an API with hardware acceleration, batching, and auto-scaling. vLLM uses PagedAttention to dramatically increase throughput for LLM serving. ORCHESTRATION LAYER LangChain · LlamaIndex · AutoGen · CrewAI · Claude SDK · Semantic Kernel Wire LLMs to tools, memory systems, and other agents. Define multi-step reasoning pipelines. Manage conversation context. This layer is where most application-level ML development happens in 2026. MONITORING LAYER Evidently AI · Arize · WhyLabs · Prometheus · Grafana · LangSmith Track model behavior in production. Detect data drift, output degradation, and latency regressions. LangSmith specifically traces LLM application chains for debugging and performance analysis. Interactive full article: [machine-learning-deepdeepest.html](https://thearchitecturedeepdive.blog/articles/machine-learning-deepdeepest.html)