As engineers, we often interact with Large Language Models (LLMs) via a simple POST request to an API endpoint. We send a string, and a few hundred milliseconds later, we get a response. But for those building production-grade AI applications, the “black box” approach isn’t enough.

To optimize for latency (TTFT), throughput (TPS), and cost, we have to peel back the layers of the Transformer architecture. We need to understand why the decoding phase is a memory hog, how tokenization can break your logic, and why the KV cache is the most important piece of memory you’re currently managing.

The Architecture Debate: Why “Decoder-Only” Won

When the Transformer was first introduced in the Attention is All You Need paper, it was a symmetrical beast: an Encoder to understand the input and a Decoder to generate the output.

However, the industry has largely converged on the Decoder-only architecture (think GPT-4, Llama 3, and Mistral). Why?

• Encoders (e.g., BERT, RoBERTa): These use “bi-directional” attention. They look at every word in a sentence simultaneously to create a dense representation (embedding). They are king for classification and NER but lack the fluid generative capabilities of their counterparts.

• Encoder-Decoders (e.g., T5, BART): These were the standard for translation. The encoder processes the source language, and the decoder generates the target.

• Decoder-Only: These use “causal” or “masked” attention. They only look at previous tokens in a sequence. Through massive scaling, we’ve discovered that these models don’t actually need a separate encoder. They can derive the context of a prompt and generate a response within the same mathematical framework. This simplifies the inference pipeline, making it easier to scale on modern GPU clusters.

Tokenization: The “Lossy” Bridge to Math

LLMs do not process strings; they process tensors of floating-point numbers. The first engineering hurdle is Tokenization.

Subword Tokenization vs. Character Level

A naive approach would be to assign an integer to every word. But dictionaries are infinite, and typos are common. If your model encounters a word it hasn’t seen (an “Out of Vocabulary” or OOV error), the system fails.

Modern LLMs use Subword Tokenization (like Byte-Pair Encoding or WordPiece). This breaks text into the smallest meaningful chunks.

Example: The word regularization might be tokenized as [regular, ization].

Example: The suffix er is a common token. This allows the model to handle tall, taller and tallest by reusing the e and est tokens, keeping the vocabulary size manageable (usually between 32k and 128k tokens).

The Engineer’s Token Tax

As an engineer, you must remember that 1 token ~= 0.75 words (or roughly 4 characters in English). However, tokenizers are model-specific. If you use a Llama-3 tokenizer to estimate the cost for a GPT-4o request, your math will be wrong. This is particularly critical when building “RAG” (Retrieval-Augmented Generation) systems where you are stuffing thousands of tokens of context into a prompt, every token counts toward your rate limits and your bill.

Embeddings and the High-Dimensional Galaxy

Once we have our list of token IDs, we move to the Embedding Layer.

Imagine a map. On a 2D map, “Paris” and “London” are close together. In an LLM, we use a “map” with thousands of dimensions. Each token ID is mapped to a vector in this high-dimensional space.

Semantic Proximity is the goal here. The vector for Queen should be closer to King than it is to Toaster. During the training phase, the model learns these relationships. In inference, these “static” embeddings are fed into the Transformer blocks where Self-Attention transforms them into Contextual Embeddings.

This is how the model knows that the word “Java” in a blog about programming is different from “Java” in a blog about coffee. The surrounding vectors pull the “Java” vector toward the “Programming” cluster in that high-dimensional space.

Positional Embeddings: Teaching the Model to Read

A massive quirk of the Transformer architecture is that it is “permutation invariant.” Because the attention mechanism calculates the relationship between all tokens in a set simultaneously, the model has no innate way of knowing if “Dog bites Man” or “Man bites Dog.”

To solve this, we use Positional Embeddings. This involves injecting information about the specific position of each token in the sequence into the vector.

• Sinusoidal Embeddings: The original method used sine and cosine functions of different frequencies.

• RoPE (Rotary Positional Embeddings): Used by modern models like Llama. Instead of adding a vector, it rotates the embedding in the high-dimensional space. This allows for better “extrapolation,” meaning the model can sometimes handle sequences longer than those it was trained on.

The Two Phases of Inference: Prefill and Decode

This is where the engineering complexity peaks. LLM inference is not a single, monolithic task; it is a tale of two phases with completely different hardware requirements.

Phase 1: The Prefill (Compute-Bound)

When you send a prompt, the model first processes the entire input. This is the Prefill Phase.

• What happens: The model takes all “N” tokens of your prompt and calculates their hidden states in parallel.

• The Bottleneck: This is compute-bound. It depends on how many TeraFLOPS your GPU can push.

• The Metric: This phase determines your Time to First Token (TTFT). If you have a massive prompt, the prefill takes longer, but modern GPUs are highly optimized for this parallel matrix multiplication.

Phase 2: The Decode (Memory-Bound)

Once the first token is generated, the model enters the Decoding Phase. Because LLMs are autoregressive, they generate one token at a time. To generate token #51, the model must look at tokens #1 through #50.

• What happens: The model does a single forward pass for one single token.

• The Bottleneck: This is memory-bound. You aren’t limited by how fast the GPU can calculate; you are limited by how fast the GPU can move the model weights and the “KV Cache” from VRAM to the processing cores.

• The Metric: This determines your Tokens Per Second (TPS).

The KV Cache: Solving the Quadratic Problem

The “Self-Attention” mechanism in Transformers has a major flaw: its computational cost grows quadratically O(n^2) with the sequence length. If you don’t optimize this, generating the 2,000th token would require re-calculating the attention for all 1,999 previous tokens.

To solve this, we use the KV (Key-Value) Cache.

We store the previously calculated “Key” and “Value” vectors in the GPU’s VRAM. When it’s time to generate the next token, the model simply fetches these from the cache instead of recomputing them.

PagedAttention and vLLM

Storing this cache is expensive. For a large model with a long context, the KV cache can consume tens of gigabytes of VRAM. Historically, this required contiguous memory, which led to massive fragmentation (the “OOM” or Out of Memory error we all dread).

The breakthrough came with PagedAttention (implemented in frameworks like vLLM). It borrows a concept from Operating Systems: virtual memory paging. It breaks the KV cache into small, non-contiguous blocks. This allows for:

1. Near-zero memory waste.

2. Continuous Batching: Processing multiple requests at once even if they finish at different times.

3. Prefix Sharing: If ten users are asking questions about the same long document, the model only stores the KV cache for that document once, saving massive amounts of VRAM.

The “Hidden” Logic: Sampling and Temperature

After the model does its math, it doesn’t actually output a word. It outputs a Logits vector, a list of probabilities for every single token in its vocabulary.

If the vocabulary is 50,000 words, the model says: “There is a 40% chance the next word is ‘Apple’, a 20% chance it’s ‘Banana’, and a 0.001% chance it’s ‘Nuclear’.”

Engineers guide this selection using Sampling Parameters:

• Temperature: High temperature flattens the probability distribution (making the “Nuclear” option more likely), leading to more creative or “hallucinatory” text. Low temperature (approaching 0) makes the model “greedy,” always picking the #1 choice.

• Top-P (Nucleus Sampling): The model only considers the smallest set of tokens whose cumulative probability exceeds ‘P’.

• Top-K: The model only considers the top 'K’ most likely tokens.

The Multimodal Future

The text you provided mentions that modern LLMs are becoming multimodal. For an AI engineer, this means the inference pipeline is getting more complex. We aren’t just tokenizing text anymore; we are using Vision Encoders (like CLIP) to turn images into patches. These patches are then mapped into the same embedding space as our text tokens.

Whether the input is a JPEG or a string of Python code, it eventually becomes a sequence of vectors that the Transformer processes using the same attention mechanism. This is why “Large Language Models” is becoming a bit of a misnomer, they are increasingly “Large Multimodal Action Models.”

Final Thoughts for the engineers

Understanding the inference pipeline is the difference between a prototype that works on your local machine and a production system that scales to thousands of users.

Key Takeaways:

1. Optimize your Prefill: Use shorter prompts or KV-cache sharing to reduce TTFT.

2. Watch your VRAM: The KV cache is your biggest bottleneck for high-concurrency systems.

3. Choose the right runtime: Don’t just run raw PyTorch in production. Use specialized inference engines like vLLM, TGI (Text Generation Inference), or TensorRT-LLM.

What’s your current stack for LLM inference? Are you hitting memory limits or compute limits? Let’s troubleshoot in the comments below.

References

PagedAttention & vLLM: Kwon, W., et al. (2023). “Efficient Memory Management for Large Language Model Serving with PagedAttention.” The foundational paper for the vLLM project that revolutionized KV cache management.

Disaggregated Inference: Microsoft Research (2024). “Splitwise: Efficient Generative LLM Inference Using Phase Splitting.” A key paper discussing the physical separation of Prefill and Decode phases.

Jay Alammar’s “The Illustrated Word2vec”: jalammar.github.io. One of the best visual explanations of how high-dimensional word embeddings work.

The Illustrated Transformer: jalammar.github.io. A visual breakdown of the encoder-decoder mechanics that paved the way for modern LLMs.

This article is aimed at anyone who is trying to install Kubeflow on AWS. If you have tried installing Kubeflow, then you might have come across several errors, and to finally get Kubeflow working on AWS cloud, it took me a couple of attempts.

Image by @clearmlapp

This article is being written mainly as a way to put together the tips and tricks that I learned during my first setups of Kubeflow so that I may go back and redo it if needed in the future.

Prerequisites

We will install Kubeflow v1.2 on AWS, and to configure it, the following services will be used.

• EKS

We are going to deploy Kubeflow on AWS EKS, which is a managed Kubernetes service. We are using an EKS cluster v1.19 with 6 nodes. Every node is an EC2 instance of type t2.large.

• IAM service roles

Create service role eks-cluster-role with policy AmazonEKSClusterPolicy and service role eks-node-role with policy AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly and AmazonEKS_CNI_Policy.

Attach these roles to the EKS cluster, which will, in turn, assume the role to access other AWS services.

• AWS Cloud9

Cloud9 is a cloud-native IDE which runs on top of an EC2 instance and lets you write, run, and debug your code with just a browser. We will use Cloud9 to connect to the EKS cluster and to run commands for Kubeflow to install on EKS.

Deploy kubeflow

Step 1: Install eksctl

eksctl is a simple CLI tool for creating and managing clusters on EKS - Amazon's managed Kubernetes service for EC2.

Let’s install eksctl with below commands.

Step 2: Configure aws-iam-authenticator

aws-iam-authenticator is a tool to use AWS IAM credentials to authenticate to a Kubernetes cluster.

If you are building a Kubernetes installer on AWS, AWS IAM Authenticator for Kubernetes can simplify your bootstrap process. You won't need to somehow smuggle your initial admin credentials securely out of your newly installed cluster. Instead, you can create a dedicated KubernetesAdmin role at cluster provisioning time and set up Authenticator to allow cluster administrator logins.

To avoid having to manage a separate credential for Kubernetes access, we will use aws-iam-authenticator.

Step 3: Install kfctl

kfctl is the control plane for deploying and managing Kubeflow.

Now, let’s update the role we have created in the kfctl configuration.

Step 4: Build Kubeflow

Now, we will build Kubeflow with the following command.

You can monitor all the resources created by KubeFlow with the following command.

Access the Kubeflow central dashboard

Run the following command to get your Kubeflow service’s endpoint hostname and copy the link in the browser.

This deployment may take 3-5 minutes to become ready. Once complete, you can verify the installation by opening the ingress address in your preferred browser.

To expose Kubeflow with a LoadBalancer Service, just change the type of the istio-ingressgateway Service to LoadBalancer.

Endnotes

Congratulations on your successful deployment of Kubeflow on AWS. As the Kubeflow team releases new versions rapidly, the steps written here might change. There are other ways to deploy Kubeflow on AWS and other platforms; you can refer to the Kubeflow docs for more details.

Thank you for reading..! 🙏

Following continuous software engineering practices, there has been an increasing interest in rapid deployment of machine learning (ML) features, called MLOps.

In this use case, we will build an ML lifecycle using tools like DVC, MLflow, Flask, Jenkins and more. Final code is available on Github.

Note: This type of use can be vast and takes a lot of time and resources, we will limit our features, but it will teach you how to build an ML lifecycle right from scratch.

Architecture  

In this use case, we are using DVC for data versioning and data pipeline, MLflow for experiment tracking, Flask for model serving and Jenkins for CI/CD pipeline. 

We are going to deploy our model on a managed Kubernetes service by Azure, but in theory, you could also use AWS, GCP or any Kubernetes service provider. 

All these tools are used here to demonstrate the end-to-end ML lifecycle. You could also try other alternatives to build an ML pipeline based on your requirements.

Project Structure

To start with the project, we will use a template.py, which will create an initial project structure for us. Create the template.py and execute the code below to create the project structure.

This will create a project structure as follows:

Data versioning & pipeline 

In this demonstration, we are going to load data from a local repository, but in production, data can be loaded from any storage like S3, Azure Blob storage, etc. We can also track versions of the dataset using DVC, in an S3 bucket.

Let’s add our dataset to DVC for tracking with the following block of code.

Now, we will create a DVC pipeline which will trigger activities like data collection, loading, train-test split, model training and evaluation. After this, it will log the model with the best metrics to production. 

The whole point of creating this dvc.yaml file is the ability to easily reproduce a pipeline, and it can be executed with the command ‘dvc repro’

The params.yaml used to configure metrics and hyperparameters, which will be passed as input to the dvc pipeline. 

Image by @overflow_meme

Let’s build our model 

First, we will get our data from storage and store it in the data/raw directory for further processing using the script ‘get_data.py’ and ‘load_data.py’

Let's create load_data.py to load data from the data/raw directory into a dataframe to prepare it for training.

When the data is loaded, the split_data.py script will be used to split the data for model training and evaluation.

Let’s build our model

First, we will get our data from storage and store it in data/raw directory for further processing using the script ‘get_data.py’ and ‘load_data.py’

Let's create load_data.py to load data from data/raw directory into a dataframe to prepare it for training.

When the data is loaded, the split_data.py script will be used to split the data for model training and evaluation.

Experiment Tracking 

The training code will use MLflow to track experiments and for model versioning. The model artefact will be stored in the artefact directory, but in production, the S3 bucket can be used to store model artefacts, and metadata for experiments can be stored in any SQL database. For our case, we are storing it in a SQLite database.

The model with the best metrics will be logged as production, and it will be moved to production_service/model directory for serving.

Model Serving 

For model serving, we will build a Flask application which will provide an API endpoint for predictions. This application will run on AKS, and we will expose it to the internet using the Load Balancer service. 

In our flask application, it will receive inputs in JSON format and will send back the response in JSON with output 0 or 1.

Use app.py to create the Flask endpoint, which will launch our Flask application.

Package model into a Docker container 

We will build the package of our model using Docker and push it to the container registry on Docker Hub. After this, we will use docker-compose.yml to run our Docker container.

Deploying our model on Kubernetes 

We are deploying our model on AKS. For this, we are using k8s-deployment.yml, which will create a deployment on K8S and a loadbalancer service to expose our application to the internet with port 5001.

CI/CD Pipeline 

Now that we have built our model, let's create a CI/CD pipeline using Jenkins, which will clone the code from the Git repository, create the Docker image, push the Docker image to Docker Hub container registry and deploy the required Kubernetes components on AKS.

Inferencing

Let's get the external IP of the loadbalancer from the cluster and make a REST request to the model endpoint, which will return the response of a model output 0 or 1.

Endnotes

In this demonstration, we have built an ML lifecycle right from creating a project structure to model deployment on AKS. This use case can be extended with features like continuous training, data pre-post processing, validation, model monitoring and much more.

Thank you for reading..! 🙏
If you liked this use case, make sure to star the GitHub repo 😉    

References

1] Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help