The CPU Tier for Agents and Small Language Models

 

Copyright: Sanjay Basu

Which Agents and Which Small Models Actually Belong There

Treat this as A field guide to running agentic AI on OCI compute without touching a GPU.

I published a piece a couple of weeks ago on running inference on the X12 Standard Acceleron shape with Intel Xeon 6 and AMX.

Here is the link — https://blogs.oracle.com/cloud-infrastructure/run-ai-inference-without-gpus

The response split cleanly into two camps. Half the readers wanted the benchmark methodology. The other half asked a better question — ‘fine, the silicon works, but which agents and which models am I supposed to put on it?’

That is the harder question, and nobody has written it down properly. So here it is.

This is not an argument that CPUs replace GPUs. I ran one of the larger GPU fleets in the industry, and I have no interest in pretending otherwise. It is an argument that a large fraction of what enterprises currently route to an NVIDIA H200 or NVIDIA B200 or AMD MI355X has no business being there, and that OCI now has four distinct CPU platforms where that work belongs. Getting the placement right is worth more to most enterprises than another 10 percent on their GPU utilization.

Part 1

Let’s discuss what OCI actually gives you on the CPU side.

Start with the inventory, because most architects I talk to still think “OCI CPU” means the E-series VM they provisioned in 2023.

Copyright: Sanjay Basu

Older generations are still in the catalog and still useful. X9 Standard, E5, E4, A1 and A2 on Ampere Altra and AmpereOne. If you have committed capacity there, most of what follows still applies, just with the throughput numbers scaled down and no AMX.

Two things about this table matter more than the specs.

First, watch the OCPU accounting. On Ampere shapes one OCPU is two physical cores. On x86 one OCPU is one core with two threads. If you compare $0.0119 against $0.0190 without normalizing, you will reach the wrong conclusion about which platform is cheaper for your workload. Normalize to tokens per dollar on your own model, not to OCPU price.

Second, look at the memory column again. 1,152 GB. 1,536 GB. 2,304 GB. An H100 gives you 80 GB of HBM and an H200 gives you 141 GB. The CPU tier is not competing with GPUs on bandwidth. It is competing on capacity, and capacity is the axis nobody optimizes for because everyone is trained to think about FLOPS. Hold that thought, because it drives half the architecture decisions later in this piece.

Part 2

I am now going to discuss the Physics you cannot Argue with.

Before model selection, get the mental model right. Transformer inference has two phases with completely different bottlenecks, and CPU serving succeeds or fails on whether you respect the difference.

Prefill is compute bound. You are multiplying the full prompt against every weight matrix. This is where AMX on Xeon 6, VNNI on EPYC, and i8mm on AmpereOne earn their keep. It is also where CPUs lose most visibly to GPUs, because a GPU has an order of magnitude more matrix throughput. A 32K token prefill on a CPU is a bad idea. A 2K token prefill is fine.

Decode is memory bandwidth bound. Every generated token requires reading the entire active weight set out of DRAM. Single-stream decode speed has a hard ceiling.

tokens/sec (upper bound) = usable memory bandwidth / bytes of active weights

A 4B model at Q4 is roughly 2.4 GB of weights. If you measure 400 GB/s of usable read bandwidth on your node, your theoretical ceiling is about 165 tokens per second for one stream. You will not hit it. Budget for 40 to 60 percent of theory after cache effects and thread scheduling.

Run STREAM on the shape before you promise anyone a latency number. Vendor peak bandwidth and achieved bandwidth are different quantities, and on a 120-core single socket the gap depends heavily on your thread pinning.

The consequence that makes CPU serving viable.

Because decode reads the same weights regardless of batch size, concurrency is nearly free until you saturate compute. Serving one agent at 40 tokens per second and serving thirty agents at 35 tokens per second each cost roughly the same DRAM traffic. Agent fleets are naturally batched workloads. This is the single most important reason the CPU tier works for agentic systems and does not work for a consumer chatbot where every user wants their own low-latency stream.

KV cache is your real memory consumer, and you have plenty.

For a typical 4B model with grouped query attention, budget roughly 150 KB per token of context in FP16. At 8K context that is about 1.2 GB per concurrent stream. Thirty-two concurrent agent sessions costs you around 38 GB. On an 80 GB GPU that hurts. On a 1,152 GB node it is a rounding error.

Part 3

So, what are the Model Tiers?

I organize candidate models into five tiers by what job they do, not by parameter count alone. Parameter count tells you memory. The job tells you whether the model is good enough.

Tier 0 — Encoders, embedders, rerankers, classifiers

This is the most underrated tier and the easiest win. Nobody should be running a 70B model to decide whether a support ticket is about billing.

• Embedding models in the BGE, E5 and GTE families, including the multilingual variants
• Cross-encoder rerankers, which are the highest value per FLOP component in any RAG pipeline
• DeBERTa and RoBERTa class classifiers for intent, sentiment, routing and PII detection
• Named entity recognition and span extraction models

These run at hundreds to thousands of items per second per node on any of the four platforms. They quantize to INT8 with almost no accuracy loss. If your architecture currently sends embedding and reranking calls to a hosted API, you are paying network latency and per-token fees for arithmetic that a $1.43 per hour bare metal node does in the background. Move this tier first. It is the lowest risk migration in the entire stack and it usually pays for the node by itself.

Tier 1 — Sub-2B, the reflex layer

Qwen3 0.6B and 1.7B, Llama 3.2 1B, SmolLM3 at its smaller configurations, Granite Guardian for safety classification, and the xLAM 1B function calling model.

These are not conversational models, and you should stop evaluating them as though they were. They are reflexes. Jobs that fit:

• Routing and dispatch. Deciding which of twelve downstream agents handles a request
• Confidence scoring on another model’s output
• Schema validation and repair on malformed tool calls
• PII scrubbing before a request leaves the tenancy
• Query rewriting and expansion in front of a retriever
• Draft models for speculative decoding against a larger model on the same node

A 1B model at Q8 is roughly 1 GB. You can hold twenty of them resident on one node with room to spare and never pay a cold start.

Tier 2–3B to 4B, the workhorse

Qwen3–4B-Instruct, Phi-4-mini, Gemma 4 E2B and E4B, Llama 3.2 3B, SmolLM3–3B, Granite 4.x small variants.

This is where most enterprise agent steps actually land, and it is where I would start any proof of concept. These models genuinely do structured extraction, tool calling, summarization of moderate length documents, classification with reasoning, and single-hop RAG answering. They are not good at multi-hop reasoning over long context, and you should not ask them to be.

Llama 3.2 3B remains the safest choice for compatibility if your agent framework predates 2026, because every library was tested against it first. Qwen3–4B is the stronger model on most tasks. Gemma 4 E2B and E4B are the ones to test if you need vision alongside text on the same node.

Memory footprint at Q4 is 2 to 2.5 GB. At Q8, 4 to 4.5 GB. On the X12 node that means you can keep a hundred distinct fine-tunes resident simultaneously, which brings me to the point I promised to come back to.

Tier 3–7B to 9B, the escalation target

Granite 4.1 8B, Qwen3–8B, Ministral 3 8B, Mistral 7B Instruct v0.3, DeepSeek-R1-Distill-Qwen-7B for math and logic heavy work.

Granite 4.1 8B deserves specific attention for enterprise buyers because IBM built it for exactly this deployment pattern and its tool calling reliability shows it. Mistral 7B v0.3 has dedicated tokens for tool calls, available tools and tool results in its vocabulary, which makes constrained agent loops noticeably cleaner.

This tier is where CPU decode starts to feel slow. A 8B model at Q4 is roughly 4.7 GB, so your single stream ceiling drops to something like 85 tokens per second theoretical and 35 to 50 in practice. That is fine for a background agent and painful for anything a human is watching. Use it as the local escalation target below a real LLM call, not as your default.

Tier 4 — Sparse MoE, the play nobody is making yet

Here is where the capacity argument pays off, and where I think the interesting work is over the next year.

Mixture of experts models activate a small fraction of their parameters per token but require all parameters resident in memory. A 35B model with 3B active parameters needs roughly 20 GB at Q4 for weights but only touches about 1.8 GB per token. On a GPU this is an awkward fit that pushes you toward multiple cards. On a node with 1.5 TB of DDR5 it is comfortable, and the decode bandwidth requirement is closer to a 3B dense model than a 35B one.

Candidates worth benchmarking are the Qwen3.5 and 3.6 MoE variants in the 30B-A3B class, gpt-oss-20b, and the Nemotron Nano family.

Be honest about the caveat. Prefill on these is still expensive, MoE routing is cache hostile, and expert activation patterns can thrash your memory subsystem in ways that dense models do not. I would not put an MoE model on the critical path of a latency-sensitive workflow today. I would absolutely benchmark one for a nightly batch pipeline where the quality jump over a 4B dense model is worth the wall clock.

Part 4

These are the Agent Patterns that belong on CPU.

Model selection is half the answer. The other half is which agentic architectures actually thrive here.

Patterns that fit well

Router and dispatcher agents. Short prompt, short output, high call volume, and every millisecond of network hop you remove is a millisecond off the critical path. Colocating the router on the same node as the tools it dispatches to turns a network call into a loopback.

Extraction and structured output agents. Invoice fields, clinical note sections, contract clauses, log line parsing. Pair a Tier 2 model with grammar constrained decoding (GBNF in llama.cpp, XGrammar or Outlines elsewhere) and a 4B model produces schema-valid JSON at a rate that embarrasses an unconstrained 70B. Constrained decoding costs almost nothing on CPU and it is the single technique that makes small models production credible for structured work.

Full RAG pipelines. Embed, retrieve, rerank, synthesize. Three of those four stages are Tier 0 work. The fourth is a Tier 2 model with 2K of retrieved context. On E6 DenseIO Acceleron you can run the vector index on 81.6 TB of local NVMe and the agent on the same 192 cores, which removes the storage network from your latency budget entirely.

Batch and background agents. Nightly reconciliation, document classification backlogs, compliance sweeps, log triage, test generation, evaluation and judging runs. Nobody is watching the clock. Throughput per dollar is the only metric. This is the single best fit for the CPU tier and it is usually 30 to 50 percent of an enterprise’s total agent token volume.

Tool-heavy agents where the model is not the bottleneck. If your agent spends 800 ms waiting on a database query and 200 ms on inference, moving inference from 200 ms to 400 ms changes end-to-end latency by 20 percent while cutting cost by an order of magnitude. Measure where your time actually goes before you assume you need a GPU.

Guardrail and policy sidecars. Input classification, output screening, jailbreak detection, PII redaction. These have to run on every single call, which makes them the worst possible use of GPU capacity and an ideal use of spare CPU cores.

Per-tenant and per-domain model fleets. This is the capacity argument made concrete. A hundred fine-tuned 4B adapters resident in DRAM, zero cold start, no MIG partitioning, no model swapping, no scheduler contention. If you serve many customers or many narrow domains, the CPU tier gives you something a GPU physically cannot at any price: model density.

Speculative decoding pairs. A 0.6B draft model plus an 8B target on the same node. Decode is memory bound, verification is batched compute, and CPUs have compute headroom during decode. This is one of the few places where CPU serving has a structural advantage worth exploiting.

Patterns that do not belong here

I would rather you hear this from me than discover it in production.

• Long context anything. Prefill over 16K tokens on CPU is slow and no amount of AMX fixes it. If your agent needs 100K of context, that call goes to a GPU or an API.
• Interactive chat with tight time to first token. If a human is watching a cursor blink, put it on GPU.
• Extended reasoning traces. Thinking mode models burn thousands of tokens before producing an answer. On a memory bound decode path that is the worst possible token profile.
• High volume vision and multimodal. Image encoders are compute bound and CPU vision throughput collapses under load.
• Real-time voice. The latency budget is unforgiving.
• Training and fine-tuning of anything nontrivial. Use GPUs.

Part 5

Now, let’s go over the Reference Architecture

The pattern I recommend, and the one I keep seeing survive contact with production, has three tiers and one router.

Tier A, always local on CPU. Embeddings, rerankers, classifiers, guardrails, routers, extraction. Tier 0 and Tier 1 models. Runs on X12 Standard Ax or A4 Standard Ax. Never leaves the tenancy.

Tier B, local on CPU with escalation. Tier 2 and Tier 3 models handling the majority of agent reasoning steps. A confidence check on the output decides whether to accept or escalate.

Tier C, remote on GPU or API. OCI Generative AI Service, or your own GPU cluster, for long context, hard multi-hop reasoning, and anything Tier B scored low confidence on.

A LiteLLM-based router sits in front, runs on the same CPU node, and adds negligible latency. Observability through LangFuse or OpenTelemetry, with cost attribution per invocation, so you can tune escalation thresholds against real data rather than intuition.

Escalation rate is the number to watch. In the deployments I have seen, a well-tuned router keeps 70 to 85 percent of calls on the CPU tier. If you are escalating more than half, your Tier B model is wrong or your prompts are doing work a schema should be doing.

Cluster shape

For anything beyond a single node, run it on OKE with one node pool per model tier. Tier 0 and Tier 1 scale horizontally on A4 Standard Acceleron, since the Arm shapes give you the best density per dollar for small models. Tier 2 and Tier 3 go on X12 Standard Acceleron for the AMX advantage on prefill. Autoscale on queue depth with KEDA rather than on CPU utilization, because CPU utilization during memory bound decode is a misleading signal and will cause you to overprovision.

Keep model weights on OCI Object Storage, pull to local NVMe on node start, mmap from there. Model load on a 4B GGUF from local NVMe is a few seconds, so node scale-out is fast enough to be reactive.

Part 6

Software Stack by Silicon

The stacks differ, and this trips people up.

Intel Xeon 6 (X12 Standard Ax). OpenVINO with OpenVINO Model Server, or vLLM with Intel Extension for PyTorch. Both activate AMX automatically on Granite Rapids for INT8 and INT4 paths. llama.cpp works natively and is what I used for my benchmarks. Prefer Q8_0 and Q4_0 GGUF layouts, since those are the formats with the best repacking support.

AMD EPYC Turin (E6 Standard Ax, E6 DenseIO Ax). llama.cpp with AVX-512 and VNNI, or ZenDNN under ONNX Runtime. No AMX equivalent, so prefill is weaker per core than Xeon 6, but you get 192 cores and 200 G of network. Better fit for embedding fleets and batch pipelines than for latency-sensitive decode.

AmpereOne M (A4 Standard Ax). llama.cpp built with GGML_CPU_KLEIDIAI=ON, which pulls in Arm's optimized microkernels for i8mm and dot product paths. Ampere also publishes tuned container images and its own quantization formats. Check the container tag matches your shape generation before you debug a phantom performance problem for an afternoon.

Everywhere. Serve through an OpenAI-compatible endpoint so LangChain, LangGraph, CrewAI or your own orchestration connects identically regardless of what is underneath. Turn on prefix caching. Agent workloads repeat the same system prompt and the same tool schema block on every call, sometimes 2,000 tokens of it, and caching that prefix is often a larger win than any quantization choice you will make.

Part 7

How to Decide

The decision procedure I give teams, in order:

1. Instrument your current agent traffic. Bucket every call by input tokens, output tokens, and whether a human is waiting.
2. Everything with input under 4K, output under 512, and no human waiting is a CPU candidate. In most enterprises that is the majority of calls.
3. Move Tier 0 work first. Embeddings and reranking. Lowest risk, immediate savings, no quality argument to have with anyone.
4. Pick one agent step and try a Tier 2 model with constrained decoding against it. Compare on your own eval set, not on MMLU.
5. Set escalation thresholds from measured confidence distributions, not from a guess.
6. Only then argue about which silicon. The model and pattern decisions dominate the hardware decision by a wide margin.

The economics follow from that ordering. At $1.43 per hour for 120 cores against GPU instance pricing, the arithmetic is not close for the workloads that fit. But the savings come from correct placement, not from the price per core. A team that moves the wrong workloads to CPU and then fights latency complaints for a quarter has lost more than it saved.

This is the part that actually matters

The industry spent three years learning to think about AI infrastructure as a single question. How much accelerator do I need? That framing was correct when the only thing anyone did with a model was chat with it.

Agentic systems broke the framing. An agent is not one inference call, it is a topology of them. A router here, a classifier there, six extraction steps, two tool calls, one genuinely hard reasoning step, a guardrail check on the way out. Those nodes have wildly different requirements, and treating them as interchangeable calls to one endpoint is the reason so many enterprise agent programs have cost curves that terrify their CFOs.

Cognition in these systems is a topology, not a feature. My entire technology thesis rests on this — topology, not a feature. Once you accept that, infrastructure placement stops being a procurement question and becomes an architecture question, which is where it belonged all along. Most of the nodes in that topology are small, repetitive, and cheap. OCI now has four CPU platforms built for exactly those nodes, with more memory than any accelerator you can buy and a price per core that makes the arithmetic obvious.

Put the reflexes on CPU. Save the accelerators for the thinking.

Sanjay Basu leads GPU and GenAI solutions at Oracle Cloud Infrastructure and writes A Technocrat’s Discernment.