Foundations of LLMs and Agentic Systems

Tokens & tokenization

LLMs do not read characters or words — they read tokens, sub-word units produced by a tokenizer (BPE, SentencePiece, tiktoken). Pricing, context limits, and latency are all measured in tokens.

• Roughly 1 token ≈ 4 English characters or ~0.75 words.
• Code, JSON, and non-Latin scripts tokenize less efficiently.
• Always count tokens before sending — long prompts can silently truncate.

The transformer in one paragraph

A transformer predicts the next token by attending to every previous token in its context. Stacked self-attention layers learn which earlier tokens matter for the current position; the final layer outputs a probability distribution over the vocabulary, from which the next token is sampled.

Context window

The context window is the maximum number of tokens the model can see at once — system prompt, conversation history, retrieved documents, tool outputs, and the model's own reply all share it. Exceeding it forces truncation or summarization.

Sampling: temperature, top-p, top-k

Decoding parameters shape how the model chooses the next token. Lower temperature is more deterministic; top-p / top-k restrict the candidate set. For tool calls and structured output, keep temperature low (0–0.3).

Embeddings & vector search

An embedding model maps text to a dense vector so semantically similar text lands nearby. Vector databases (pgvector, Pinecone, Qdrant, Weaviate) index these vectors for fast similarity search — the retrieval half of RAG.

Retrieval-Augmented Generation (RAG)

RAG grounds an LLM in your data: at query time, retrieve the top-k relevant chunks from a vector store and inject them into the prompt. It is the standard way to give a model fresh, private, or domain-specific knowledge without fine-tuning.

User question
   │
   ▼
Embed query ──► Vector DB ──► Top-k chunks
                                   │
                                   ▼
                       Prompt = system + chunks + question
                                   │
                                   ▼
                                  LLM ──► Grounded answer

Prompting vs fine-tuning vs RAG

Reach for prompting first, RAG when you need fresh or private knowledge, and fine-tuning only when you need a specific style, format, or skill the base model cannot reliably produce with context alone.

What is an agent?

An agent is an LLM placed in a loop where it can choose actions (call tools, query data, hand off to another agent) based on observations, until a goal is met. The model is the reasoner; the surrounding framework is the runtime.

Tool use & function calling

Modern models accept a schema of available tools and return structured calls instead of free text when a tool is appropriate. The runtime executes the tool, feeds the result back, and the loop continues.

{
  "name": "search_orders",
  "description": "Find customer orders by email",
  "parameters": {
    "type": "object",
    "properties": { "email": { "type": "string" } },
    "required": ["email"]
  }
}

ReAct: Reason + Act

The canonical agent loop. The model alternates between a thought (private reasoning), an action (tool call), and an observation (tool result) until it produces a final answer. Most agent frameworks implement a variant of this loop.

Thought: I need the user's recent orders.
Action: search_orders(email="ada@example.com")
Observation: [{ id: 42, total: 19.99 }]
Thought: I have what I need.
Final Answer: Your most recent order is #42 for $19.99.

Planning & decomposition

For multi-step tasks, agents often plan first (Plan-and-Execute, Tree of Thoughts, Reflexion) and then execute sub-tasks. Planners trade extra tokens for fewer wrong turns on complex problems.

Memory

Short-term memory is the conversation in the context window. Long-term memory persists across sessions — usually summaries, key facts, or embeddings stored in a database and retrieved on demand.

• Episodic: past conversations and outcomes.
• Semantic: stable facts about the user, domain, or org.
• Procedural: learned workflows and reusable skills.

Multi-agent systems

Specialized agents collaborate via shared state or message passing. Common topologies: supervisor → workers (LangGraph, CrewAI), peer-to-peer chat (AutoGen), and pipeline / DAG (LlamaIndex workflows).

Evaluation & observability

Agents are non-deterministic and easy to break in subtle ways. Trace every run (LangSmith, OpenTelemetry), build a regression set of real prompts, and score outputs with rubrics or LLM-as-judge before shipping changes.

Safety & guardrails

Validate tool inputs, sandbox code execution, allow-list URLs, redact PII, and cap loop iterations. Treat any text that reaches the model — including tool outputs and retrieved documents — as untrusted input that may contain prompt injection.