Ollama Tutorial: Run Local LLMs with Python

What Is Ollama? — The Story of the Laptop That Became a Brain

Ollama is a free, open-source tool that lets you download, manage, and run large language models (LLMs) entirely on your local machine. It wraps the complexity of running these models — GPU acceleration, memory management, quantisation — behind a clean command-line interface and a local REST API.

What Are "Billion Parameters"? — The Chef with a Million Recipes

A parameter (also called a weight) is a single floating-point number inside a neural network. During training, the model processes trillions of words and adjusts these numbers — incrementally, over weeks on giant GPU clusters — until they collectively capture the patterns of human language.

Installing Ollama & Running Your First Model

🆕 Download and Run a Model in 30 Seconds

Ollama uses a simple pull + run workflow. Think of it like docker pull but for AI models:

# Pull the Llama 3.2 3B model (~2 GB download)
ollama pull llama3.2

# Or pull and start an interactive chat immediately
ollama run llama3.2

# Start a chat with a specific model
ollama run mistral

# Ask a single question and exit (great for scripts)
ollama run llama3.2 "What is the capital of France?"

# List all downloaded models on your machine
ollama list

# Check what models are currently loaded in memory
ollama ps

# Remove a model to free up disk space
ollama rm llama3.2

The Ollama REST API — Your Local AI Endpoint

When Ollama is running, it exposes a local REST API at http://localhost:11434. This is how Python (and any other language) talks to your local models — no internet, no API key, no rate limits.

Using Ollama with Python — Example 1: Basic Generation

Method A — Using the requests library (raw HTTP)

The most transparent approach — no extra packages beyond what Python already has nearby:

import requests
import json

# ── Configuration ───────────────────────────────────────────
OLLAMA_URL = "http://localhost:11434/api/generate"
MODEL_NAME = "llama3.2"  # must be already downloaded via: ollama pull llama3.2

def ask_ollama(prompt: str, stream: bool = False) -> str:
    """Send a prompt to local Ollama and return the full response."""
    payload = {
        "model": MODEL_NAME,
        "prompt": prompt,
        "stream": stream       # False = wait for complete response
    }
    response = requests.post(OLLAMA_URL, json=payload, timeout=120)
    response.raise_for_status()
    data = response.json()
    return data["response"]

# ── Example usage ────────────────────────────────────────────
answer = ask_ollama("Explain quantum entanglement in simple terms.")
print(answer)

Method B — Streaming responses (token by token)

For a ChatGPT-like typewriter effect where tokens appear as they're generated:

import requests
import json

def stream_ollama(prompt: str) -> None:
    """Stream the response token-by-token to stdout."""
    payload = {
        "model": "llama3.2",
        "prompt": prompt,
        "stream": True      # ← key difference
    }
    with requests.post(
        "http://localhost:11434/api/generate",
        json=payload,
        stream=True,
        timeout=120
    ) as resp:
        for line in resp.iter_lines():
            if line:
                chunk = json.loads(line)
                print(chunk.get("response", ""), end="", flush=True)
                if chunk.get("done"):
                    break
    print()  # newline at end

# Usage — tokens will appear progressively in terminal
stream_ollama("Write a haiku about machine learning.")

Using the Official Ollama Python Library

The ollama Python package provides a cleaner, higher-level interface with auto-retry, better streaming, and a fully typed API. Install it with: pip install ollama

import ollama

# ── Single prompt — simplest form ────────────────────────────
response = ollama.generate(
    model="llama3.2",
    prompt="What are three benefits of local LLMs?"
)
print(response["response"])

# ── Chat with message history (multi-turn) ────────────────────
messages = [
    {"role": "system",    "content": "You are a helpful Python tutor."},
    {"role": "user",      "content": "What is a list comprehension?"},
]
response = ollama.chat(model="llama3.2", messages=messages)
reply = response["message"]["content"]
print(reply)

# Continue the conversation by appending assistant + new user message
messages.append({"role": "assistant", "content": reply})
messages.append({"role": "user",      "content": "Can you give me an example with numbers?"})

response2 = ollama.chat(model="llama3.2", messages=messages)
print(response2["message"]["content"])

Advanced Example — Building a Local AI Code Reviewer

import ollama
import sys
from pathlib import Path

# ── Local Code Reviewer using Ollama ─────────────────────────
SYSTEM_PROMPT = """You are a senior Python developer doing a code review.
For each piece of code you receive:
1. Identify bugs or logical errors
2. Point out style issues (PEP 8)
3. Suggest performance improvements
4. Highlight any security concerns
Be specific, concise, and constructive. Use bullet points."""

def review_code(code: str, filename: str = "snippet") -> str:
    """Ask the local LLM to review a code snippet."""
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user",   "content": f"Please review this Python code from '{filename}':\n\n```python\n{code}\n```"}
    ]
    response = ollama.chat(model="codellama:7b", messages=messages)
    return response["message"]["content"]

# ── Review a code snippet directly ───────────────────────────
sample_code = """
def get_user_data(user_id):
    import sqlite3
    conn = sqlite3.connect("users.db")
    query = "SELECT * FROM users WHERE id = " + user_id
    result = conn.execute(query)
    return result.fetchall()
"""

review = review_code(sample_code, "user_service.py")
print("=== CODE REVIEW ===")
print(review)

# ── Or review a file from disk ────────────────────────────────
def review_file(filepath: str) -> None:
    code = Path(filepath).read_text()
    print(f"\n🔍 Reviewing: {filepath}")
    print(review_code(code, filepath))

# review_file("my_script.py")   # ← uncomment to use

Advanced Example — RAG: Chatting with Your Own Documents

import ollama

# ── Simple RAG: Chat with a text document ─────────────────────

def load_document(filepath: str, chunk_size: int = 500) -> list[str]:
    """Load a text file and split into chunks."""
    with open(filepath, "r", encoding="utf-8") as f:
        text = f.read()
    words = text.split()
    chunks = [
        " ".join(words[i : i + chunk_size])
        for i in range(0, len(words), chunk_size)
    ]
    return chunks

def find_relevant_chunks(chunks: list, question: str, top_k: int = 3) -> list[str]:
    """Simple keyword-based retrieval (no vector DB needed!)."""
    question_words = set(question.lower().split())
    scored = []
    for chunk in chunks:
        chunk_words = set(chunk.lower().split())
        score = len(question_words & chunk_words)  # word overlap
        scored.append((score, chunk))
    scored.sort(reverse=True)
    return [chunk for _, chunk in scored[:top_k]]

def answer_question(document_path: str, question: str) -> str:
    """Full RAG pipeline: load doc → retrieve → generate answer."""
    # Step 1: Load and chunk
    chunks = load_document(document_path)
    
    # Step 2: Retrieve relevant chunks
    relevant = find_relevant_chunks(chunks, question)
    context = "\n\n---\n\n".join(relevant)
    
    # Step 3: Build augmented prompt
    prompt = f"""You are a helpful assistant. Answer the question using ONLY 
the context below. If the answer isn't in the context, say so.

CONTEXT:
{context}

QUESTION: {question}

ANSWER:"""
    
    # Step 4: Generate answer
    response = ollama.generate(model="llama3.2", prompt=prompt)
    return response["response"]

# ── Usage ─────────────────────────────────────────────────────
answer = answer_question("manual.txt", "Does this device support Bluetooth 5.2?")
print(answer)

Generating Embeddings with Ollama

An embedding is a numerical vector that represents the meaning of text. Similar sentences produce similar vectors. This is the foundation of semantic search, recommendation systems, and clustering.

import ollama
import math

# ── Generate an embedding vector ─────────────────────────────
response = ollama.embeddings(
    model="nomic-embed-text",   # pull this first: ollama pull nomic-embed-text
    prompt="The cat sat on the mat"
)
vector = response["embedding"]      # list of 768 floats
print(f"Embedding dimensions: {len(vector)}")   # 768

# ── Cosine similarity function ────────────────────────────────
def cosine_similarity(v1: list, v2: list) -> float:
    dot   = sum(a * b for a, b in zip(v1, v2))
    norm1 = math.sqrt(sum(a**2 for a in v1))
    norm2 = math.sqrt(sum(b**2 for b in v2))
    return dot / (norm1 * norm2)

# ── Compare two sentences ─────────────────────────────────────
sentences = [
    "A feline rested on the rug.",          # very similar meaning
    "Machine learning is fascinating.",     # completely different topic
]

def embed(text: str) -> list:
    return ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]

base   = embed("The cat sat on the mat")

for s in sentences:
    sim = cosine_similarity(base, embed(s))
    print(f"Similarity: {sim:.4f}  |  '{s}'")

Model Comparison & Choosing the Right One

Golden Rules — Running LLMs Locally Like a Pro

Quick Reference — All The Commands You'll Use Daily