I use Claude for most of my scripting and analysis, and I'm not switching. But there's a specific case where a local model wins: when the data can't leave the box. Customer logs, an internal config full of hostnames and keys, a client's incident dump — I'm not pasting that into a web form, full stop. For that, Ollama runs a decent model on my own hardware, offline, and the data never touches anyone's API. It's the right tool for "good enough, and private", not "the smartest possible answer".

TL;DR:

# one-line install
curl -fsSL https://ollama.com/install.sh | sh

# pull a model and chat
ollama pull llama3.1:8b
ollama run llama3.1:8b "explain this ufw rule: -A ufw-user-input -p tcp --dport 22 -j DROP"

# or hit the local API from a script
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b", "prompt": "summarize:", "stream": false
}'

Details below. This pairs with how I use hosted models in /how-i-use-claude-powershell-scripts/.

Install and first run

The install script sets Ollama up as a systemd service listening on 127.0.0.1:11434:

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama          # it's already running

Pull a model — this downloads the weights once, then it's local forever:

ollama pull llama3.1:8b          # ~4.7 GB, solid general model
ollama pull qwen2.5-coder:7b     # better at reading configs/code
ollama list                      # what you've got

ollama run <model> "<prompt>" gives you a one-shot answer; ollama run <model> with no prompt drops you into an interactive chat. That's the whole basic UX.

Hardware, honestly

This is where expectations meet reality. The rule of thumb: a model needs roughly its file size in free RAM (or VRAM on a GPU).

  • 7–8B models (llama3.1:8b, qwen2.5-coder:7b) run on any box with ~8 GB free RAM. On CPU only, expect a few tokens per second — usable for a paragraph, tedious for an essay.
  • A consumer GPU (even an 8 GB card) makes those same models genuinely snappy. Ollama auto-detects NVIDIA/AMD and offloads.
  • 70B models want ~40 GB+ and a real GPU. Don't bother on a typical VPS; you'll wait minutes per answer.

For sysadmin summarizing and Q&A, an 8B model on a machine with a mid-range GPU is the sweet spot. Don't chase the biggest model — chase the one that answers in seconds.

The API is the useful part

The CLI is a toy compared to the HTTP API. Everything Ollama does is exposed at localhost:11434, so you can pipe real work through it from bash or PowerShell. Set "stream": false for scripts so you get one JSON blob instead of a token stream:

curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:7b",
  "prompt": "Explain what this cron line does: 0 */6 * * * /opt/sync.sh",
  "stream": false
}' | jq -r .response

Because it's OpenAI-API-compatible at /v1/chat/completions, most tools and SDKs that speak OpenAI point at it with just a base-URL change — handy for wiring it into something you already have.

The case it's actually for: offline log analysis

Here's the workflow I keep coming back to — hand a model a log excerpt and ask what's wrong, without any of it leaving the machine:

#!/usr/bin/env bash
# whatsup.sh — summarize the last hour of a service's journal, locally
LOG=$(journalctl -u "$1" --since "1 hour ago" --no-pager | tail -n 200)

curl -s http://localhost:11434/api/generate -d "$(jq -n --arg log "$LOG" '{
  model: "llama3.1:8b",
  prompt: ("You are a Linux sysadmin. Summarize errors and likely causes:\n\n" + $log),
  stream: false
}')" | jq -r .response

./whatsup.sh nginx

jq -n builds the JSON so newlines and quotes in the log get escaped properly — hand-concatenating strings into that -d payload breaks the moment a log line contains a quote. This is the local mirror of the hosted-model pipeline in /ai-log-analysis-claude-pipe/; use the local one when the logs are sensitive, the hosted one when you want the sharpest read.

Gotchas

  • It's not Claude. An 8B local model is fine for summarizing, explaining a command, or a first-pass triage. It will confidently invent flags and hallucinate config syntax. Verify anything it tells you to run. Local means private, not infallible.
  • First token is slow, then it warms up. The model loads into RAM on first use and stays for a few minutes (OLLAMA_KEEP_ALIVE). A cold call feels broken; it isn't.
  • RAM is the wall, not disk. Pulling a model you can't fit just makes it swap and crawl. Match model size to free memory, not free disk.
  • Don't expose 11434 to the network naively. There's no auth. If you must reach it from another host, put it behind a reverse proxy with a token — see /nginx-reverse-proxy-setup/ — don't just bind it to 0.0.0.0.
  • CPU-only 70B is a trap. It technically runs. You'll age waiting. Stay at 7–13B without a GPU.

Where it fits

Local models aren't a replacement for a good hosted one — they're the answer to "this data can't go to a cloud". Keep both: Ollama for the private, offline, good-enough jobs, and a hosted model for the hard problems. If you want to go further and give a model real tools instead of just chat, that's the MCP path in /build-mcp-server-sysadmin-tools/. Running it in a container instead? /docker-install-ubuntu/ plus the official ollama/ollama image gets you the same thing isolated.

Related posts