Run LLMs Locally: Start Small and Verify the Response

To run LLMs locally, first establish a small model's request path on the actual machine. Do not start with a 120B download or a promised tokens-per-second figure. The primary path here uses Ollama, a documented small model, loopback binding, and a short HTTP check. It is a proposed procedure, not a report that this hardware or model was tested for the article.

Before downloading: identify the machine and the model

Record the operating system, processor, available RAM, GPU/VRAM or unified memory, free disk, and runner version. Check the current Ollama GPU documentation for the actual device and driver instead of using a model's parameter count as a compatibility certificate.

For this first check, the official llama3.2:1b library entry documents a small text model and a roughly 1.3 GB artifact. That is download size, not a guarantee that 1.3 GB of free RAM is sufficient. Check its model license and leave headroom for runtime, context, and the operating system.

This model is a convenient small starting example, not a recommendation that it is the strongest or newest model in 2026. Use non-sensitive prompts until the route is established. Downloading weights and installing software require network access; neither was done for this revision.

Install the runner, then control the server you are testing

Use the official Ollama download and quickstart for the operating system. After installation, record:

ollama --version
ollama list

An installed desktop app or service may already own port 11434. Decide whether to use that configured instance or stop it and launch a controlled foreground server. Starting a second server does not change the first one's settings.

For the macOS/Linux terminal path, after stopping any existing Ollama instance through its normal controls, use:

OLLAMA_NO_CLOUD=1 OLLAMA_HOST=127.0.0.1:11434 ollama serve

Keep this terminal open and run client commands in a second terminal. On Windows, use the documented environment-variable procedure, restart the app, and confirm the effective settings rather than pasting POSIX shell syntax into PowerShell.

Ollama documents local-only mode through OLLAMA_NO_CLOUD=1 or its server setting, followed by a restart. Check the log's cloud-disabled status. The loopback address avoids deliberately exposing this development server to another machine. Ollama configuration FAQ

Pull one small model and inspect the request path

Once the server configuration is known, download only the selected model:

ollama pull llama3.2:1b
ollama list

Then make a short non-streaming request. The following would perform actual local inference if executed; it was not run for this article.

curl --fail-with-body --silent --show-error \
  http://127.0.0.1:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "llama3.2:1b",
    "prompt": "Reply with only the word READY.",
    "stream": false,
    "options": {"num_ctx": 2048, "num_predict": 64}
  }' \
  -o local-response.json

If curl reports failure, inspect the sanitized error before continuing. stream: false makes this check a single JSON response instead of a sequence of stream events. The generate endpoint reference defines the completion, output, and timing fields.

Verify completion, not just an HTTP 200

Check the returned model, completed generation, stop reason, and expected marker. A length-limited response is not an acceptable result for this particular marker test, even if the HTTP request succeeded. The verifier below reads a response file; it never sends a request:

"""Validate a saved, non-streaming Ollama smoke-test response without networking."""

import json
import sys
from pathlib import Path


def verify_response(data, expected_model="llama3.2:1b"):
    if not isinstance(data, dict) or "error" in data:
        raise ValueError("Expected a successful response object")
    if data.get("model") != expected_model:
        raise ValueError("Unexpected model")
    if data.get("done") is not True or data.get("done_reason") != "stop":
        raise ValueError("Generation did not stop normally")
    output = data.get("response")
    if not isinstance(output, str) or output.strip() != "READY":
        raise ValueError("The smoke-test marker did not match")
    return {"model": expected_model, "complete": True, "marker_matches": True}


def main():
    if len(sys.argv) != 2:
        raise SystemExit("usage: python3 verify_response.py local-response.json")
    try:
        data = json.loads(Path(sys.argv[1]).read_text())
        result = verify_response(data)
    except (OSError, ValueError) as error:
        raise SystemExit(str(error))
    print(json.dumps(result))


if __name__ == "__main__":
    main()

Use it after a successful curl request:

python3 verify_response.py local-response.json

The offline checks for this verifier use fabricated responses, including wrong model, unfinished generation, truncated output, and malformed JSON. They test the parser and acceptance rule. They are not a local inference run or a hardware performance test.

While the real model remains loaded, ollama ps can show CPU/GPU placement. If it has already unloaded, an empty list is not evidence that the completed request used a cloud model. Record timing and placement at the time of the check. One short successful response does not establish quality for coding, extraction, or an Agent loop.

Exercise the response check before downloading a model

Download verify_response.py, a synthetic READY response, and its tests. This supplies the actual filename used by the command above. The fixture is fabricated and marked as such; it is not a captured Ollama response.

python3 -m unittest -v test_verify_response.py
python3 verify_response.py synthetic-ready.json

A fixture pass shows that the verifier accepts a complete marker response and rejects the tested malformed or incomplete cases. Replace the fixture with the real saved response only after an approved local request. It says nothing about whether your hardware loads the weights.

Budget memory beyond the weight file

Think in separate categories: model weights, context/KV cache, runtime buffers, concurrent requests, and other applications. Increasing context or concurrency can turn a loadable model into an out-of-memory failure without changing the downloaded artifact.

Reduce one variable at a time. Keep the first prompt short, the context explicit, and one request in flight. If memory is insufficient, choose a smaller supported artifact or a different quantization and recheck quality. Do not infer a speed improvement solely from fitting the weights into memory.

The llama.cpp project documents quantized and hybrid CPU/GPU inference. Those features are options to evaluate, not proof of a specific laptop's throughput. This guide does not publish new RAM-tier benchmarks or promise that every 8 GB machine can run the same workload.

Common failures and the next useful check

Stop if the process crashes, consumes resources needed by other work, or sends data through an unapproved route. Do not automatically retry an out-of-memory request with unchanged settings.

Local-only inference is a data-path claim

Ollama's documentation distinguishes local inference from cloud features. Disabling its cloud features does not establish the behavior of every connected editor, extension, tool, or web-search integration. Review those components separately before using private material.

For an offline check, download the chosen weights first, disable cloud features, restart the correct server, and test a harmless prompt after disconnecting external network access if that fits the environment. Keep the endpoint, settings, logs, and result together. This procedure is stronger evidence than the word "local" in an application's name, but it is not a universal privacy certification.

When the other runners make sense

These are workflow choices, not a benchmark ranking. Local inference has hardware, electricity, and maintenance costs even when there is no hosted per-token invoice. Compare total costs for the actual usage pattern rather than declaring either local or hosted universally cheaper.

Conclusion: preserve a reproducible local check

Run LLMs locally with a small artifact first, then retain the version, memory conditions, server settings, and completed response. Stop the foreground server from its own terminal when finished; an app or system service must be managed through its own controls. Preserve previous settings before changing a shared installation.

FAQ

Was a local model run for this article?

No. The instructions were checked against documentation and the response verifier was tested offline. No weights were downloaded and no device speed was measured.

Is the model's file size the required RAM?

No. Context, runtime buffers, concurrency, and other applications add memory demand. A loadable artifact is not a guarantee that a longer workload fits.

Does a loopback response establish application privacy?

No. It checks this local request path. Connected tools, extensions, cloud features, and other application traffic require separate review.