Can I run an LLM without a graphics card?

Yes, with llama.cpp or Ollama on the CPU, using small quantized models. It is much slower than a GPU but sufficient for experiments and light use.

Running LLMs Locally: Ollama vs llama.cpp vs vLLM Compared

Once the model is tuned, it is time to run it. The serving tool you choose determines your cost, response latency, and how many users you can serve. Here is a practical comparison of the three most popular tools.

	Ollama	llama.cpp	vLLM
Ease of use	Highest: a single command	Medium	Needs a GPU and setup
Hardware	CPU, GPU, and Apple Silicon	CPU, GPU, and Apple Silicon	GPU (primarily NVIDIA)
Format	GGUF via Modelfile	GGUF	Safetensors (Hugging Face)
Concurrent users	Limited	Good with tuning	Excellent (continuous batching)
Best for	Development, experiments, and office machines	Fine-grained control and limited hardware	High-load production

Ollama: the fastest way to start

It downloads models and runs them as a local service with a REST API. Suited to experiments and internal tools with light load.

bash

ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Define RAG in one sentence"

To run your own tuned model, create a Modelfile that points to a GGUF file:

bash

# Modelfile
FROM ./mit-merged-Q4_K_M.gguf
PARAMETER temperature 0.3
SYSTEM "You are Mit, the digital assistant of Mit AI Technology."

# then:
ollama create mit-assistant -f Modelfile
ollama run mit-assistant

llama.cpp: full control and modest hardware

The engine that many tools are built on. llama-server provides an OpenAI-compatible server, supports multiple quantization levels (Q4_K_M is a good balance of size and quality), and can offload some layers to the GPU.

bash

llama-server -m mit-merged-Q4_K_M.gguf --port 8080 -c 4096 -ngl 99

vLLM: high-load production

Designed to serve a large number of concurrent requests efficiently through continuous batching and KV-cache memory management. It needs a GPU and reads models in Hugging Face format.

bash

pip install vllm
vllm serve ./out/mit-merged --port 8000 --max-model-len 4096

Connecting to your app

All three tools expose an OpenAI-compatible API, so changing base_url is enough. This is how our website backend connects to local Qwen models (AI_PROVIDER=custom).

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

reply = client.chat.completions.create(
    model="./out/mit-merged",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

Running LLMs Locally: Ollama vs llama.cpp vs vLLM Compared

Ollama: the fastest way to start

llama.cpp: full control and modest hardware

vLLM: high-load production

Connecting to your app

FAQ

Is your data sensitive, and do you want a solution that runs inside your company?