Local LLMs with Ollama: Building AI Solutions On-Premises

Cloud-based AI services like ChatGPT and Claude are powerful, but they come with a significant trade-off: your data leaves your network. For IT professionals handling sensitive information, whether that's internal infrastructure details, customer records, or proprietary configurations, sending that data to an external API is often a non-starter. That's where local LLMs and tools like Ollama change the game.

In this post, we'll walk through what Ollama is, how to get it running, and how you can start building practical on-premises AI solutions that keep your data exactly where it belongs.

What Is Ollama?

Ollama is a tool that makes it straightforward to download, run, and manage large language models locally on your own hardware. Think of it as a package manager for AI models. Instead of configuring complex Python environments or dealing with CUDA drivers manually, Ollama handles the heavy lifting and exposes a clean REST API that your scripts and applications can talk to.

It supports a wide range of models, including llama3.2, mistral, gemma3, phi4, and many others. You can find the full model library at ollama.com/library.

Installing Ollama

Ollama runs on Linux, macOS, and Windows. Installation steps vary slightly by operating system.

Linux: Installation is a single command:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, the Ollama service starts automatically. You can verify it's running with:

systemctl status ollama

macOS: Download the native application from ollama.com/download and follow the installer prompts. Ollama will run as a menu bar application.

Windows: A native Windows installer is also available at ollama.com/download. After installation, Ollama runs as a background service accessible via the same REST API.

To pull your first model, use the ollama pull command followed by the model name. A solid starting point for general IT use is llama3.2:

ollama pull llama3.2

Depending on your connection speed, this downloads a file between 2GB and 8GB depending on the model variant. Once complete, you can interact with the model directly in the terminal:

ollama run llama3.2

The Built-In API

Where Ollama really shines for automation is its REST API, which listens on http://localhost:11434 by default. You can query it with a simple curl command:

curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.2",
    "prompt": "Summarize the following syslog output and identify any critical errors:",
    "stream": false
  }'

This is the foundation for building real on-premises AI solutions. Any tool or script that can make an HTTP request can now interact with a local LLM.

A Practical Example: Log Analysis with Python

Here's a working Python script that reads a log file and asks a local LLM to summarize and flag issues. No external API keys, no data leaving your network:

import requests
import json

def analyze_log(log_text):
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": "llama3.2",
        "prompt": f"Analyze the following system log. Identify any critical errors, warnings, or unusual patterns. Be concise.\n\nLog:\n{log_text}",
        "stream": False
    }

    response = requests.post(url, json=payload)
    result = response.json()
    return result["response"]

# Read a local log file
with open("/var/log/syslog", "r") as f:
    log_content = f.read()[-3000:]  # Last 3000 characters

summary = analyze_log(log_content)
print("AI Log Analysis:\n")
print(summary)

Run this with python3 analyze_log.py and you get an AI-generated summary of your system logs, processed entirely on your own hardware.

Choosing the Right Model for the Job

Not every task needs the biggest model. Here's a quick reference for selecting models based on your use case and hardware:

• phi4:mini: Lightweight and fast, good for simple text tasks on machines with limited RAM (8GB minimum)
• mistral: A strong all-around performer, great for summarization, Q&A, and scripting assistance
• llama3.2: Excellent general-purpose reasoning, recommended if you have 16GB RAM or more
• codellama: Optimized for code generation and review, ideal for automation scripts and config generation

You can switch models in your API calls simply by changing the "model" field in your request payload.

Why This Matters for IT Professionals

Running local LLMs with Ollama gives you several advantages that cloud-based tools simply cannot offer:

• Data privacy: Sensitive configs, logs, and internal documentation never leave your environment
• No API costs: Once the model is downloaded, inference is free regardless of how many queries you run
• Offline capability: Air-gapped networks and restricted environments become viable deployment targets
• Customization: You control the model version, parameters, and system prompts without vendor limitations

What's Next

Now that you have a local LLM running and serving an API, the logical next step is building more structured integrations. In the next post in this series, we'll look at how to use Python libraries like ollama-python and LangChain to build more sophisticated on-premises AI pipelines, including multi-step workflows that can assist with network documentation, configuration auditing, and automated reporting.