Deploying Local LLMs with Ollama: A Hands-On Guide

Running large language models locally offers IT professionals unprecedented control over their AI workflows. Instead of relying on cloud-based services that send your data across the internet, local LLMs keep everything on your infrastructure. Today, we'll explore Ollama, a powerful tool that makes deploying local LLMs as straightforward as installing any other software.

Why Local LLMs Matter for IT Teams

Local LLMs provide several advantages that cloud solutions simply can't match. Your sensitive network configurations, security policies, and internal documentation never leave your environment. You eliminate API costs and network dependencies, ensuring your AI tools work even during internet outages. Plus, you get consistent performance without worrying about rate limits or service availability.

Installing Ollama

Ollama simplifies local LLM deployment across Linux, macOS, and Windows. Let's start with a Linux installation, which is ideal for server environments.

Download and install Ollama using the official install script from ollama.com:

curl -fsSL https://ollama.com/install.sh | sh

For macOS, download the installer from ollama.com or install via Homebrew:

brew install ollama

Windows users can download the official installer from ollama.com directly, or install via winget:

winget install Ollama.Ollama

Verify your installation by checking the version:

ollama --version

Deploying Your First Model

Ollama provides a curated model library at ollama.com/library with over 100 open-source models covering general use, coding, reasoning, and embedding tasks. For IT projects, a good starting point is Llama 3.1 8B, which provides strong general-purpose performance and runs well on standard hardware with 8GB or more of VRAM.

Pull the model from Ollama's library:

ollama pull llama3.1

This downloads approximately 4.7GB, so ensure you have adequate disk space and bandwidth. The process typically takes 5 to 15 minutes depending on your internet connection.

Once downloaded, start an interactive session:

ollama run llama3.1

You'll see a prompt where you can immediately begin testing the model with IT-specific queries.

Practical IT Use Cases

Local LLMs excel at several IT tasks that would be risky to send to external services. Try these practical examples:

Network troubleshooting assistance: Paste sanitized log entries and ask for analysis. The model can identify patterns and suggest troubleshooting steps without exposing sensitive network details.

Configuration review: Submit configuration snippets (with sensitive data removed) for optimization suggestions or security analysis.

Documentation generation: Provide the model with technical specifications and ask it to generate user-friendly documentation or procedure guides.

Setting Up API Access

For integration with existing tools and scripts, Ollama provides a REST API that runs automatically when you start a model:

curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.1",
    "prompt": "Explain OSPF routing protocol",
    "stream": false
  }'

This API endpoint allows you to integrate local LLM capabilities into your existing automation scripts, monitoring dashboards, or custom applications.

Resource Management and Performance

Different models have varying resource requirements. The 7B and 8B models like Llama 3.1 8B require approximately 8GB of VRAM, while larger 13B models need around 10 to 16GB depending on quantization. Models of 70B or more require 32GB or more of VRAM and are generally not suited to standard workstations. Monitor system resources during initial testing:

htop

If performance is sluggish, consider using a smaller quantized model. Ollama's library pages show available quantization options for each model, and Q4 quantized versions significantly reduce memory requirements with minimal quality impact for most IT documentation and analysis tasks.

You can also limit GPU usage if running on systems with limited VRAM by setting environment variables before starting Ollama.

Security Considerations

While local LLMs enhance security by keeping data on-premises, implement proper access controls. By default, Ollama binds to localhost only, but you can configure network access for team use. Always use HTTPS in production environments and implement authentication for multi-user scenarios.

Regular model updates ensure you benefit from the latest improvements:

ollama pull llama3.1

What's Next

Now that you have Ollama running with local LLMs, the next step is exploring model customization and fine-tuning for your specific IT environment. We'll cover how to create custom models trained on your organization's documentation and procedures, making your AI assistant even more valuable for daily operations.