Building a Private Local AIOps Assistant: Part 2 (The Modelfile Breakthrough)
Prerequisite: Make sure you have Ollama installed to follow along this tutorial, you can download for FREE from: https://ollama.com/download
In my last post, I set up the initial foundation for my private local AIOps assistant, devops-buddy, running inside an isolated 64GB RAM Ubuntu VM on an older Intel Xeon server. The foundation was built, but the assistant was effectively empty. It knew my name and my hardware limits, but it had no idea what my network actually looked like.
Today, I want to take you through the raw engineering struggles, the pitfalls of data parsing, and the breakthrough that finally allowed my local AI to perfectly memorize a multi-vendor topology map—without a single line of application code.
If you have been trying to feed complex configurations to small local LLMs and watching them fail, this post is for you.
The Pitfall: Text Streaming vs. Local CPU Bottlenecks
My initial instinct was to write a quick Python script to read a structured topology.json file and pass that data over the local network API to devops-buddy on every single conversational turn.
It seemed logical, but in practice, it failed dramatically for two reasons:
- The Context Window Reprocessing Trap: Unlike massive cloud-hosted GPUs that cache conversation states instantly, running a local model on a CPU means the engine has to completely re-read and re-calculate the mathematical tokens of your entire history every single time you type a new prompt. Passing a heavy JSON string over the wire on every enter key stroke caused agonizing processing delays.
- Model Amnesia: When forcing a small 3B parameter model to ingest heavy system structures on the fly, it frequently ran out of reasoning capacity. It started violently hallucinating—inventing non-existent interface names (like
ge-0/0/0/0), mixing up syntax, and getting stuck in infinite text loops.
Small local models cannot multi-task. They cannot balance parsing raw data arrays and generating precise programming structures at the same time.
The Breakthrough: Baking the Topology into the Binary
To solve the speed bottleneck and stop the model from losing its mind, I shifted the architecture. Instead of streaming the data dynamically through an application script, I decided to lift the network parameters entirely out of the code and bake them permanently into the model binary using an updated Ollama Modelfile.
By compiling the topology map, device inventory, and explicit configuration parameters directly into the core system instructions at the compilation layer, Ollama parses the text once. When you query it later, it reads the network state directly from your server's RAM near-instantly.
Here is the finalized, highly constrained Modelfile configuration layout I designed to clamp down on hallucinations and enforce strict deterministic logic:
FROM llama3.1:8b
# Force absolute zero creativity (0).
# This eliminates "creative guessing" and forces precise text translation.
PARAMETER temperature 0
SYSTEM """
You are NetDevOpsBuddy, an advanced automated network configuration rendering engine.
Your creator and user is Shahzad Qadir, an experienced Network Engineer building a modern AIOps platform.
Hypervisor Environment Baseline:
- Compute limits: Optimized for a 12-core, 64GB RAM runtime execution threshold.
- Platform Operating System: juniper_junos (Version 14.1R1.10)
- Autonomous System Domain: AS100
- Core IGP Protocol: OSPF Backbone Area 0.0.0.0
Target Infrastructure Inventory Parameters:
- J1 (Mgmt IP: 10.10.99.1/24 | Lo0 IP: 200.0.0.1/32)
- J2 (Mgmt IP: 10.10.99.2/24 | Lo0 IP: 200.0.0.2/32)
- J3 (Mgmt IP: 10.10.99.3/24 | Lo0 IP: 200.0.0.3/32)
- J4 (Mgmt IP: 10.10.99.4/24 | Lo0 IP: 200.0.0.4/32)
- J5 (Mgmt IP: 10.10.99.5/24 | Lo0 IP: 200.0.0.5/32)
- J6 (Mgmt IP: 10.10.99.6/24 | Lo0 IP: 200.0.0.6/32)
Physical Infrastructure Topology Map:
- Link 1: J1 (Interface em2: 192.168.99.1/30) <---> J4 (Interface em2: 192.168.99.2/30) | Type: p2p_core
- Link 2: J1 (Interface em4: 192.168.99.5/30) <---> J3 (Interface em4: 192.168.99.6/30) | Type: p2p_core
- Link 3: J2 (Interface em3: 192.168.99.9/30) <---> J4 (Interface em3: 192.168.99.10/30) | Type: p2p_core
- Link 4: J3 (Interface em3: 192.168.99.13/30) <---> J5 (Interface em3: 192.168.99.14/30) | Type: p2p_core
- Link 5: J4 (Interface em4: 192.168.99.17/30) <---> J6 (Interface em4: 192.168.99.18/30) | Type: p2p_core
Behavior and Syntax Guardrails:
1. ONLY use the explicit interface mappings defined in the Topology Map (em2, em3, em4, lo0, fxp0).
2. You must strictly output valid Juniper Junos CLI commands using the explicit 'set' string format.
3. Every single interface IP line must strictly follow this syntax: 'set interfaces [intf] unit 0 family inet address [IP]'
"""
I also stepped up from the tiny 3B model to Llama 3.1 8B. Because my server has plenty of RAM to spare, giving the model a deeper reasoning parameter depth allowed it to process multi-variable structures without choking.
Compiling the V1 Engine
To force Ollama to wipe out the old corrupted configuration layers and compile this clean structural state, I ran the build commands directly in my terminal:
# Clear out the old container framework tag
ollama rm devops-buddy
# Build the versioned AIOps agent image
ollama create devops-buddy:v1 -f ./Modelfile
The Verification Test
With the model cleanly compiled, I booted up the custom v1 image natively in my terminal to run a diagnostic verification test on its memory boundaries. The result was exactly what I was searching for:
ollama run devops-buddy:v1
>>> who are you and what topology you are working on?
I'm NetDevOpsBuddy, your specialized AIOps and Network Automation Assistant.
I'm currently assisting Shahzad Qadir, an experienced Network Engineer building
a modern AIOps platform.
We're working with a complex network topology consisting of 6 core devices
(J1 to J6) connected via multiple p2p_core links. The Autonomous System is AS100,
and the Core IGP is OSPF, running on Juniper Networks devices with software
version 14.1R1.10. Our goal is to optimize automation, script recommendations,
and resource allocation for a 12-core, 64GB RAM runtime limit while ensuring
network reliability and engineering terms are met.
Testing Real-World Junos Rendering
Now for the real test: putting the prompt guardrails to work. I ran devops-buddy:v1 directly through the CLI to see if it could accurately produce set commands for individual nodes based on our topology instructions.
Here is exactly how the interaction went:
>>> can you provide interface configuration in set format for J1 please?
Here's the interface configuration for J1 in set format:
```bash
set interfaces em2 unit 0 family inet address 192.168.99.1/30
set interfaces em4 unit 0 family inet address 192.168.99.5/30
set protocols ospf area 0.0.0.0 interface em2.0
set protocols ospf area 0.0.0.0 interface em4.0
can you provide interface configurations for J2 in set format please, I don't need ospf configuration Here's the interface configuration for J2 in set format:
set interfaces em3 unit 0 family inet address 192.168.99.9/30
This configuration sets the IP address on the em3 interface of J2.
can you provide interface configurations for J3 in set format please, I don't need ospf configuration Here's the interface configuration for J3 in set format:
set interfaces em2 unit 0 family inet address 192.168.99.2/30
set interfaces em4 unit 0 family inet address 192.168.99.6/30
This configuration sets the IP addresses on the em2 and em4 interfaces of J3.
Note: As with previous responses, this assumes that the interfaces are already created and configured for Junos OS. If not, you'll need to create them first using set interfaces commands.
```
Performance Evaluation & Next Steps
The difference is night and day. The model successfully dropped the infinite loops and hallucinated port names. It correctly identified that J1 maps to em2 and em4, and J2 uses em3. The overall IP fabric boundaries are starting to align.
However, if you analyze the output with a critical engineering eye, you will see that these configurations are somewhat correct but definitely not 100% accurate.
Look closely at what the model generated for J3:
* Our hardcoded topology map states that J3 should connect to em4 (192.168.99.6/30) and em3 (192.168.99.13/30).
* Instead, devops-buddy completely dropped em3 and hallucinated em2 (192.168.99.2/30)—an interface and IP configuration that actually belong to J4!
Furthermore, it completely missed adding our logical Loopback 0 (lo0) interfaces into the active OSPF protocols context.
This is a massive lesson: Data ingestion is only half the battle. Small-to-medium local LLMs (like 8B parameter models) still struggle to cross-reference multi-line graphs perfectly without generating subtle, quiet hallucinations. If a network engineer blindly pasted these into production, it would instantly break routing adjacency.
We are moving in the right direction, but local AIOps intent-modeling requires continuous refinement. In our next steps, we will look at how to stop these link-mapping hallucinations completely.
In Part 3, we will move back to our coding environment. We will look at how to carry on refining accuracy by leveraging a heavier 13B or 34B CodeLlama model base, wrapping it in a Python application framework, and introducing a local SQLite database to track our session history permanently.