Furkan İkkan Türkçe
Back to posts
Post

· Network & System Administrator

AI CLI Tools for Sysadmins: Automating Log Analysis and Error Detection

How AI-powered CLI tools automate log analysis and error detection for sysadmins. Practical examples with lg, ollama, and shell pipelines.

Yapay ZekaAI CLILog AnalysisSystem AdministrationOllamaAutomationMTTD

AI ile sistem yönetimi otomasyonu isn't just a buzzword anymore — it's something I use daily. In my environment, AI-powered CLI tools handle log analysis and error detection faster than I ever could by eye. I pipe journald dumps, syslog streams, and container logs into local models or API-backed CLIs, and they surface anomalies, stack traces, and root-cause hints in seconds. Here's how I actually do it, what works, and what to avoid.

Why AI CLI Tools Beat Manual Log Grep

I used to run grep -i error /var/log/syslog and then scroll for ten minutes. That still works, but when you're managing Proxmox clusters, a dozen Windows VMs, and a couple of e-commerce stacks behind a CDN, the volume is brutal. AI CLI tools change the workflow: instead of filtering, you ask.

The key advantage isn't speed alone — it's context. A grep tells you a line matches "error." An AI CLI tool tells you why that error matters, what likely caused it, and what to check next. That's the difference between a search and a diagnosis.

As I mentioned before in my post about reducing MTTD (https://furkanikkan.com/urun/veri-ihlalini-ne-kadar-hizli-tespit-edersiniz-mttd-rehberi-57), faster detection is everything. AI CLI tools compress that detection window significantly.

Local LLMs with Ollama for Offline Log Analysis

I don't send every log to OpenAI. Some logs contain customer data, internal IPs, or infrastructure details that shouldn't leave the network. That's where ollama comes in.

I run a local model — usually llama3 or qwen2.5 — on a box with 32GB RAM and a basic GPU. Here's a typical pipeline:

journalctl -u nginx --since "1 hour ago" --no-pager | \
  ollama run llama3 "Analyze these Nginx logs. List any anomalies, \
  error patterns, and suggest root causes. Be concise."

This gives me a readable summary in 10–20 seconds. The model catches things like upstream timeouts, 499 client disconnects, and rate-limit hits — stuff that's easy to miss when you're tired at 2 AM.

Using the lg CLI for Natural Language Log Queries

There's a neat tool called lg (Language-Grep) that lets you search logs using natural language instead of regex. It's perfect for those moments when you know what you're looking for but can't build the right grep pattern.

pip install lg-ai

# Instead of crafting complex grep patterns:
lg "find all failed SSH login attempts from the same IP" /var/log/auth.log

# Or with journalctl:
journalctl --since today --no-pager | lg "show me entries about disk space warnings"

I use this mostly for one-off investigations. It's not replacing my monitoring stack — it's a fast diagnostic tool when something breaks and I need answers now.

Warning: lg sends log content to an LLM API by default. If you're dealing with sensitive logs, either configure it to use a local endpoint or stick with ollama-based pipelines.

Building Automated Error Detection Pipelines

For ongoing automation, I don't want to run commands manually. I want the system to alert me when AI detects something worth looking at. Here's a simple approach I use:

#!/bin/bash
# ai-log-scanner.sh — runs every 15 minutes via cron
LOGS=$(journalctl --since "15 min ago" --no-pager -p err..alert)

if [ -n "$LOGS" ]; then
  ANALYSIS=$(echo "$LOGS" | ollama run qwen2.5:7b \
    "Summarize these system errors. If any indicate hardware failure, \
    disk issues, or network outages, flag them as CRITICAL. \
    Output format: [SEVERITY] service: description")
  
  echo "$ANALYSIS" | grep -q "CRITICAL" && \
    /usr/local/bin/send-alert.sh "$ANALYSIS"
fi

This script sits on my Proxmox hosts and a few critical VMs. It catches things that traditional threshold-based monitoring misses — like a RAID controller throwing predictive failure warnings buried in kernel logs, or a backup job that "succeeds" but logs corruption warnings.

AI CLI Tools I Actually Use Daily

Here's my shortlist — tools that earned a permanent spot in my workflow:

  • ollama — Local model runner. The backbone of my offline log analysis. No data leaves the network.
  • lg — Natural language log search. Great for ad-hoc queries when regex would take too long to build.
  • aichat — A lightweight CLI chat client I use for quick questions about config files and error messages. Supports multiple backends.
  • shell-gpt (sgpt) — Generates shell commands from natural language. I use it less for logs and more for crafting complex one-liners. As I covered before in my Linux sysadmin tools post (https://furkanikkan.com/urun/haftada-saatler-kazandiran-25-linux-sistem-yoneticisi-araci-54), command generation saves real time.
  • ggshield — Scans code and config for secrets before they hit git. Not log analysis per se, but catches sensitive data leaking into logs and commits.

Pitfalls and Practical Warnings

A few things I learned the hard way:

Don't trust AI output blindly. Models hallucinate root causes. I always verify before taking action. AI is a starting point, not a final answer. If it says "failing disk," I still run smartctl -a /dev/sda before ordering a replacement.

Token limits matter. A full day of syslog from a busy e-commerce frontend can be 500MB+. Local models choke on that. I pre-filter with grep or journalctl -p err before sending to the model. The AI gets the interesting parts, not the firehose.

Cost adds up with API-backed tools. When I used OpenAI's API for log analysis on a high-traffic stack, I hit $40 in a week before noticing. Local models are free after the hardware investment. Use API calls selectively — for complex analysis, not bulk processing.

Log format changes break pipelines. An app update changes log format, and your AI model suddenly misinterprets everything. I pin model versions and test pipelines after major updates.

Where This Fits in the Bigger Picture

AI CLI tools don't replace my monitoring stack — Zabbix handles metrics, Grafana shows trends, and alerting rules catch threshold breaches. What AI adds is the interpretation layer. It reads the messy, unstructured stuff and tells me what matters.

For someone managing both Linux and Windows Server environments (as I do), the ability to throw PowerShell event logs and journald output at the same model and get consistent analysis is genuinely useful. Different log formats, same diagnostic workflow.

The tools I described here cost nothing to try. Install ollama, pull a 7B model, pipe your next error log through it. You'll know within a day whether it fits your workflow.

Cover image: ₡ґǘșϯγ Ɗᶏ Ⱪᶅṏⱳդ · CC0 (Openverse / kamu malı) · https://www.flickr.com/photos/148598741@N02/51894347967