AI ile log anomaly detection: GPT-4o mini syslog hatalarını bulma

Son bir kaç ay devoted I spent tuning anomaly detection on our logging pipeline, and the jump from classic threshold-based alerts to LLM-assisted pattern spotting was noticeable. In my environment, syslog streams from hundreds of Linux nodes generate gigabytes of text daily, and spotting the occasional kernel oops or module load failure buried in noise used to mean writing ever-more complex grep chains. I decided to see whether a small, fast model like GPT-4o mini could flag those rare but critical lines with minimal false positives, and then trigger a remediation playbook automatically.

Why GPT-4o mini for syslog anomaly detection

I already run a centralized ELK stack, so the raw data is there. The challenge was not collection but interpretation: a single line like kernel: BUG: unable to handle kernel paging request at ffffffffc0a00000 looks innocuous until you realize it precedes a hard lock. Traditional regex rules either missed variants or flooded us with false positives on similar-looking debug output from development kernels. Because GPT-4o mini is cheap per token and responds in under a second, I could run it on a sliding window of the last 5k lines every minute without blowing the budget.

Prompt engineering that worked

After a few iterations I settled on this system prompt:

You are a senior Linux kernel engineer reviewing syslog entries.
Flag ONLY lines that indicate a genuine anomaly: kernel oops, lockup, hardware error, filesystem corruption, or module load failure that requires human investigation.
Ignore informational messages, routine debug output, and known harmless patterns.
If uncertain, respond with EXACTLY the word "NULL".
If anomalous, output the original line unchanged.

I keep the user message simply the batched log chunk. The model either echoes back the offending line or returns NULL. This strict contract makes post-processing trivial: any non-NULL response gets forwarded to our alert manager.

Implementation sketch

Here’s the Python snippet I run as a systemd service on the log collector:

import openai, os, sys, json
from datetime import datetime, timedelta

def fetch_recent_lines():
    # In my setup, Fluentd writes to /var/log/aggregate/syslog- recent
    cmd = ["tail", "-n", "5000", "/var/log/aggregate/syslog- recent"]
    return os.popen(" ".join(cmd)).read().splitlines()

def call_model(lines):
    client = openai.OpenAI(api_key=os.getenv"OPENAI_API_KEY")
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": "\n".join(lines)}
        ],
        temperature=0.0,
        max_tokens=1024,
    )
    return resp.choices[0].message.content.strip()

if __name__ == "__main__":
    lines = fetch_recent_lines()
    out = call_model(lines)
    if out and out != "NULL":
        payload = {
            "timestamp": datetime.utcnow().isoformat() + "Z",
            "host": os.uname().nodename,
            "anomaly": out
        }
        # send to webhook or logger
        sys.stdout.write(json.dumps(payload) + "\n")

I wrapped this in a one-minute timer unit so it never overlaps itself. CPU usage stays under 2 % on a modest VM; the OpenAI API bill for our ~300 MB/day syslog volume averages under $0.03.

From detection to action

Detecting the line is only half the battle; I wanted automated containment. When the script emits a JSON payload, a sidecar Fluentd filter matches it and pushes the event to a dedicated Kafka topic. A small Go consumer then:

Adds enrichment: recent dmesg tail, associated systemd unit status, and the last 20 journalctl lines from the same host.
Looks up the host in our CMDB to determine ownership and slack channel.
Posts a formatted message to that channel with a one-click "Run diagnostics" button that triggers a pre-approved Ansible tower job template (collects sosreport, runs memtester, and opens a ticket).

Because the model’s false-positive rate stayed below 5 % in my six-week trial, the signal-to-noise ratio improved enough that engineers actually started clicking the button instead of muting the alert.

Pitfalls and tuning tips

Token limits: GPT-4o mini’s context window is 128k, but I keep batches under 4k tokens to stay fast and cheap. If you see truncation, lower the batch size or increase the sliding overlap.
Prompt drift: After a kernel upgrade, the model started flagging new -[smpboot] Booting Node lines as anomalous. I added a simple allow-list regex for known-boot messages as a preprocessing step.
Cost surprises: Watch the usage field in the API response; a runaway loop can burn dollars quickly. I added a hard daily cap via OpenAI’s project-level budget.
Privacy: If your syslog contains PII or internal IPs, either scrub before sending or use Azure OpenAI with private network endpoints.

Sonuç

Using GPT-4o mini for syslog anomaly detection gave me a practical lift in spotting real kernel faults without drowning in noise. The approach is lightweight enough to run beside existing pipelines, and the feedback loop—detect → enrich → act—closed in under two minutes. If you’re already sending logs to a central place, wrapping the tail of the stream in a well‑prompted LLM call is worth a weekend experiment.

As I mentioned before in my post about Auditd and crontab changes (<https://furkanikkan.com/urun/auditd-ile-yetkisiz-crontab-degisikliklerini-anlik-siem-e-aktarma-78>), the pattern of "detect a line, enrich, then automate" repeats across use cases; swapping regex for an LLM is just another tool in that same belt.

Cover image: ₡ґǘșϯγ Ɗᶏ Ⱪᶅṏⱳդ · CC0 (Openverse / kamu malı) · https://www.flickr.com/photos/148598741@N02/51973552248