I’ve been running SSH honeypots and production jump hosts for years, and the noise in /var/log/auth.log never stops. Brute-force scanners, credential stuffing bots, and the occasional lucky guess keep coming. Traditional rule-based tools like fail2ban help, but they’re reactive and often miss low-and-slow attacks that stay under threshold limits. Recently I started experimenting with LLMs to look at log streams not just for known bad patterns, but for anomalous behavior that doesn’t match my baseline of normal admin access.
Why LLMs for Log Anomaly Detection
Most log analyzers work on signatures or simple thresholds: more than 5 failed attempts from an IP in 10 minutes, block it. That works for noisy botnets, but attackers have adapted. They slow down, rotate through proxy networks, or use credentials harvested from leaks that actually exist in your user database. An LLM doesn’t care about a fixed count; it learns what "normal" looks like for your environment—time of day, usual users, typical source networks, even the rhythm of keystrokes in sudo logs—and flags deviations.
In my setup, I feed journalctl output from sshd into a small local LLM (I use Llama 3 8B via Ollama) every 30 seconds. The prompt asks: "Given the following SSH login events from the last 5 minutes, does anything look unusual compared to typical admin access patterns? List any suspicious IPs, users, or timing anomalies." The model returns a short JSON blob with risk scores and reasoning.
Building the Pipeline
First, I configured rsyslog to duplicate auth logs to a named pipe so I can stream them without losing locality:
# /etc/rsyslog.d/ssh-llm.conf
if $programname == 'sshd' then ^/var/run/ssh-llm.pipe
& stopThen a simple Python script reads the pipe, windows the last 5 minutes of events, and calls the local Ollama API:
import json, subprocess, time, collections
from datetime import datetime, timedelta
def window_events(pipe_path, minutes=5):
events = []
cutoff = datetime.now() - timedelta(minutes=minutes)
with open(pipe_path, 'r') as p:
for line in p:
if 'Failed password' in line or 'Accepted password' in line:
# parse timestamp, ip, user etc. (simplified)
events.append(line.strip())
return [e for e in events if datetime.now() - parse_time(e) < timedelta(minutes=minutes)]
while True:
try:
events = window_events('/var/run/ssh-llm.pipe')
if not events:
time.sleep(10)
continue
prompt = f"""Analyze these SSH log entries for anomalies:
{chr(10).join(events[-20:])}
Return JSON: {{"anomalous": true/false, "reason": "short explanation", "risk_score": 0-100}}
"""
result = subprocess.run(['ollama', 'run', 'llama3'], input=prompt.encode(), capture_output=True)
analysis = json.loads(result.stdout.decode().strip())
if analysis.get('risk_score', 0) > 70:
trigger_alert(analysis, events)
except Exception as e:
log_error(e)
time.sleep(30)Tuning and False Positives
The first week was noisy. The LLM flagged my own morning login from a new coffee shop Wi-Fi as anomalous because it didn’t recognize the ASN. I added a whitelist of known mobile ASNs and started including the previous hour’s log in the context window so the model sees trends, not just spikes. I also adjusted the prompt to ask for "unusual for this specific server" rather than "unusual in general," which helped ignore internet-wide scanning background radiation.
A useful trick: I log the LLM’s raw output to a separate file and review false positives weekly to refine the prompt. Over time, the signal-to-noise ratio improved dramatically. Now I get maybe one actionable alert per day, usually a slow brute-force from a residential IP that fail2ban misses because it stays under 3 attempts per 10 minutes.
Alerting and Response
When the risk score crosses 70, the script sends a message to my Mattermost channel with the log snippet, the LLM’s reasoning, and a one-click button to run iptables -I INPUT -s <ip> -j DROP. For high-confidence cases (score >90), I auto-add the IP to a temporary blocklist that expires after 4 hours unless I manually extend it. This keeps the response proportional—I’m not locking out my own typo-prone mornings, but I am stopping credential stuffing before it gets lucky.
Limitations and Next Steps
This isn’t a silver bullet. LLMs can be slow if you’re streaming high-volume logs, and they occasionally hallucinate reasons. I keep the model local for latency and privacy; sending auth logs to a third-party API is a non-starter. For now, I treat the LLM as a sophisticated anomaly scorer that feeds into my existing alerting pipeline, not a replacement for it.
If you’re running SSH accessible from the internet and want to get ahead of attackers instead of just blocking them after the fact, give this a try. Start small: pipe a few hours of logs into a local model and see what it flags as weird. You might be surprised what "normal" really looks like in your environment.
As I mentioned before in my post about AI CLI tools, the goal isn’t to replace sysadmin judgment but to augment it with pattern recognition that scales beyond what grep and awk can do easily.
Quick Checklist
- Stream auth logs to a named pipe via rsyslog
- Window events (5-15 minutes) for context
- Prompt LLM for anomaly score and reasoning
- Set actionable thresholds (e.g., >70 for alert, >90 for auto-block)
- Review false positives weekly to tune prompt and whitelist
- Keep response proportional and reversible