I Gave My Firewall a Brain: Autonomous Breach Detection on an Observability Stack I Already Had

Posted Jul 2, 2026

By Byrn Baker

29 min read

TL;DR: My pfSense firewall was great at blocking traffic and terrible at telling me when something slipped through. So I built detection. Not with a new bespoke agent like I tried back in Part 5, but by extending the observability stack I already had. Firewall logs land in Loki, Loki’s built-in ruler evaluates security queries against measured baselines, alerts flow through the one Alertmanager front door, and a webhook hands them to NetClaw for autonomous investigation. Almost no new infrastructure. This post covers the foundation, the whole detection journey, the gotchas, and why this approach beats the one I built before.

The gap in a home firewall

My pfSense firewall blocks bad traffic all day long. But blocking is only half the job. If something actually got through, if a device on my network got popped and started quietly phoning home, my firewall would never tell me. It’s great at stopping the traffic it recognizes and silent about the traffic it doesn’t.

Those are two different jobs. Prevention is the bouncer at the door. Detection is the security camera watching what happens after someone gets inside. I had the bouncer. I had no cameras.

Here’s the thing about the cameras: on most consumer gear, detection is a product you rent. Buy an eero, a Firewall, or one of the mesh-wifi systems from the big hardware brands, and the “advanced security” that flags a compromised device phoning out to a known-bad host is a paid subscription tier. It almost always ships your traffic metadata to the vendor’s cloud to do the analysis, and it stops working the day you stop paying.

What I ended up building does the same job with none of that. It’s free, it runs entirely on open-source projects, and it never phones a vendor cloud. The analysis is at least as good as what a basic paid tier gives you, and the whole thing is light enough to run off a Raspberry Pi. That’s the part worth the write-up: you can have real breach detection on your own network without renting it.

Before I show you how, I need to talk about the last time I tried to solve this, because the contrast is the whole point.

What I tried before (and why I’m doing it differently now)

If you’ve followed the Convergence series, you know I already built a firewall-defense system back in Parts 4 and 5. It was a lot of engineering, and it worked. Here’s the honest post-mortem.

Part 4 enriched the top blocked IPs with threat intelligence (AbuseIPDB, GreyNoise, OTX, IPInfo), computed a composite score, and generated a plain-English narrative. Part 5 closed the loop with a purpose-built automation agent: a FastAPI service running a polling loop every ten minutes, filtering and deduplicating high-risk IPs, asking Claude Haiku to propose one safe, reversible pfSense block, gating anything ambiguous behind a Discord approval flow, executing through a three-path waterfall (REST API → XML-RPC → SSH), and recording every decision to an immutable git audit trail.

I’m still proud of that code. But look at what it actually was: thousands of lines of bespoke Python that did exactly one thing, block bad inbound IPs on pfSense. It had its own polling loop, its own dedup layer, its own rate limiter, its own model prompt, its own notification path, its own execution logic. Every one of those was a thing I had to build, debug, and maintain. And it was narrow: it couldn’t investigate why something was happening, correlate across devices, or reason about a threat I hadn’t pre-scored. It reacted to a number crossing a threshold.

Around Phase 9 I said the quiet part out loud: I was building a smaller, worse version of a tool that was already sitting in my server stack doing nothing: NetClaw, a CCIE-level autonomous network engineer. Part 10 deleted 3,800 lines of my custom agents and replaced them with a handful of NetClaw skills.

So this time, the design brief is different:

Don’t build a detector-and-responder from scratch. Build detection as config-as-code on the stack I already run, and hand the response to a general autonomous agent that can actually investigate.

Here’s the shift in one table:

	The Part 5 approach	This approach
Detection logic	Bespoke Python polling loop	Loki ruler rules (YAML, config-as-code)
What triggers it	Threat-intel composite score	Log patterns crossing measured baselines
The “brain”	Claude Haiku, one bounded prompt, one action	NetClaw, a general multi-turn agent that enriches and investigates
Notification path	Custom Discord bot + webhook	The stack’s existing Alertmanager → one webhook
New code to maintain	~thousands of lines, single-purpose	A YAML rules file and two dashboards
What it can do	Block a known-bad inbound IP	Investigate any alert with the right tools for the device

The old way treated the LLM like a calculator: feed it a scored IP, get back a block decision. The new way treats detection and response as separate concerns. The stack detects and delivers; a real agent decides and acts. That separation is what makes the whole thing simpler and more capable at the same time.

Now let’s build it.

The foundation: the stack this all rides on

Everything below assumes an observability stack that already exists. It’s worth a quick tour, because the entire detection design is “reuse what’s here,” and you can’t reuse what you don’t understand. (If you want the full deployment reference, every container, port, and memory limit, that lives in its own doc. This is the operator’s-eye view.)

It’s one Ubuntu box, home-obs-stack, running ten containers under Docker Compose. Three kinds of telemetry, one place to see them:

Signal	Example question it answers
Metrics	Is the AI box’s GPU overheating? Is an interface saturated? Is the firewall reachable?
Logs	What did the firewall block? What DNS did a device ask for? Did anyone change config?
Flows	How much traffic is the firewall exporting? Is NetFlow healthy?

The pieces that matter for this post:

OTEL Collector: receives syslog from the network on UDP 1514, enriches it (more on that in a second), and fans it out.
Loki: short-term log store (14 days), fast to query, and home to a built-in alerting engine called the ruler that we’re about to switch on.
VictoriaLogs: long-term log archive (365 days) for forensics.
Prometheus: scrapes metrics and evaluates metric-based alert rules.
Alertmanager: the single front door for all alerts, already wired to NetClaw.
Grafana: the single pane of glass, reading every data source.

Two design principles from the stack are load-bearing here:

One alert front door. Both alert engines, Prometheus for metrics and Loki for logs, feed a single Alertmanager, which POSTs to a single NetClaw webhook. No parallel notification paths to drift out of sync. This is the exact opposite of Part 5, where I stood up my own Discord bot as a separate path.

Config-as-code. Datasources, dashboards, scrape jobs, and alert rules are all files in git. Dashboards reload in ~10 seconds, Loki rules in ~60, and nothing needs a rebuild. That property is what lets me add a whole detection layer by dropping in a couple of files.

And there’s a hard memory budget. Every container has a mem_limit and the total is capped so the box never over-commits. That constraint shows up later in a real way when I have to choose between a bigger query and a leaner one.

Step 1: Understand how the data actually flows

Here’s the shape of the system I was working with. Read it top to bottom. It’s the whole game:

                 pfSense firewall (192.168.3.1)
                          │
     ┌────────────────────┼─────────────────────┐
     │ syslog (1514)       │ SNMP (161)           │ NetFlow (2055)
     ▼                     ▼                      ▼
  OTEL Collector      OTEL Collector           goflow2
     │ adds label          │                      │
     │ device_name=pfsense │                      ▼
     ▼                     ▼                  Prometheus
   Loki  ───────────▶ VictoriaMetrics             │
  (14 days)            (365 days)                  │
     │                                             │
     ▼ also copies to                              │
  VictoriaLogs (365 days)                          │
     │                                             │
     └──────────────▶ Grafana ◀────────────────────┘
                  (single pane of glass)

The key realization: pfSense was already shipping its logs here. The OTEL Collector was listening for syslog on UDP port 1514, and pfSense was configured to send to it. Even better, the collector was tagging every log line from the firewall with a label, device_name="pfsense", so I could filter to just the firewall’s logs.

Here’s the piece of the OTEL config that does that enrichment. It looks at the source IP of each incoming syslog packet and stamps a friendly name on it:

        
      
# otel-collector/otel-config.yaml
processors:
  transform/syslog_device:
    log_statements:
      - context: log
        statements:
          - set(attributes["device_name"], "pfsense") where attributes["net.peer.ip"] == "192.168.3.1"
          - set(attributes["device_ip"], "192.168.3.1")  where attributes["net.peer.ip"] == "192.168.3.1"

That one label is the hook everything else hangs on. It’s also the same enrichment pattern the whole series leans on. Raw telemetry is anonymous numbers until you give it identity. In Part 5 I had to fetch that identity myself with threat-intel lookups. Here it’s already stamped on at ingest, for free.

Step 2: The two gotchas that reshaped everything

I had drafted a set of generic dashboards ahead of time, assuming a plain Loki setup. Reality had other plans. Two things tripped me up, and both are worth knowing if you build this yourself.

Gotcha #1: The logs are wrapped in JSON

I expected raw syslog lines. What I actually found when I queried Loki was this:

        
      
{"body":"<134>Jul  2 11:31:58 filterlog[21808]: 0,6,,1000000103,lagg0.100,match,block,in,4,0x0,,64,...,192.168.100.191,192.168.100.255,52585,32412,29","attributes":{"net.peer.ip":"192.168.3.1"}}

The actual firewall log is a string inside a body field of a JSON object. That changes how you query. A simple keyword filter still works because the word “block” appears in the raw text:

{device_name="pfsense"} |= "block"

But the moment you want to extract a field like the source IP, you have to unwrap the JSON first, then pattern-match:

{device_name="pfsense"} |= "filterlog" |= "block"
  | json
  | line_format "{{.body}}"
  | regexp "(?P<src>[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+),[0-9]+[.][0-9]+[.][0-9]+[.][0-9]+,[0-9]+,[0-9]+,"

Read that as: “take the JSON, pull out the body text, then grab the first IP that’s followed by another IP and two port numbers.” That’s the source IP of a blocked connection.

Gotcha #2: There was no Zeek data

My original plan leaned heavily on Zeek (a network security monitor) for deep DNS and SSL analysis. Turns out Zeek was installed on the pfSense box but its logs were never shipped into the stack. Rather than pretend that data existed, I rebuilt around what was actually flowing: Unbound’s DNS logs, which pfSense sends via syslog.

And those turned out to be gold. Here’s a real DNS reply log:

unbound[92343]: [92343:3] reply: 192.168.100.42 nrdp.nccp.netflix.com. AAAA IN NOERROR 0.000000 1 116

That tells me which device asked for which domain and what the answer was. That’s everything I need to spot DNS-based attacks. No Zeek required.

Lesson two: build for the data you have, not the data you wish you had. This was the exact same lesson the BGP observability work taught me a few parts back. I wanted BMP everywhere and had to build around SNMP where BMP wasn’t available. Reality keeps teaching me the same thing.

Step 3: Test every query before trusting it

I didn’t want to ship dashboards full of panels that silently show “No Data.” So before writing a single dashboard, I hammered each query against the live Loki API and confirmed real numbers came back:

        
      
# Does the block query actually return anything?
curl -sG "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode 'query=sum(count_over_time({device_name="pfsense"} |= "block" [5m]))'
# → "2778"  ✓ real data

This habit paid off immediately. When I tested “top blocked source IPs,” it exploded:

maximum of series (500) reached for a single query

That’s Loki telling me my query touched too many unique values. I’ll come back to how I fixed that. The point is I found it before a user did, because I tested first.

Step 4: Build the pane of glass

With validated queries, I built two dashboards and dropped them straight into the stack’s auto-provisioning folder:

grafana/provisioning/dashboards/pfsense-security-overview.json
grafana/provisioning/dashboards/pfsense-dns-security.json

Grafana watches that folder and reloads within about 10 seconds, no restart, no clicking around the UI. I confirmed they loaded:

        
      
curl -s "http://localhost:3000/api/search?query=pfSense"
# pfsense-security-overview | pfSense — Security Operations Overview
# pfsense-dns-security      | pfSense — DNS Security Monitor

The Security Overview answers “is anything on fire right now?” with blocked connections, auth failures, config changes, top offenders, blocks by VLAN, and a live event stream. The DNS Security Monitor goes deep on the sneaky stuff: NXDOMAIN spikes (a sign of malware generating random domains), and DNS tunneling (data smuggled inside DNS queries).

Every panel points at the existing data sources by their real IDs. I copied these from the stack’s own datasource config rather than inventing new ones:

        
      
"datasource": { "type": "loki", "uid": "loki" }

Step 5: Now the interesting part, alerting

Dashboards are great, but I’m not going to stare at them 24/7. I needed the system to tap me on the shoulder when something looked wrong. This is where I had to make an architectural decision, and it’s exactly where the new approach parts ways with Part 5.

In Part 5, “alerting” meant my automation agent polled a threat-intel API, scored IPs, and pushed embeds to a Discord channel I built. It was a whole parallel notification system. This time, the stack already had Alertmanager running, and it was already wired to NetClaw at 192.168.3.252:8099. NetClaw is an autonomous responder. It takes an alert, looks up the device in a source-of-truth database (Nautobot), and investigates using the right tools for that device. That’s a huge deal: an alert doesn’t just ping me, it kicks off an automatic investigation, and I didn’t have to write the investigator.

So the goal became clear: get my security alerts into Alertmanager, so they flow to NetClaw like everything else.

Why webhooks?

Here’s the thing about connecting two independent systems like Alertmanager and NetClaw: you want them loosely coupled. Alertmanager shouldn’t need to know how NetClaw works, and NetClaw shouldn’t need to poll Alertmanager constantly asking “anything new? anything new?”

A webhook solves this elegantly. It’s just an HTTP POST that one system sends to another when something happens. “Hey, here’s an alert, do your thing.” Fire-and-forget. The sender doesn’t wait around; the receiver decides what to do. It’s the difference between someone calling you when there’s news versus you refreshing a page all day.

This is exactly how Alertmanager was already talking to NetClaw:

        
      
# alertmanager/alertmanager.yml
route:
  receiver: 'netclaw'
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: 'netclaw'
    webhook_configs:
      - url: 'http://192.168.3.252:8099/webhook'
        send_resolved: true

That send_resolved: true is a nice touch. NetClaw gets told both when a problem starts and when it clears.

The routing problem

Here was the catch. My alerts are based on logs (firewall blocks, DNS queries). But Alertmanager is fed by Prometheus, which handles metrics. Prometheus can’t read Loki’s logs.

I had two options:

Use Grafana’s built-in alerting to watch Loki, and point it at a webhook.
Use Loki’s own ruler, a built-in alerting engine that evaluates log queries and pushes alerts straight to Alertmanager.

I picked option 2, and here’s the reasoning: the whole stack already treated Alertmanager as the single front door for alerts, and Alertmanager already knew how to reach NetClaw. If I used the Loki ruler, my log alerts would land in that same front door and flow to NetClaw automatically, with no separate notification path and no duplicate config. It also keeps alerts as config-as-code (YAML files in git), matching how the Prometheus alerts were already written.

Loki ruler  ──▶  Alertmanager  ──▶  NetClaw webhook  ──▶  autonomous investigation
 (reads logs)    (already there)     (already wired)

One path. One place to manage alerts. Everything converges. Compare that to Part 5’s diagram, which had its own polling service, its own Redis dedup, its own Discord bot, and its own execution waterfall bolted on the side. The new picture is smaller, and it does more.

Step 6: Turn on the Loki ruler

Loki has a ruler built in, but it wasn’t switched on. Turning it on meant two edits.

First, add a ruler block to Loki’s config telling it where to find rules and where to send alerts:

        
      
# loki/loki-config.yml
ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /tmp/loki-rule-eval
  alertmanager_url: http://alertmanager:9093
  enable_alertmanager_v2: true
  evaluation_interval: 1m
  poll_interval: 1m
  ring:
    kvstore:
      store: inmemory

Notice alertmanager_url: http://alertmanager:9093. Because both containers share a Docker network, Loki can reach Alertmanager just by its name. And evaluation_interval: 1m means the ruler re-checks every rule once a minute.

Second, mount a folder from the host into the Loki container so I can manage rule files as plain YAML:

        
      
# docker-compose.yml (loki service)
volumes:
  - loki_data:/loki
  - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro
  - ./loki/rules:/etc/loki/rules:ro     # <-- added this

One quirk worth knowing: since Loki runs without authentication, it uses a default tenant literally named fake. So rule files have to live in a fake subfolder: loki/rules/fake/.

Then recreate just the Loki container to pick up the new mount:

docker compose up -d loki

I flagged this to myself as a medium-risk action. It briefly restarts a shared service, so there’s a few seconds where syslog isn’t being ingested. But the data volume persists, so nothing is lost, and it’s fully reversible. I verified Loki came back healthy before moving on.

Step 7: Write alerts that fire on real threats, not noise

It’s easy to write an alert. It’s hard to write one that doesn’t cry wolf every ten minutes. An alert that fires constantly gets ignored, and worse here, every false alarm would kick off a pointless NetClaw investigation.

This is also where I get to reuse a hard lesson from Part 5. Back then, my rate limiter existed largely to protect me from my own noisy triggers, a safety cap so the agent couldn’t go on a blocking spree overnight. The better fix, it turns out, isn’t a cap on the response; it’s a quiet trigger in the first place. So before setting any threshold, I measured the baseline. What does normal look like?

        
      
# How many auth failures in a normal 5 minutes?
curl -sG "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode 'query=sum(count_over_time({device_name="pfsense"} |~ "(?i)(authentication failure|failed password)" [5m]))'
# → 0

# How many blocked TCP hits does a single WAN source normally rack up?
# → max was 6 in 5 minutes

Now I could set thresholds with real headroom. A few examples from the rules file (loki/rules/fake/pfsense-security.yml):

Brute force. Normal is 0 failures, so 10 is clearly an attack:

        
      
- alert: PfSenseBruteForceAuth
  expr: |
    sum(count_over_time({device_name="pfsense"} |~ "(?i)(authentication failure|failed password|login failed|sshguard)" [5m])) > 10
  for: 2m
  labels:
    severity: critical
    category: security
    device_name: pfsense
    device_ip: 192.168.3.1
    source: loki
  annotations:
    summary: "Brute force authentication against pfSense"
    description: "More than 10 failed auth attempts in 5 minutes. Baseline is 0."

DNS tunneling. This one took real thought. My first instinct was “alert on long domain names.” But when I checked, plenty of legitimate domains are long:

87 chars: wxt-registration-ingressgateway-ds.wxtreg-prod-achm-registration2.prod.infra.webex.com
80 chars: dualstack.srta-bpr-dedup-ingress-prod-560-958937635.us-east-1.elb.amazonaws.com

Those are just Webex and AWS. If I alerted on total length, I’d get false alarms all day. The insight: real DNS tunneling crams data into a single chunk (one label between the dots), because that’s where the encoded payload goes. So I measured how often a single label hits 45+ characters in normal traffic:

# single no-dot label of 45+ chars in 30 minutes:
# → 0

Zero. Perfect signal. So the rule detects a single oversized label, not a long full name:

        
      
- alert: PfSenseDNSTunneling
  expr: |
    sum(count_over_time({device_name="pfsense"} |= "unbound" |= "reply"
      | json | line_format "{{.body}}"
      | regexp "reply: [0-9.]+ (?P<qname>[^ ]+) "
      | qname =~ "[^ .]{45,}" [10m])) > 2
  for: 0m
  labels:
    severity: critical
    category: security
    device_name: pfsense
    device_ip: 192.168.3.1
    source: loki

Here’s the full set I ended up with:

Alert	Fires when	Normal baseline	Catches
PfSenseBruteForceAuth	>10 auth fails / 5m	0	Password guessing
PfSenseDNSTunneling	>2 long single-labels / 10m	0	Data exfiltration via DNS
PfSenseWANPortScan	>50 blocks / source / 5m	max 6	Someone scanning your WAN
PfSenseDGASpike	>500 NXDOMAIN / 10m	~43	Malware finding its C2 server
PfSenseDNSBLBeaconing	>150 DNSBL blocks / 10m	low	Malware calling home
PfSenseInternalHostExcessiveBlocks	>1500 blocks / source / 10m	peak ~640	A compromised inside device
PfSenseConfigChange	any config change / 5m	0	Unauthorized firewall edits

Notice every alert carries device_name: pfsense and device_ip: 192.168.3.1. Those labels are what let NetClaw look up the device and know which tools to use. The alert tells the responder exactly what it’s looking at, not just that something’s wrong somewhere. That’s the same enrichment idea from Step 2 doing double duty: it identified the log at ingest, and now it points the investigation at a specific device.

Look at what these seven rules replaced. In Part 5, “detect a bad actor” meant a threat-intel pipeline (four external APIs), a composite scoring function, and a filtering/dedup layer. Here, “detect a bad actor” is a LogQL expression with a number on the end of it, and the number comes from a measurement I can rerun any time. Detection went from a service to a sentence.

Step 8: The bugs I hit deploying the rules

Real work is never clean. Two things broke, and both are instructive.

Bug 1: A missing time window

When the ruler first loaded my rules, it complained:

could not parse expression for alert 'PfSenseWANPortScan':
parse error: unexpected )

I’d written count_over_time(...) but forgotten the [5m] time window inside it. count_over_time literally means “count over this much time.” Without the window, it’s a sentence with no ending. Fixed by adding it back:

count_over_time({...} | regexp "..." | src != "" [5m])

Bug 2: An alert that lied on day one

After fixing the syntax, the rules loaded, and one immediately went pending:

PfSenseInternalHostExcessiveBlocks   state=pending

I’d set it to fire when an internal device generated more than 300 blocks in 10 minutes. But when I checked reality:

# top internal sources by blocks in 10m:
168.100.33     640
168.30.50      548
168.100.191    480

Several devices were normally over 300, not because they’re hacked, but because of ordinary broadcast, mDNS, and SNMP chatter getting blocked. My threshold was below the noise floor. It would have fired a false alarm at NetClaw the moment it turned on.

So I raised it to 1500, comfortably above the ~640 peak, and documented why right in the rule:

        
      
expr: |
  sum by (src) (count_over_time({device_name="pfsense"} |= "filterlog" |= "block"
    | json | line_format "{{.body}}"
    | regexp "(?P<src>192[.]168[.][0-9]+[.][0-9]+),...," | src != "" [10m])) > 1500

Lesson three: a threshold is a claim about what’s normal. Measure before you claim.

Step 9: Watch it catch something real

Once the rules settled, I checked their status through Loki’s API:

curl -s "http://localhost:3100/prometheus/api/v1/rules"

Six were quietly inactive. One was firing:

PfSenseWANPortScan   state=firing

Not a test. Not a false alarm. A real external IP was scanning my firewall:

# top WAN scanners in the last 5 minutes:
132.195.71      64 blocks/5m    ← the scanner
52.24.181       6 blocks/5m    ← background noise
94.146.63       6 blocks/5m    ← background noise

One source at 64, everything else at 6. My threshold of 50 sliced right between them. That’s a well-tuned alert doing exactly its job: flagging the genuine scanner while ignoring the internet’s constant background hum.

And I confirmed it made the full journey to NetClaw:

        
curl -s "http://localhost:9093/api/v2/alerts"
# PfSenseWANPortScan  severity=warning  src=66.132.195.71  state=active

To be precise about what I verified: the alert traveled from a log line on the firewall, to a rule evaluation in Loki, to Alertmanager, to a webhook POST that NetClaw accepted ({"status":"accepted"}). That’s the delivery path, confirmed end to end.

What I did not verify is what NetClaw did next. NetClaw is designed to investigate autonomously, enriching the alert with device context and reasoning over it, but I only observed the hand-off, not the outcome. When I later checked pfSense for side-effects (a new block alias or rule for 66.132.195.71), there were none. That could mean NetClaw investigated and decided no action was warranted (a single scanner hitting a closed port is low-stakes; the firewall already blocked it), or it could mean the investigation is opaque from the firewall’s side. Either way, the honest claim is: the alert was delivered and accepted. The response is NetClaw’s business, and I didn’t confirm it from here.

This is a real philosophical difference from Part 5, and it’s worth sitting with. Part 5’s agent was deterministic: propose one action, gate it, execute it, verify it, roll back if it failed. I could tell you exactly what it did to pfSense because I wrote every branch. NetClaw is autonomous: I hand it a well-labeled alert and trust it to reason. I gave up some of that end-to-end certainty in exchange for an investigator that can handle alerts I never anticipated. For a home lab, that’s the right trade. For something with a bigger blast radius, you’d want the same audit trail I built in Part 5, and NetClaw has its own GAIT-style session logging for exactly that, which is the subject of the next post.

For the record, that scanner was hitting port 8443, the alternate-HTTPS port, which is textbook recon for an exposed web admin panel. The firewall blocked every attempt.

Step 10: Trust but verify the webhook

When I checked Alertmanager’s delivery stats, I saw a wall of failures:

        
      
curl -s "http://localhost:9093/metrics" | grep alertmanager_notifications
# alertmanager_notifications_total{integration="webhook"}         1864
# alertmanager_notifications_failed_total{...reason="other"}       1859

1859 failed deliveries! My heart sank. Was NetClaw not receiving anything? But a quick reachability test told a different story:

        
docker exec alertmanager sh -c 'nc -zv 192.168.3.252 8099'
# 192.168.3.252 (192.168.3.252:8099) open

The port was open. And a manual test POST to the webhook got a clean acceptance:

        
      
{"status":"accepted","alerts_received":1,"timestamp":"2026-07-02T18:09:23Z"}

So what were those 1859 failures? History. They were all from before NetClaw was started. Alertmanager had been trying to deliver to a listener that wasn’t up yet, and those failures accumulated in the counter. Once NetClaw came online, deliveries succeeded. The counter just never forgets the past.

Lesson four: a scary-looking metric isn’t always a current problem. Check the live path before you panic.

Step 11: Fix the panel that broke

Remember that “maximum of series (500)” error from Step 4? Here’s how I actually solved it, and it’s a decision the stack’s memory budget made for me.

The “Top Blocked Source IPs” panel was set to look back one hour. Loki refuses to process more than 500 distinct series in one query, and I measured the reality:

# unique blocked source IPs by time window:
5m:   65 sources
15m:  196 sources
30m:  338 sources
1h:   500+  ← boom

Over an hour, mostly thanks to internet scanners, there were more than 500 unique IPs. Two ways to fix it:

Raise Loki’s 500-series limit globally.
Shrink the panel’s time window.

I chose to shrink the window to 15 minutes. Why not just raise the limit? Because this Loki runs with a 1 GB memory cap on a modest VPS. That’s the memory budget from the foundation section, enforced by a script that fails CI if the total creeps over the ceiling. Raising the series limit means more memory per query, which eats into that budget. A 15-minute view (about 200 sources, comfortable headroom) is genuinely more useful for a live security screen anyway. You want to know who’s hitting you right now, not an hour ago.

        
"expr": "topk(10, sum by (src) (count_over_time({device_name=\"pfsense\"} |= \"block\" ... [15m])))",
"title": "Top Blocked Source IPs (15m)"

Lesson five: the “right” fix depends on your constraints. On a memory-tight box, the cheaper query beats the bigger limit.

The whole thing, in one breath

Here’s what happens now when something sketchy occurs on my network:

A device queries a shady domain. Unbound logs it; pfBlockerNG blocks it.
pfSense ships the logs via syslog to the OTEL Collector, which stamps them device_name=pfsense.
The logs land in Loki (searchable for 14 days) and VictoriaLogs (archived for a year).
Grafana shows it live on the dashboards.
If it crosses a threshold, the Loki ruler fires an alert carrying the device’s identity.
Alertmanager receives the alert and POSTs it via webhook to NetClaw.
NetClaw accepts the hand-off and, by design, looks up the device and investigates. (That last step is NetClaw’s job; the observability stack’s responsibility ends at reliably delivering the alert with enough context to act on.)

I went from “I’d never know” to “my firewall’s incidents get handed to an autonomous responder the instant they’re detected.” And the best part: almost none of it required new infrastructure. It was mostly about understanding what I already had, adapting to the real data, and wiring the last mile with a humble webhook.

Which brings me back to where I started. Part 5’s version of this was thousands of lines of my own code that could block a bad IP. This version is a YAML rules file, two dashboards, and a config block that turns on a ruler that was there all along, feeding an agent that can do far more than block an IP. The best code I wrote for this project was the code I got to delete.

The five lessons, collected

Never assume a greenfield. Look at what’s running before you build.
Build for the data you have, not the data you wish you had. (RIP, my Zeek dashboards.)
A threshold is a claim about “normal.” Measure the baseline before you set it.
A scary metric isn’t always a live problem. Check the actual path before panicking.
The right fix depends on your constraints. On a small box, the lean query beats the big limit.

And the meta-lesson tying this to the rest of the series: converge on what you already run. A general observability stack plus a general autonomous agent beat a pile of single-purpose glue code, every time.

Appendix: the files I touched

observability-stack/
├── docker-compose.yml                         # added the rules volume mount to loki
├── loki/
│   ├── loki-config.yml                         # added the ruler block
│   └── rules/fake/pfsense-security.yml         # NEW — the 7 security alert rules
├── alertmanager/alertmanager.yml               # (already routed to NetClaw — unchanged)
├── otel-collector/otel-config.yaml             # (already enriched pfSense logs — unchanged)
└── grafana/provisioning/dashboards/
    ├── pfsense-security-overview.json          # NEW — the operations pane of glass
    └── pfsense-dns-security.json               # NEW — the DNS deep-dive

Handy verification commands

        
      
# Are my alert rules loaded and healthy?
curl -s http://localhost:3100/prometheus/api/v1/rules

# What's firing right now?
curl -s http://localhost:3100/prometheus/api/v1/alerts

# Did it reach Alertmanager?
curl -s http://localhost:9093/api/v2/alerts

# Is the webhook to NetClaw actually delivering?
curl -s http://localhost:9093/metrics | grep alertmanager_notifications

To add or tweak an alert, edit the YAML in loki/rules/fake/. The ruler reloads within a minute, no restart needed. To change a dashboard, edit the JSON in grafana/provisioning/dashboards/. Grafana reloads within ten seconds. Config-as-code all the way down.

In the next post I follow the alert across the fence to NetClaw’s side and answer the question that decides whether any of this is affordable to run: how do you keep an autonomous agent’s token bill from getting silly?

Stay curious, measure your baselines, and give your firewall a brain.

Convergence

observability OpenTelemetry Grafana Loki VictoriaMetrics Alertmanager pfSense security network-automation netclaw nautobot Docker infrastructure

This post is licensed under CC BY 4.0 by the author.