Building Convergence – A Journey from Network Observability to AI-Driven Automation Part 15: When Routes Flap — AI-Driven Root Cause Analysis
In Part 14, I made NetClaw a BGP peer. I injected a prefix, withdrew it, watched the control plane react. Great theater. Wrong monitoring plane.
The Grafana dashboard I’d built for “route stability” stayed empty. Jitter panels showed nothing. Syslog was in Loki but the BGP board couldn’t find it. bgp_rib_size tracked NetClaw’s synthetic speaker — not RR1’s RIB. I did what any of us would do: closed the tab and SSH’d to the router.
Part 15 is the fix I actually wanted that day — router-native observability, the same telemetry model you’d see at an ISP, validated in my ContainerLab, wired into NetClaw skills, and documented spec-first with GitHub spec-kit.
Spec: specs/031-bgp-route-observability — Phases 1–6 complete (June 2026)
The One-Sentence Version
I collect BGP truth from the routers (BMP + gNMI + SNMP), normalize it as netclaw_* metrics, correlate with syslog and IP SLA, and let the agent investigate via PromQL → Loki → pyATS/gNMI — Protocol MCP stays for injection demos only.
I built this for the moment BGP looks fine in one place and wrong everywhere else — a flap on a PE uplink, prefixes draining at the RR, jitter climbing on a probe path while the session still says Established. If you already reach for show ip bgp summary before you trust a dashboard, you’re the reader I had in mind. I’m not trying to turn you into a Grafana person. I’m trying to make the metrics agree with the CLI.
Concretely, I put a monitoring plane on real routers in my Nautobot Workshop lab. BMP on RR1 for per-prefix withdrawals. SNMP on Cisco PEs/RRs for peer state, prefix counts, UPDATE counters. gNMI on Arista spines for the same metric names with a source="gnmi" label. Syslog into Loki with device_name you can query without JSON archaeology. IP SLA when the path is getting ugly but BGP hasn’t caught up yet. Everything lands as netclaw_* in VictoriaMetrics; Grafana rules live in the Lab Network folder; pyATS scenarios A–F (plus a BMP inject) prove each alert fires for a reason I can explain.
It matters mid-incident and right after a change — the “did I shake anything loose?” window when you’d rather run PromQL and LogQL than open five SSH sessions. Part 14’s Protocol MCP speaker is still useful, but only as Scenario D: a scripted inject/withdraw loop for demos, not something I’d point a production NOC at.
In the stack, this sits between golden config and agent triage. Nautobot config context renders BMP, SNMP, syslog, and IP SLA onto the devices. bgp-snmp-exporter, bgp-bmp-consumer, and bgp-gnmi-exporter feed VictoriaMetrics and Loki on clab-mgmt. Grafana evaluates; the skills (bgp-route-stability-watch, lab-alert-triage) run the same queries I document in failure-scenarios.md. I can prove the whole chain from a shell — bash scripts/run-all-scenarios.sh --all — before I open a single panel.
Short version: stop watching NetClaw’s BGP speaker; watch the network. Routers →
netclaw_*+ Loki → Grafana alerts → agent skills → pyATS drill-down. Scenarios A–F + BMP inject are the receipt. Protocol MCP is Scenario D only.
Part 14 was teaching the AI to ride a motorcycle on the highway. Part 15 puts cameras on the overpasses. Both are fun. Only one tells you why traffic slowed down.
| Part 14 (Protocol MCP) | Part 15 (router-native) | |
|---|---|---|
| What you’re seeing | NetClaw’s synthetic session (AS 65099) | RR1 RIB, PE peer tables, spine gNMI |
| When I’d use it | Demo flap, training inject | Incident, post-change, agent RCA |
| Per-prefix truth | While MCP is running | BMP netclaw_bgp_prefix_* from RR1 |
| Physical layer | Not in the picture | interface_status + Loki UPDOWN/ADJCHANGE |
| How I prove it | run-scenario-d.sh |
run-all-scenarios.sh --all |
Why Protocol MCP Failed as the Metrics Plane
Part 14’s Protocol MCP speaker (AS 65099) is real BGP. The UPDATEs are real. But the observability contract was wrong:
| What I hoped | What happened |
|---|---|
| Continuous withdrawal metrics | bgp_route_* only when the MCP process runs |
| Router RIB perspective | NetClaw’s local RIB; RR1 peer often Idle after gateway restarts |
| Dashboard always populated | VictoriaMetrics empty until someone re-injects |
| Per-prefix flap truth | Only during manual inject/withdraw loops |
Protocol MCP is still valuable for Scenario D — a controlled flap you can run from a script for training. It is not the monitoring source of truth. Production NOCs monitor the routers that carry traffic.
The Three Telemetry Planes
1
2
3
4
5
6
7
8
9
10
┌──────────────────────────────────────────────────────────────────────────┐
│ PLANE 1: BMP "Which prefix was withdrawn, from which peer?" │
│ PLANE 2: gNMI/SNMP "Is the session up? How many prefixes? UPDATE rate?" │
│ PLANE 3: Syslog "What did the router *say* happened?" │
└───────────────────────────────┬──────────────────────────────────────────┘
▼
netclaw_* metrics (VictoriaMetrics)
device_name streams (Loki)
▼
Grafana alerts → NetClaw skills → pyATS / gNMI CLI
Plane 1: BMP — the prefix gossip channel
A BGP UPDATE can advertise (NLRI) or withdraw routes. The RIB changes. Traditional SNMP gives you bgpPeerInUpdates — a counter that says “something happened” but not which prefix.
BMP (BGP Monitoring Protocol) streams pre-policy views to a collector. That’s how you get per-prefix withdrawal counts at scale.
Lab path:
1
2
3
4
5
6
7
8
9
10
RR1 (CSR1000v) ──BMP TCP──> gobmp (192.168.220.205:5000)
│
▼
Redpanda (Kafka)
│
▼
bgp-bmp-consumer (:9100)
│
▼
VictoriaMetrics (netclaw_bgp_prefix_*)
Checkpoint query (no Grafana required):
1
2
3
curl -sf 'http://localhost:8428/api/v1/query' \
--data-urlencode 'query=netclaw_bgp_peer_prefixes_received{device_name="rr1",neighbor="100.0.254.13"}'
# expect value 4 in the lab
Plane 2: SNMP + gNMI — peer state and UPDATE counters
| Platform | Adapter | What you get |
|---|---|---|
| Cisco CSR / IOS-XE | SNMP (bgp_snmp_exporter.py) |
netclaw_bgp_peer_state, prefix counts, In/Out UPDATE totals |
| Arista cEOS | gNMI OpenConfig subscribe | Same metric names, source="gnmi" label |
netclaw_bgp_peer_state == 6 maps to BGP4-MIB established. Same number as show ip bgp summary “Established” — I’m just polling/streaming it.
The exporter polls SNMP every 60s and sets gauge values from MIB counters:
1
2
3
# observability/exporters/bgp_snmp_exporter.py (simplified)
peer_in_updates.labels(**labels).set(data["in_updates"])
peer_prefixes.labels(...).set(row["count"])
Dashboard lesson I learned the hard way: bgpPeerInUpdates is not “withdrawals.” It’s every UPDATE message. PE1 has two BGP neighbors → two graph lines. A line showing “9” is cumulative UPDATE activity on that peer, not “9 routes withdrawn.” I split the Grafana panels so SNMP peer UPDATEs and BMP prefix withdrawals don’t share a misleading title.
Plane 3: Syslog — the story your TAC engineer tells
Routers already log %BGP-5-ADJCHANGE and %LINEPROTO-5-UPDOWN. I ship syslog to the OTEL Collector → Loki with device_name as a label (not buried in JSON you have to parse in every query).
PE routers need VRF-aware logging when management rides clab-mgmt:
1
2
{# ios/observability.j2 — PE syslog uses context VRF #}
logging host vrf transport port
LogQL checkpoint:
1
2
3
4
5
curl -sf -G 'http://localhost:3100/loki/api/v1/query_range' \
--data-urlencode 'query={device_name="pe1"} |~ "UPDOWN"' \
--data-urlencode "start=$(($(date +%s)-3600))000000000" \
--data-urlencode "end=$(date +%s)000000000" \
--data-urlencode 'limit=5'
How I Wired the Lab (Golden Config → Router → Metric)
Same pattern as Part 13: Nautobot config context → Jinja → device config → exporter → VM.
1. Config context
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Nautobot-Workshop-Datasource/config_contexts/observability.yml
observability:
mgmt_vrf: clab-mgmt # PE syslog/SNMP VRF
snmp:
community: public
access: ro
syslog:
host: 192.168.220.200
port: 1514
bmp:
enabled: true
host: 192.168.220.205 # gobmp on clab-mgmt
port: 5000
gnmi:
enabled: true
port: 6030
Why mgmt_vrf: clab-mgmt? On ContainerLab, PE management interfaces live in clab-mgmt. Syslog without VRF goes nowhere useful. RR1’s BMP socket, conversely, must use the global routing table on CSR — a BMP update-source on a VRF-forwarded interface leaves you staring at TCB 0x0 and a very sad show bmp summary.
2. BMP template (RR1 only)
Only IOS-XE CSR route reflectors — IOL doesn’t speak BMP:
1
2
{# ios/bmp.j2 — RR1 only, IOS-XE only #}
Templates exist in two repos (Ansible deploy + Golden Config intended state) — same lesson as Part 13. I pushed both Nautobot-Workshop and nautobot_workshop_golden_config_templates.
3. BMP consumer
Kafka message in, Prometheus counter out:
1
2
3
4
5
# observability/exporters/bgp-normalizer.py (simplified)
if action == "add":
prefix_announcements.labels(**labels).inc()
elif action == "del":
prefix_withdrawals.labels(**labels).inc()
Metric names you’ll query everywhere (dashboards, alerts, skills):
netclaw_bgp_prefix_announcements_totalnetclaw_bgp_prefix_withdrawals_total
Lab Environment
| Piece | Location |
|---|---|
| Topology | ~/Nautobot-Workshop/clabs/nautobot-workshop-topology.clab.yml (18 nodes) |
| Network | Docker clab-mgmt — 192.168.220.0/24 |
| pyATS testbed | netclaw/testbed/testbed.yaml |
| Python / pyATS | netclaw/.venv — always use .venv/bin/python or run-all-scenarios.sh |
| Metrics | VictoriaMetrics :8428, Grafana :3000, Loki :3100 |
Cisco CSR devices (PE/RR/P) export interface_status and netclaw_bgp_* via bgp-snmp-exporter (job=netclaw-bgp-snmp). OTEL parallel SNMP to 18 devices at once timed out on IOL — metrics went stale. Sequential polling fixed that.
Prove It Works Without Grafana
Fancy graphs are optional. Engineers trust reproducible CLI checks.
One-command demo
1
2
cd ~/netclaw
bash scripts/demo-part15-observability.sh
Automated scenario + alert validation (pyATS)
1
2
bash scripts/validate-alert-scenarios.sh # prerequisites + alert map
bash scripts/run-all-scenarios.sh --all # scenarios A–F + BMP inject
Each scenario restores the lab after itself. Grafana alerts in folder Lab Network are polled automatically.
This walks all four planes with curl + PromQL/LogQL, explains what each query means, and prints ✓/⚠ against lab baselines.
Scenario B (link flap) — interactive, uses ContainerLab if the lab is up:
1
bash scripts/demo-part15-observability.sh --scenario-b
BMP pipeline inject — synthetic Kafka withdraw (no router config change):
1
bash scripts/demo-part15-observability.sh --bmp-inject
Phase checkpoints (spec-kit gates)
1
2
3
4
5
6
bash scripts/validate-bgp-metrics.sh --phase 1 # SNMP RR1 prefixes == 4
bash scripts/validate-bgp-metrics.sh --phase 2 # IP SLA jitter + Loki labels
bash scripts/validate-bgp-metrics.sh --phase 3 # BMP live from RR1
bash scripts/validate-bgp-metrics.sh --phase 4 # gNMI west-spine01
bash scripts/validate-bgp-metrics.sh --phase 5 # Alerts + skills + baselines
bash scripts/validate-bgp-metrics.sh --phase 6 # Golden config BMP/gNMI rendered
Four-source chain (Part 13 + Part 15 together)
1
bash scripts/validate-part15-chain.sh
Demo Scenarios (What to Break, What Should Scream)
Full playbook: docs/failure-scenarios.md
| Scenario | Script | Grafana alert (validated) | Skill |
|---|---|---|---|
| A — port down | run-scenario-a.sh |
Lab Interface Down (~3 min) | lab-alert-triage |
| B — single link flap | run-scenario-b.sh |
Lab Interface Down; ALERT-007 optional | bgp-route-stability-watch |
| C — dual link loss | run-scenario-c.sh |
ALERT-001, ALERT-003 (~7 min) | bgp-route-stability-watch |
| D — synthetic flap | run-scenario-d.sh |
Protocol MCP bgp_route_* only |
synthetic-demo |
| E — rapid flaps | run-scenario-e.sh |
Lab Interface Rapid Flaps | lab-alert-triage |
| F — path loss | run-scenario-f.sh |
ALERT-006 (netem on P1 eth2, for: 3m) |
bgp-route-stability-watch |
| BMP — synthetic withdraw | run-scenarios.py --scenario bmp |
ALERT-004 (sustained Kafka inject) | bgp-route-stability-watch |
Scenario B walkthrough (validated 2026-06-06)
Run: bash scripts/run-scenario-b.sh (pyATS via .venv, testbed PE1)
PE1 is dual-homed — BGP to RR1 stays Established on Gi3. You’re testing observability, not forcing a meltdown.
What moves within ~3 minutes (60s SNMP poll + Grafana for: 2m):
| Signal | Query | Validated |
|---|---|---|
| Interface down | interface_status{device_name="pe1",interface="GigabitEthernet2",job="netclaw-bgp-snmp"} |
→ 2 |
| Transition | changes(interface_status{...,job="netclaw-bgp-snmp"}[5m]) |
> 0 |
| Syslog | {device_name="pe1"} \|~ "UPDOWN" |
%LINEPROTO-5-UPDOWN |
| Grafana | Lab Interface Down | Fires on pe1/Gi2 |
| ALERT-007 | iface change + BGP UPDATE rate | Usually absent (UPDATE rate 0) |
Scenario C (Gi2 + Gi3 down): RR peer goes Idle in SNMP ~200s; ALERT-001 fires ~2m later; ALERT-003 after links are restored and sessions re-establish. Use run-scenario-c.sh — allow 10+ minutes.
Scenario F (netem, not no ip sla responder): disabling the responder causes No connection (jitter drops to 0), not elevated jitter. Lab uses ContainerLab netem on P1 eth2 so PE1→PE2 probes see delay/jitter while interfaces stay up.
BMP inject: Kafka messages need a trailing newline for rpk topic produce; sustained inject (~180s @ 2/s) drives ALERT-004 (increase[2m] > 10, for: 1m). Exporter counter delta is the reliable gate — VictoriaMetrics rate() lags on sparse BMP scrapes.
Baselines: docs/baselines/bgp-route-stability.md
Scenario D — honest labeling
1
bash scripts/run-scenario-d.sh
This injects/withdraws 192.168.99.0/24 via Protocol MCP. Good for blog narrative. Router-native BMP on RR1 won’t mirror it unless the fabric actually carries that prefix. The skill should say “synthetic-demo” when only MCP metrics move.
After the Scenarios: Grafana, Alerts, and the Agent
Everything above is proven from a shell — curl, PromQL, pyATS, run-all-scenarios.sh. Grafana comes after that on purpose. Graphs are for humans who want to stare at a trend while coffee brews; they are not the source of truth. The source of truth is the same VictoriaMetrics series the scenarios already gate on.
The dashboard (optional, but worth fixing if you open it)
I have a BGP Route Stability & Path Quality dashboard. I only mention it because I spent time fixing panels that were lying — and you might hit the same confusion if you open Grafana before reading this.
Early on I had one panel that looked like “withdrawals” but was actually SNMP bgpPeerInUpdates — every UPDATE, not per-prefix withdraws. PE1 has two neighbors, so you get two lines; a value of “9” means nine UPDATE messages on that peer, not nine routes gone. I split the panels so SNMP UPDATE counters and BMP prefix counters never share a title again:
| Panel | Source | What it actually means |
|---|---|---|
| Peer In UPDATEs (5m) | netclaw-bgp-snmp |
All inbound UPDATEs per neighbor — ads and withdraws |
| Peer Out UPDATEs (5m) | netclaw-bgp-snmp |
Same, outbound |
| BMP Prefix Withdrawals (5m) | netclaw-bgp-bmp |
True per-prefix withdrawals from RR1 BMP |
| BMP Prefix Announcements (5m) | netclaw-bgp-bmp |
True per-prefix announcements |
Dashboards use round(increase(...[5m])) so you see whole numbers — “4 events in five minutes” — not 0.0033/s. Alerts still use rate() where the threshold is per-second. Same underlying counters, different math for the job.
I also filter on job=netclaw-bgp-snmp / job=netclaw-bgp-bmp so OTEL and the dedicated exporters don’t draw duplicate lines from the same MIB.
Alerts — the contract between metrics and action
Scenarios prove the signals move. Alerts prove something will fire when I’m not watching. I provision rules in Grafana’s Lab Network folder — seven BGP/path rules in bgp-route-stability.yaml, plus general lab rules (interface down, rapid flaps) in lab-network.yaml. Each scenario in the table earlier maps to at least one of these:
| ID | What it catches | Scenario that proves it |
|---|---|---|
| ALERT-001 | Peer not Established | C (dual link loss) |
| ALERT-002 | Prefix count drops >20% vs 1h avg | C (when prefixes drain) |
| ALERT-003 | Session re-establishment | C (after restore) |
| ALERT-004 | BMP withdrawal burst | BMP inject |
| ALERT-005 | SNMP UPDATE rate spike | B/C (noisy fabric) |
| ALERT-006 | Path jitter or RTT elevated | F (netem on P1 eth2) |
| ALERT-007 | Interface change + BGP activity | B (optional on dual-homed PE) |
I didn’t turn these on until I had baselines in bgp-route-stability.md and scenarios passing. Paging on day-one thresholds is how you train yourself to ignore alerts.
How the agent uses the same data
The skills are not magic — they’re the investigation order I’d run manually, encoded so the agent doesn’t reach for Protocol MCP’s RIB when a real peer flaps.
When something looks like route instability, bgp-route-stability-watch walks: BGP signal first (peer down? prefix drop? BMP withdrawals? UPDATE spike?), then physical layer (interface_status on the same device), then syslog in Loki (ADJCHANGE, UPDOWN), then path quality (netclaw_path_jitter_ms), then drill-down via pyATS on Cisco or gNMI on Arista. Verdict ends up as physical-layer, prefix-upstream, path-quality, or synthetic-demo if only MCP metrics moved.
lab-alert-triage is the entry point when Grafana fires — it reads the alert labels and points at the right PromQL/Loki starting points. Same queries as the dashboard; same queries as the scenarios.
Things That Lied to Me (and What I Changed)
This stack was not “configure exporters and go.” A few failures ate whole afternoons. I’m keeping them here so you know why the architecture looks the way it does — not as a troubleshooting appendix you’ll never read.
The worst early mistake was trusting bgp_rib_size from Protocol MCP as if it were RR1’s RIB. Wrong plane entirely — switched to netclaw_bgp_peer_prefixes_received from SNMP. Jitter panels stayed empty until I fixed the IP SLA OID (.1.5.2.1.4.{probe} → netclaw_path_jitter_ms). Syslog panels were useless until device_name became a real Loki label, not a field buried in JSON. BMP on RR1 sat in TCP Down until I stopped trying to source it from a VRF-forwarded interface — BMP rides the global table on CSR; PE syslog rides clab-mgmt. Different animals.
On the Cisco side, parallel SNMP from OTEL to 18 IOL agents at once meant interface_status went stale during flaps — I’d shut Gi2 and the metric still said up. I moved Cisco to sequential polling in bgp-snmp-exporter and taught agents to query job="netclaw-bgp-snmp" with interface="GigabitEthernet2", not just interface_index. Arista cEOS kept tripping Lab Interface Down on a management ifIndex that’s always down; I scoped that alert to CSR roles on the BGP SNMP job.
If you see something similar, check the war-story pattern: symptom → wrong assumption → concrete fix. The fixes are in the repo; the point is don’t repeat my assumptions.
Lab vs Production (Same Pipe, Different Adapters)
I keep getting asked whether this is a toy metrics layout. It isn’t — it’s a production layout with lab-sized fan-out.
| Capability | My lab today | Production |
|---|---|---|
| Per-prefix withdrawals | BMP on RR1 only | BMP on RR/PE edges |
| Peer/prefix counts | SNMP on Cisco CSR | gNMI primary, SNMP fallback |
| Fabric spines | gNMI stream | Same pattern |
| Path quality | IP SLA via SNMP | IP SLA / TWAMP |
| Syslog | OTEL → Loki | Same pipeline |
Same Docker compose, same netclaw_* names, same alert rules and skills. What changes per box is the adapter: CSR gets SNMP + BMP (RR only), Arista gets gNMI, production PE edges would get BMP added when I extend the golden-config template beyond the reflector role. The lab proves the pipe; production adds more speakers on the same pipe.
If You Want to Reproduce It
I built this in six spec-kit phases (specs/031-bgp-route-observability) — each with a checkpoint script so I couldn’t cheat past a broken layer. You don’t need the phase map to use it; you need the stack up and one scenario run.
1
2
3
4
5
6
7
8
9
10
11
12
13
# Stack
docker compose -f observability/docker-compose.observability.yml \
-f observability/docker-compose.bmp.yml \
-f observability/docker-compose.gnmi.yml up -d
# Read-only proof (no Grafana)
bash scripts/demo-part15-observability.sh
# Break something + gate alerts (pyATS, ~15 min for --all)
bash scripts/run-all-scenarios.sh --all
# See what's firing
bash scripts/validate-alert-scenarios.sh
If you use the agent path: fire scenario B on PE1, run lab-alert-triage, then bgp-route-stability-watch with device_name=pe1. Open Grafana only when you want the pretty layer — BGP Route Stability & Path Quality should match what you already saw in PromQL.
Phase checkpoints if you’re validating the build itself: bash scripts/validate-bgp-metrics.sh --phase 1 through --phase 6. Tasks live in specs/031-bgp-route-observability/tasks.md.
Closing Thought
Route stability observability is a router telemetry problem, not an AI speaker problem.
I proved that in my CSR + cEOS lab: BMP from RR1, gNMI from spines, SNMP from PEs, syslog in Loki, alerts that scenarios actually trigger, CLI demos that don’t need Grafana, and agent skills that never touch Protocol MCP for RCA. Protocol MCP keeps the injection demo honest. The routers keep the monitoring story real.
Part 16: NetFlow and traffic intelligence
Further reading: architecture · failure scenarios · baselines · quickstart