TAAC (Test-As-A-Config) is a thrift-defined integration test framework for validating datacenter network products and platforms.
by facebookPython
Last 12 weeks · 560 commits
4 of 6 standards met
OTG: drive a live ixia-c endpoint, and port the 8 hardening playbooks Makes able to drive a real ixia-c endpoint, and ports the 8 playbooks from : agent warmboot, bgpd restart, qsfp_service restart, fsdb restart, malformed BGP packet, ECMP member overload, ECMP group overload, CPU high-priority queue overload. New thrift: (review first) (new structs, field 6); ( field 23). The declarative route-range model can only describe conformant BGP. This is the escape hatch: replaying a captured stream, precisely-timed churn, deliberately malformed messages. Maps to snappi , arm. Kept as a general byte sequence so the schema doesn't encode one test's use case. Durability — the main thing to weigh in on. Not in configerator yet, and the hashes are stale, so a sync clobbers it. Blast radius is contained: the passthrough uses and forwards the field only when set. A revert therefore leaves every other BGP config working, and the one config that needs the struct fails loudly at its builder. Happy to gate this PR on landing the schema first. One semantic worth knowing. OTG re-sends the sequence on every re-establishment. That makes a peer down/up a usable trigger without . It also means a peer whose UPDATEs draw a NOTIFICATION flaps forever, so replay peers are exempt from the setup-time all-sessions-up gate. Gating on them turns startup into a race against the flap cycle. Four silent-wrong-behaviour fixes Each produced a passing run that measured the wrong thing: — OTG reports ND neighbours expanded, configs write them compressed. IPv6 gateway MAC lookup missed every time, so v6 flows had no destination MAC. Dual-stack storage — used / and kept only the v4 address. "v6 flows" transmitted IPv4. — is an but neither nor , so dropped every . Any thrift optional list checked that way has this bug. — snappi's route-range step counts prefix blocks; is an address delta. The v4 branch used the raw integer, so at /24 walked 256× too far and overlapped the next speaker. Both families now share one helper, which also rejects a step that shifts to 0. Backend capabilities Step-facing APIs — , , . Playbook steps need these to drive peers, gate individual flows, and override header fields. — a disabled group is built in full, then its peers are driven DOWN and exempted from the setup gate. A playbook's toggle-up is then a real transition, not a no-op against an established session. — N devices per group, each with its own MAC and derived router ID. Duplicate router IDs prevent establishment. Prefix-targeted traffic — resolves to an address inside the advertised prefix and sweeps across it, giving the DUT's ECMP hash something to vary on. Without it nothing addresses the ECMP-routed prefixes and wrong next-hop selection is invisible. Upstream gets this from its items. Needless-reset avoidance — skips the re-push when the config is unchanged. A re-push restarts protocols and flaps every session. Ixia-C limitations ixia-c community edition caps control-plane interfaces at 4 and sessions at 4, per IP address per simulated device per AF. The full set needs far more and is rejected with an opaque HTTP 500 at . So it ships as three independently runnable profiles sharing : The builders count both dimensions and log them on every build, so exceeding the cap fails legibly instead of as an HTTP 500. Malformed BGP hand-builds UPDATE bytes per RFC 4271, using RFC 5737 documentation prefixes. Each malformation gets distinct NLRI so its effect is attributable: missing NEXT_HOP (), malformed AS_PATH (), invalid ORIGIN (), bad attribute length (). Two things are load-bearing. The attribute-length case overruns the message, which RFC 7606 §3 makes a session reset, so it goes last — anything after it is never sent on any replay cycle. And ASNs are 4-octet, because the peer negotiates AS4; 2-octet encoding would make every segment malformed and mask the defect each UPDATE exists to isolate. The assertion is blast radius: unrelated sessions and the forwarding path survive a peer that repeatedly establishes, injects garbage, and is torn down. Not "the DUT rejects the route and keeps this peer up." Rate choice Flows use absolute pps (, ) rather than percent-of-line-rate. ixia-c is a software TE: 10% line rate produced ~34% steady loss from receive-side bottlenecking, not from the DUT. Testing : 468 passed, smoke passes. All three profiles have passed against live ixia-c — restarts at 0.011% loss with genuine v6 flows on independent counters, ECMP with at 0.0007%, malformed with at 0.0009% and confirmed on the wire. DUT setup — ports, addressing, prefixes, CoPP for the CPU-queue test: . Known gaps A disabled peer flaps once at setup. OTG has no device-level disable and protocol start is all-or-nothing, so a held-down peer can establish briefly before the hold-down lands. One short up/down per disabled group, before any snapshot window opens. warns if the hold-down failed, since a playbook's toggle-up would otherwise measure nothing. is capped at by the licence, so ECMP_2 coming up adds one next-hop. Enough to exercise path selection, not enough to approach a platform limit. records the real-licence shape: 500 prefixes × 32 next-hops = 16000 members, 34 interfaces. The malformed test cannot prove routes were rejected, only that damage did not spread. compares endpoint dicts by identity, so a bidirectional item paired with itself would address the wrong leg. No profile does that, so it is annotated rather than changed. The related flow-name collision now raises instead of silently duplicating a flow.
Summary Add a continuous-polling to the OSS-side health-check stack (base: existing ) and migrate three health checks to query it in place of one-shot SSH reads. The one-shot SSH pattern reads systemd's last run state, so a service that crashes and auto-restarts between the playbook body and the postcheck reports by the time it's polled. A continuous poll (5s default) catches the crash before restart overwrites it. A fallback closes the sampler's temporal-resolution gap for fast-restarting services like where the poller cadence can miss a crash-restart cycle entirely. Three commits, reviewable independently: 1. Add SystemdStateCollector + migrate UncleanExitHealthCheck OSS path — introduces the collector (batched per poll, per-service series keyed by sample timestamp), migrates to , adds the fallback. 2. SystemctlActiveStateHealthCheck: migrate OSS path to SystemdStateCollector — adds with split verdict: non-allowlisted via , allowlisted () via . / default True. 3. ServiceRestartHealthCheck: migrate OSS path to SystemdStateCollector — adds via (NRestarts delta > 0 across two loaded+enabled samples); split verdict for (full-window for non-allowlisted, final-sample-only for allowlisted). New files: Modified: (journalctl helpers) Testing — 58/58 pass locally. doesn't run on baseline either (a thrift-codegen import chain not wired in the OSS repo tree) — not introduced by this PR.
Summary Extract the domain-agnostic continuous-polling machinery out of FPF and wire it up so the CPU and memory utilization health checks work under TAAC_OSS, where there is no ODS to query. New taac/libs/collectors: base_collector.py: BaseCollector (poll loop, background thread with its own event loop, atexit cleanup, per-poll timeout / NULL-data recording, timestamped rows) plus _now_str/_parse_ts, lifted verbatim from fpf_stress_checks.py. Adds an _atexit_registered guard so a restart doesn't stack duplicate shutdown hooks. registry.py: register/get/unregister/clear collectors and the test-case start timestamp, with no Meta-internal imports. service_polling_collector.py: per-service systemd polling over SSH. cpu_utilization_collector.py, memory_utilization_collector.py. fpf_stress_checks.py and fpf_collector_registry.py re-export the moved symbols, so existing FPF call sites are unchanged; clear_all() still owns the FPF-specific state and delegates the shared part. Health checks: CpuUtilizationHealthCheck / MemoryUtilizationHealthCheck gain an OSS path that reads MAX-over-window values from the registered collector, guarded behind TAAC_OSS so the ODS imports stay internal-only. Both are re-enabled in OSS_HEALTH_CHECKS. health_check_utils.collector_window_start() picks the later of the start_time jq var and the registry's test-case start, so a check measures the interval its author asked for rather than the whole iteration. Runner + handlers: CollectorsTestHandler starts/stops a CPU + memory collector per test config, on by default under TAAC_OSS with a opt-out and the existing tag opt-in internally. BaseCustomTestHandler.should_run() makes handler selection the handler's own concern; the runner just calls it. taac_runner uses the real set_test_case_start_time in both modes and drops the OSS stub and _start_test_case_time_window helper. Test plan Adds unit tests for the collectors, the registry, handler selection, and the window-start helper, plus a collector_playbook in examples/live_smoke_config.py that exercises the path end to end against a live DUT.
Repository: facebook/DNE-TaaC. Description: TAAC (Test-As-A-Config) is a thrift-defined integration test framework for validating datacenter network products and platforms. Stars: 2, Forks: 4. Primary language: Python. Languages: Python (99.1%), Thrift (0.6%), Shell (0.2%), CMake (0.2%). License: Apache-2.0. Open PRs: 1, open issues: 0. Last activity: 5h ago. Community health: 75%. Top contributors: pavanpatil92, Sahil-18, meta-codesync[bot], yelgokul, vik-nexthop, rtl-nexthop, shah-harshal-ai-dev, mloo3, facebook-github-bot, Michael-C-Buckley.