NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations
2026-07-11 • Cryptography and Security
Cryptography and SecurityMachine Learning
AI summaryⓘ
The authors created NetInjectBench, a test with 130 real-world network operation scenarios, to study how language models handle unsafe instructions disguised in routine texts. They tested several language models and safety methods, finding that simple protections still allowed many unsafe actions, while advanced approaches combining metadata checks prevented unsafe tool use almost entirely without blocking useful actions. The study highlights that for safe network operation automation, language models need not just careful prompt design but also real-time checks on what actions they are allowed to perform.
large language modelsprompt injectionnetwork operationstool use safetymetadata policyallowlistingbenchmarkingQwen2.5-7BLlama3.1-8BMistral-7B
Authors
Ruksat Khan Shayoni, Muhammad Faraz Shoaib, S M Asif Hossain, M. F. Mridha
Abstract
Tool-using large language model (LLM) agents are attractive for network operations, but tickets, alerts, logs, runbooks, and ChatOps messages can carry indirect prompt injections. We present NetInjectBench, a 130-scenario benchmark that separates untrusted artifact text, trusted policy metadata, and evaluation labels for network-operation tool use. The sample contains 40 benign, 40 weak-attack, 40 strong-attack, and 10 approved high-impact change scenarios; each is evaluated with Qwen2.5-7B, Llama3.1-8B, and Mistral-7B. Across 240 attack instances, naive execution reached an 82.50% unsafe tool-action rate. Prompt-only safety, Self-Reminder, Spotlighting, and a Two-Pass LLM Judge reduced this rate to 25.63%, 21.67%, 18.33%, and 10.00%, respectively. Static allowlisting reached 5.00% but blocked all approved changes, yielding 0.00% usefulness and 100.00% overblocking on approved cases. Under the stated metadata-integrity assumption, the metadata-aware policy gate produced 0/240 unsafe attack actions, with a 95% Wilson upper bound of 1.58%, while preserving 99.17% attack-scenario usefulness and 100.00% approved-change usefulness. The findings show that network-operation agents need execution-time authorization boundaries alongside prompt-level instruction hygiene.