Abstract
This paper presents a controlled study of adversarial reinforcement learning in network security through a custom OpenAI Gym environment that models brute-force attacks and reactive defenses on multi-port services. The environment captures realistic security trade-offs including background traffic noise, progressive exploitation mechanics, IP-based evasion tactics, honeypot traps, and multi-level rate-limiting defenses. Competing attacker and defender agents are trained using Deep Q-Networks (DQN) within a zero-sum reward framework, where successful exploits yield large terminal rewards while incremental actions incur small costs. Through systematic evaluation across multiple configurations (varying trap detection probabilities, exploitation difficulty thresholds, and training regimens), the results demonstrate that defender observability and trap effectiveness create substantial barriers to successful attacks. The experiments reveal that reward shaping and careful training scheduling are critical for learning stability in this adversarial setting. The defender consistently maintains strategic advantage across 50,000+ training episodes, with performance gains amplifying when exposed to complex defensive strategies including adaptive IP blocking and port-specific controls. Complete implementation details, reproducible hyperparameter configurations, and architectural guidelines are provided to support future research in adversarial RL for cybersecurity. The zero-sum formulation and realistic operational constraints make this environment suitable for studying autonomous defense systems, attacker-defender co-evolution, and transfer learning to real-world network security scenarios.
Abstract (translated)
本文通过一个自定义的OpenAI Gym环境，开展了一项关于网络安全中对抗强化学习的控制研究。该环境模拟了针对多端口服务的暴力破解攻击和响应式防御机制。环境中包含了现实中的安全权衡因素，包括背景流量噪音、逐步渗透机制、基于IP地址的规避战术、蜜罐陷阱以及多层次的速率限制防御措施。 利用深度Q网络（DQN）在零和奖励框架内对竞争双方——进攻者与防守者进行了训练。在这个框架中，成功的攻击会带来大量的最终奖励，而每次行动则会产生小额成本。通过系统性的评估，在不同的配置下（包括陷阱检测概率、攻击难度阈值以及不同培训方案），结果表明防守者的观察能力和蜜罐的有效性为成功攻击设置了巨大的障碍。 实验揭示了在这一对抗环境中，奖赏塑形和谨慎的训练调度对于学习稳定性至关重要。防守者在整个超过50,000次以上的训练过程中始终保持着战略优势，并且当暴露于包括自适应IP封锁和端口特定控制在内的复杂防御策略时，其性能显著提升。 本文提供了详尽的实施细节、可重复的超参数配置以及架构指南，以支持未来在网络安全对抗强化学习领域的研究工作。零和公式化和现实操作限制使得该环境非常适合用于研究自主防御系统、攻防共进化及向实际网络安全场景中的迁移学习。
URL
https://arxiv.org/abs/2510.05157
