cid fe1fe34f81761e122581537eb94bced14e8724509ba227e6d1ee6511bad82ef6 · original · open access · via openalex
Title: SyzScout: Automated Syzkaller Specification Generation via LLM-Based Agents Authors: Wenyu Zhen; Minhuan Huang; Xiang Li; Yuanping Nie; Yuxian Wan; Fan Zhong Venue: International Conference on Computer and Communication Systems (ICCCS) Year: 2026 Canonical: doi:10.1109/icccs69761.2026.11613509 URL: https://doi.org/10.1109/icccs69761.2026.11613509 Collector: openalex OA: true ABSTRACT The security of operating system kernels underpins the reliable operation of the entire software stack. Kernel fuzzing has emerged as a primary technique for detecting kernel vulnerabilities, yet its effectiveness hinges on precise formal descriptions of kernel interfaces, known as system call specifications. While prior work has explored automated specification generation through static analysis, dynamic analysis, and direct large language model (LLM) inference, the process remains largely manual, leaving a substantial number of driver interfaces inadequately covered. In this paper, we propose SyzScout, an automated system call specification generation system powered by LLM-based agents. SyzScout operates through a three-phase closed-loop workflow comprising code exploration, specification synthesis, and compilation-driven iterative repair. The agent retrieves kernel source code on demand within a bounded context window, performs semantic reasoning over interface definitions, and leverages compiler feedback to iteratively correct specification errors. We evaluate SyzScout on Linux kernel v6.12 and show that Syzkaller equipped with SyzScout-generated specifications achieves a 4.7 % increase in basic block coverage over the vanilla configuration, and uniquely triggers 9 kernel crashes, of which 4 have been confirmed by maintainers and 3 have received patches.