AI found the cause of network slowdown in a Rust library.
Understanding a complex Linux crash in minutes instead of hours sounds like science fiction. But developer Josef Bacsik assures us that this is precisely the result of the release of Systing 1.0 , a system tracing tool based on eBPF that can now analyze data using artificial intelligence.
Systing was originally created as a tool for collecting detailed system traces for subsequent visualization in Perfetto. The program records all key events in a single timeline, helping to visualize the interrelationships between processes. This approach proved particularly useful for the developers of sched_ext, who needed to visually monitor the task scheduler.
However, over time, problems emerged. The traces were too voluminous and overloaded with detail, making analysis a complex task. Some data, such as application interactions with the network, was difficult to visualize: half of the operations occurred within the context of the application, the other half within the context of the Linux kernel . Analyzing these situations required manual work.
In version 1.0, Bachik changed his approach. Instead of storing data only in trace format, Perfetto Systing can now write results to the DuckDB database. This format allows for direct SQL queries, without lengthy conversion. According to the author, DuckDB is noticeably faster when working with large volumes of data.
The main innovation is integration with Claude Code via a dedicated module. Instead of writing and maintaining his own analysis scripts, Bachik delegated trace analysis to artificial intelligence. Claude accesses the DuckDB database and answers questions about system behavior in real time.
One example involved a network application that wasn't achieving the expected data transfer rate. After running Systing and submitting a trace to Claude Code, the system detected internal locks in the Rust library. After fixing them, the transfer time dropped from 12 to 8 seconds. It was then discovered that data unpacking was occurring in the same thread as network operations. After separating the tasks, the time dropped to 4 seconds. The final culprit was too small transfer buffers, which were creating unnecessary load. After fixing them, the time dropped to 2 seconds. The entire project took about an hour.
But the story didn't end there. In production, the transfer time suddenly increased to 24 seconds. A new trace revealed that the production machine was spending significant time kretprobe in a "hot" network function. Systing stores kernel version information, and Claude immediately noticed the difference: the test environment was using Linux kernel 6.12, while the production environment was using 6.6. One of the security tools was connecting to the network function via an eBPF hook , and in version 6.6, such hooks are noticeably slower. Optimizations increased the number of network operations per second, causing the kretprobe overhead to become critical and slow the system.
According to Bachik, previously, identifying the cause of such a problem would have taken hours. Most of the time was spent recording the trace in production, while analysis using Claude took less than five minutes. After migrating production workloads to a newer kernel version, the transfer speed returned to two seconds. The developer calls the new version of Systing a turning point. The tool has gradually evolved from a visualization tool into a helper that not only collects data but also helps quickly identify the root cause.
Understanding a complex Linux crash in minutes instead of hours sounds like science fiction. But developer Josef Bacsik assures us that this is precisely the result of the release of Systing 1.0 , a system tracing tool based on eBPF that can now analyze data using artificial intelligence.
Systing was originally created as a tool for collecting detailed system traces for subsequent visualization in Perfetto. The program records all key events in a single timeline, helping to visualize the interrelationships between processes. This approach proved particularly useful for the developers of sched_ext, who needed to visually monitor the task scheduler.
However, over time, problems emerged. The traces were too voluminous and overloaded with detail, making analysis a complex task. Some data, such as application interactions with the network, was difficult to visualize: half of the operations occurred within the context of the application, the other half within the context of the Linux kernel . Analyzing these situations required manual work.
In version 1.0, Bachik changed his approach. Instead of storing data only in trace format, Perfetto Systing can now write results to the DuckDB database. This format allows for direct SQL queries, without lengthy conversion. According to the author, DuckDB is noticeably faster when working with large volumes of data.
The main innovation is integration with Claude Code via a dedicated module. Instead of writing and maintaining his own analysis scripts, Bachik delegated trace analysis to artificial intelligence. Claude accesses the DuckDB database and answers questions about system behavior in real time.
One example involved a network application that wasn't achieving the expected data transfer rate. After running Systing and submitting a trace to Claude Code, the system detected internal locks in the Rust library. After fixing them, the transfer time dropped from 12 to 8 seconds. It was then discovered that data unpacking was occurring in the same thread as network operations. After separating the tasks, the time dropped to 4 seconds. The final culprit was too small transfer buffers, which were creating unnecessary load. After fixing them, the time dropped to 2 seconds. The entire project took about an hour.
But the story didn't end there. In production, the transfer time suddenly increased to 24 seconds. A new trace revealed that the production machine was spending significant time kretprobe in a "hot" network function. Systing stores kernel version information, and Claude immediately noticed the difference: the test environment was using Linux kernel 6.12, while the production environment was using 6.6. One of the security tools was connecting to the network function via an eBPF hook , and in version 6.6, such hooks are noticeably slower. Optimizations increased the number of network operations per second, causing the kretprobe overhead to become critical and slow the system.
According to Bachik, previously, identifying the cause of such a problem would have taken hours. Most of the time was spent recording the trace in production, while analysis using Claude took less than five minutes. After migrating production workloads to a newer kernel version, the transfer speed returned to two seconds. The developer calls the new version of Systing a turning point. The tool has gradually evolved from a visualization tool into a helper that not only collects data but also helps quickly identify the root cause.
