cb-multios | DARPA Challenges Sets for Linux , Windows , and macOS

The DARPA Challenge Binaries (CBs) are custom-made programs specifically designed to contain vulnerabilities that represent a wide variety of crashing software flaws. They are more than simple test cases, they approximate real software with enough complexity to stress both manual and automated vulnerability discovery. The CBs come with extensive functionality tests, triggers for introduced bugs, patches, and performance monitoring tools, enabling benchmarking of patching tools and bug mitigation strategies. The CBs were originally developed for DECREE -- a custom Linux-derived operating system that has no signals, no shared memory, no threads, no standard libc runtime, and only seven system calls -- making them incompatible with most existing analysis tools. In this repository, we have modified the CBs to work on Linux and OS X by replacing the build system and re-implementing CGC system calls via standard libc functionality and native operating system semantics. Scripts have been provided that help modify the CBs to support other operating systems.
How good are tools from the Cyber Grand Challenge vs. existing program analysis and bug finding tools?
When a new tool is released, how does it stack up against the current best?
Do static analysis tools that work with source code find more bugs than dynamic analysis tools that work with binaries?
Are tools written for Mac OS X better than tools written for Linux, and are they better than tools written for Windows?