SymSan+i2s versus AFL++ CmpLog/RedQueen
====================================

Source: _tmp/symsan-eval-results-20261103 in the blog workspace.
Exported from tables/per_trial_labs.csv; all 300 exported rows were checked
against the corresponding raw coverage_final.csv and fuzzer_stats files.

15 targets, 5 one-hour runs per configuration. Saved AFL++ run_time values
are 3599 seconds. AFL++ version: ++4.31c.

trials.csv: per-run gcov covered-line counts (l_abs) and AFL++ recorded
execution counts (execs). Labels:
  cm  = AFL++ CmpLog/RedQueen (-l 2AT) + ordinary mutations
  i2s = AFL++ SymSan custom mutator, i2s only + ordinary mutations
  cmo = SymSan+i2s, AFL_CUSTOM_MUTATOR_ONLY=1, ordinary mutations disabled
  rq  = CmpLog/RedQueen, AFL_CUSTOM_MUTATOR_ONLY=1, ordinary mutations disabled
The without-havoc configurations disable ordinary deterministic, havoc,
and splice stages. CmpLog/RedQueen still runs in the rq configuration.
Neither JIGSAW nor Z3 was enabled in these configurations.

comparisons.csv: 30 rows, one per target and setting (with_havoc or
without_havoc). Both settings include median gcov lines, median recorded
execution counts, the execution ratio (SymSan / RedQueen), and the relative
line-coverage change. Each median is over five runs; these are ratios of
medians, not medians of individually paired run ratios.

summary.csv: per-target medians of five runs. exec_ratio is the ratio of
median AFL++ execution counts WITH havoc (i2s / cm). line_change_pct is the
relative change in median covered lines (100 * (i2s / cm - 1)). The median of the
15 execution ratios is 1.2362718934; it is not a pooled execution ratio.
symsan_only_lines and redqueen_only_lines are medians for cmo and rq.
symsan_only_execs and redqueen_only_execs are median execution counts for
cmo and rq. without_havoc_exec_ratio and without_havoc_line_change_pct
report the corresponding execution ratio and line-coverage change.

With havoc: SymSan has higher recorded executions on 13/15 targets;
higher median line coverage on 8/15, ties on 2, lower on 5.
Without havoc: higher recorded executions on 11/15 targets;
higher median line coverage on 12/15, no ties, lower on 3.
The without-havoc comparison removes ordinary mutation stages, but does
not isolate tracing cost or solver speed. Execution counts also depend on
how many candidates each implementation generates and submits to AFL++.

Execution counts are AFL++ counters, not an independently reconciled count
of all auxiliary executions; the one-hour budget includes analysis time.
Coverage is the absolute number of lines reported by gcov after replay.
The supplied notes report matching starting inputs and test drivers.

These exports support inspection of the results, not full reproduction of
the experiment: the bundle does not contain queues, full fuzzing logs,
complete build/run scripts, machine specifications, or exact target and
SymSan source revisions. Historical restart notes are in the source bundle;
all four exported configurations have five completed coverage cells per target.
The older LibAFL campaigns are deliberately omitted from this comparison.

The chart shows the WITH-havoc comparison, derived from summary.csv.
The 9.2x geometric-mean tracing slowdown comes from Table 5 of the SymSan
USENIX Security 2022 paper, not these new campaigns:
https://www.usenix.org/system/files/sec22-chen-ju.pdf

Signed execution-change columns: execution_change_pct = 100 * (exec_ratio - 1).
In summary.csv this is WITH havoc; without_havoc_execution_change_pct is
the corresponding WITHOUT-havoc value. Negative values mean SymSan recorded
fewer executions. These signed percentages remain available in the downloads.

Blog tables show median gcov lines and throughput (recorded executions per
second) in separate columns for both methods. Throughput is the median
execution count divided by the fixed 3,600-second campaign budget.
comparisons.csv includes redqueen_execs_per_sec and symsan_execs_per_sec.
The ratio chart is retained as an asset but is no longer embedded in the post.