# kv_bench Reproduction Guide (Mace vs RocksDB)

This repository is used to reproduce and compare `mace` and `rocksdb` benchmark results across phase0~phase4.

## 1. Prerequisites
- Linux
- A high-speed storage mount directory you choose (typically an NVMe mount point)
- Rust/Cargo
- CMake (to build `rocksdb_bench`)
- Python 3 (for result aggregation and plotting)

## 2. Storage Directory Configuration (Important)
`/nvme` is no longer hardcoded. You can use any mount directory.

Recommended: set one shared variable first:

```bash
export KV_BENCH_STORAGE_ROOT=/path/to/your/nvme_mount/kvbench
mkdir -p "${KV_BENCH_STORAGE_ROOT}"
```

All scripts below take this directory (or one of its subdirectories) as the first argument.

## 3. Initialization
```bash
cd /home/abby/kv_bench/scripts
./init.sh
source ./bin/activate
cd /home/abby/kv_bench
```

## 4. Quick Baseline Comparison (W1~W6)
Clean old data first:

```bash
rm -rf "${KV_BENCH_STORAGE_ROOT}/basic_mace" "${KV_BENCH_STORAGE_ROOT}/basic_rocks"
mkdir -p "${KV_BENCH_STORAGE_ROOT}/basic_mace" "${KV_BENCH_STORAGE_ROOT}/basic_rocks"
```

Run both engines:

```bash
./scripts/mace.sh "${KV_BENCH_STORAGE_ROOT}/basic_mace" ./scripts/benchmark_results.csv
./scripts/rocksdb.sh "${KV_BENCH_STORAGE_ROOT}/basic_rocks" ./scripts/benchmark_results.csv
```

Generate plots:

```bash
./scripts/bin/python ./scripts/plot.py ./scripts/benchmark_results.csv ./scripts
```

## 5. Phase Reproduction Commands

### Phase 1
```bash
rm -rf "${KV_BENCH_STORAGE_ROOT}/phase1"
mkdir -p "${KV_BENCH_STORAGE_ROOT}/phase1"
./scripts/phase1.sh "${KV_BENCH_STORAGE_ROOT}/phase1" ./scripts/phase1_results.csv
```

### Phase 2
```bash
rm -rf "${KV_BENCH_STORAGE_ROOT}/phase2"
mkdir -p "${KV_BENCH_STORAGE_ROOT}/phase2"
./scripts/phase2.sh "${KV_BENCH_STORAGE_ROOT}/phase2" ./scripts/phase2_results.csv
```

Optional: enable tier-l representative subset:

```bash
RUN_TIER_L_REPRESENTATIVE=1 TIER_L_REPEATS=1 \
./scripts/phase2.sh "${KV_BENCH_STORAGE_ROOT}/phase2" ./scripts/phase2_results.csv
```

### Phase 3
```bash
rm -rf "${KV_BENCH_STORAGE_ROOT}/phase3"
mkdir -p "${KV_BENCH_STORAGE_ROOT}/phase3"
./scripts/phase3.sh "${KV_BENCH_STORAGE_ROOT}/phase3" ./scripts/phase3_results.csv
```

### Phase 4 (run one engine at a time)
Mace:
```bash
rm -rf "${KV_BENCH_STORAGE_ROOT}/phase4_mace"
./scripts/phase4_soak.sh mace "${KV_BENCH_STORAGE_ROOT}/phase4_mace" \
  ./scripts/phase4_results_mace.csv ./scripts/phase4_restart_mace.csv
```

RocksDB:
```bash
rm -rf "${KV_BENCH_STORAGE_ROOT}/phase4_rocks"
./scripts/phase4_soak.sh rocksdb "${KV_BENCH_STORAGE_ROOT}/phase4_rocks" \
  ./scripts/phase4_results_rocks.csv ./scripts/phase4_restart_rocks.csv
```

## 6. Where Result Inputs (CSV) Are Stored
Default output files:
- `./scripts/benchmark_results.csv`
- `./scripts/phase1_results.csv`
- `./scripts/phase2_results.csv`
- `./scripts/phase3_results.csv`
- `./scripts/phase4_results_*.csv`
- `./scripts/phase4_restart_*.csv`

The unified schema is emitted by both engine binaries (same format for mace/rocksdb). Key columns:
- `engine`: `mace` / `rocksdb`
- `workload_id`: `W1..W6`
- `durability_mode`: `relaxed` / `durable`
- `threads,key_size,value_size,prefill_keys`: case configuration
- `ops_per_sec`: throughput
- `p50_us,p95_us,p99_us,p999_us`: latency percentiles
- `error_ops`: number of failed operations
- `read_path`: `snapshot` / `rw_txn`

## 7. Where to Interpret Results

### Phase 1 (stability)
```bash
./scripts/bin/python ./scripts/phase1_eval.py ./scripts/phase1_results.csv
```
Check:
- `throughput_cv` (<=10%)
- `p99_cv` (<=15%)
- `stable` and overall pass ratio

### Phase 2 (core report)
```bash
./scripts/bin/python ./scripts/phase2_report.py ./scripts/phase2_results.csv
```
Check:
- `throughput_median`
- `p95_median`, `p99_median`
- `slower_engine`, `slower_ratio`

### Phase 3 (durability cost)
```bash
./scripts/bin/python ./scripts/phase3_report.py ./scripts/phase3_results.csv
```
Check:
- `throughput_drop_pct` (durable vs relaxed throughput drop)
- `p99_inflation_pct` (durable vs relaxed p99 inflation)

### Phase 4 (recovery capability)
```bash
./scripts/bin/python ./scripts/phase4_report.py ./scripts/phase4_restart_mace.csv
./scripts/bin/python ./scripts/phase4_report.py ./scripts/phase4_restart_rocks.csv
```
Check:
- `restart_success`
- `restart_ready_ms` at `p50/p95/p99/max`

## 8. CLI Configurability (No Hardcoded Disk Prefix)
- Both benchmark binaries support `--path` to set the DB directory.
- All scripts use the first argument as storage root/path.
- You can point `${KV_BENCH_STORAGE_ROOT}` to any mount point (NVMe, SSD, RAID, ephemeral disk).

## 9. Comparison Best Practices
Only compare cases under identical dimensions:
- `workload_id`
- `key_size/value_size`
- `threads`
- `durability_mode`
- `read_path`

If `error_ops > 0`, investigate that case first before drawing performance conclusions.