Skip to content

[Bug-fix] Patching ArtEvalBench's Acto (SOSP'23) entry #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bastoica wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[Bug-fix] Patching ArtEvalBench's Acto (SOSP'23) entry #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b33c015
feat: apply patch that enables Acto to support loading operator image…
bastoica Dec 23, 2025
c02b3ed
fix: relax version comparison in OracleEnvSetup to allow Python 3.8 o…
bastoica Dec 23, 2025
0ce918c
fix: add changelog notes to README; remove spurious/unused source files
bastoica Dec 23, 2025
ebfefc1
fix: update reference results for running Acto on a smaller subset of…
bastoica Dec 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions benchmarks/arteval_bench/data/benchmark/sosp23_acto/_agent_eval/oracle_env_setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,14 @@ class Dependency:
name="docker", binary="docker",
),

# Python v3.10+
# Python v3.8+
Dependency(
name="python3", binary="python3",
cmd=["python3", "--version"], parse_regex=r"Python\s+([0-9.]+)",
require=(3, 8), compare="eq",
require=(3, 8), compare="gte",
),

# pip3 for Python 3.10+
# pip3 for Python 3.8+
Dependency(
name="pip3", binary="pip3",
),
Expand Down
42 changes: 5 additions & 37 deletions benchmarks/arteval_bench/data/benchmark/sosp23_acto/_agent_eval/refs/table5.ref.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,5 @@
{
"operators": [
{
"operator": "CassOp",
"undesired_state": 2,
"system_error": 0,
"operator_error": 0,
"recovery_failure": 2,
"total": 4
},
{
"operator": "CockroachOp",
"undesired_state": 3,
Expand All @@ -16,22 +8,6 @@
"recovery_failure": 0,
"total": 5
},
{
"operator": "KnativeOp",
"undesired_state": 1,
"system_error": 0,
"operator_error": 2,
"recovery_failure": 0,
"total": 3
},
{
"operator": "OCK-RedisOp",
"undesired_state": 4,
"system_error": 1,
"operator_error": 3,
"recovery_failure": 1,
"total": 9
},
{
"operator": "OFC-MongoDBOp",
"undesired_state": 3,
Expand All @@ -56,14 +32,6 @@
"recovery_failure": 0,
"total": 3
},
{
"operator": "SAH-RedisOp",
"undesired_state": 2,
"system_error": 0,
"operator_error": 0,
"recovery_failure": 1,
"total": 3
},
{
"operator": "TiDBOp",
"undesired_state": 2,
Expand All @@ -90,10 +58,10 @@
}
],
"totals": {
"undesired_state": 32,
"system_error": 4,
"operator_error": 10,
"recovery_failure": 10,
"total_all": 56
"undesired_state": 23,
"system_error": 3,
"operator_error": 5,
"recovery_failure": 6,
"total_all": 37
}
}
10 changes: 5 additions & 5 deletions benchmarks/arteval_bench/data/benchmark/sosp23_acto/_agent_eval/refs/table6.ref.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,27 @@
"symptoms": [
{
"symptom": "System failure",
"bugs": 5
"bugs": 2
},
{
"symptom": "Reliability issue",
"bugs": 15
"bugs": 6
},
{
"symptom": "Security issue",
"bugs": 2
},
{
"symptom": "Resource issue",
"bugs": 9
"bugs": 5
},
{
"symptom": "Operation outage",
"bugs": 18
"bugs": 10
},
{
"symptom": "Misconfiguration",
"bugs": 15
"bugs": 11
}
]
}
16 changes: 8 additions & 8 deletions benchmarks/arteval_bench/data/benchmark/sosp23_acto/_agent_eval/refs/table7.ref.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,23 @@
"test_oracles": [
{
"test_oracle": "Consistency oracle",
"bugs": 23,
"percentage": 41.07
"bugs": 14,
"percentage": 42.42
},
{
"test_oracle": "Differential oracle for normal state transition",
"bugs": 25,
"percentage": 44.64
"bugs": 19,
"percentage": 57.58
},
{
"test_oracle": "Differential oracle for rollback state transition",
"bugs": 10,
"percentage": 17.86
"bugs": 5,
"percentage": 15.15
},
{
"test_oracle": "Regular error check (e.g., exceptions, error codes)",
"bugs": 14,
"percentage": 25.00
"bugs": 7,
"percentage": 21.21
}
]
}
3 changes: 3 additions & 0 deletions benchmarks/arteval_bench/data/benchmark/sosp23_acto/acto/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ The entire artifact process can take around 2 hours if run with a concurrency of

If you have any questions, please contact us via email or HotCRP.

> [!NOTE]
> This artifact version has removed the Reddis and Knative operators since the originally faulty images are no longer available. The current version also applies patch [#8ecdcda](https://github.com/xlab-uiuc/acto/commit/8ecdcda5a51d7c5625e49802bbd1bc75c0cf07ef) that allows Acto to load operators from pre-packaged archives when the original images are no longer availble for download.

# 2. Prerequisites

## Setting up local environment
Expand Down
Loading
Loading