About the upper bound accuracy of the generator

[xiaoer2498]

Hi, How is the upper bound accuracy of the generator mentioned in the paper calculated?

[Peng-fei-Wang]

Hi @xiaoer2498,

The upper bound accuracy is calculated based on the "Pass@k" logic. For each instance, we sample multiple candidate SQLs from the generator. If any one of these candidates executes correctly (i.e., the execution result matches the ground truth), the instance is considered a success for the upper bound.

[xiaoer2498]

Thanks, but What is the number of candidates?

[Peng-fei-Wang]

Thanks, but What is the number of candidates?

9 candidates from the ICL SQL Generator, and 8 candidates from the Reasoning SQL Generator.