Repository navigation
How to handle early stopping patience with failure cases #294
Description
Activity
You can modify the evaluator to return a distinct metric key for failures (e.g.,
{"error": 1.0, "combined_score": 0.0}) instead of just{"combined_score": 0.0}. Then update the early stopping logic inprocess_parallel.py:600-642to skip iterations where "error" or "timeout" keys are present when countingiterations_without_improvement. This way, early stopping only triggers on successful evaluations that plateau, not on repeated failures.any chance you'd be willing to add in support for hard coded error handling logic? I can keep patching openevolve but I would much rather have these dependencies in the dependencies 😂
I do want to fix all such pending issues when I get some time but meanwhile feel free to send in your PRs, happy to add them in.
Reacted by Amol KapoorWorking on this — I'm following the fix direction codelion suggested above: the early-stopping logic in
process_parallel.pynow skips any iteration whose metrics carry anerrorortimeoutkey when countingiterations_without_improvement, with both the patience-based branch and the negative-patience event branch wrapped in that guard, so failures can no longer reset or tick the counter (or trigger the "successfully solved" path). A PR with tests covering failed, timeout-shaped, and negative-best scenarios is coming shortly.PR up: #502 (includes tests for failed, timeout-shaped, and negative-best scenarios; full suite green).
- added a commit that references this issue
on Oct 10, 2026
Say I have an evaluator that returns a score based on some written code. If the code does not compile (e.g. a syntax error) I simply return a score of 0.
Say I also have a config with an early stopping patience set. Say that my patience score is a smallish number, like 3.
Right now what I am observing is if my evaluator cannot run a bit of iterated code 3 times, it will assume that it has succeeded and move on, even though it actually failed three times in a row.
Is there a way to distinguish between 'this was successful and that is why the score did not change (and you should stop running)' and 'this was a failure and that is why the score did not change (and you should keep running)'?