fix double reap of tasks on app failure

[btovar]

@JinZhou5042, this is related to the fix you did for &=.
@colinthomas-z80, this is fix I was talking about that might be hiding why you tasks get stuck.

Merge Checklist

The following items must be completed before PRs can be merged.
Check these off to verify you have completed all steps.

• make test Run local tests prior to pushing.
• make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
• make lint Run lint on source code prior to pushing.
• Manual Update: Update the manual to reflect user-visible changes.
• Type Labels: Select a github label for the type: bugfix, enhancement, etc.
• Product Labels: Select a github label for the product: TaskVine, Makeflow, etc.
• PR RTM: Mark your PR as ready to merge.

[dthain]

This looks good, we should be using switch-on-enum like this more consistently throughout the code.

[dthain]

Odd build failures on this one, is it just a transient issue?

The bug is a serious one, so it would be good to get the fix deployed soon.

[btovar]

I agree! A simpler hot fix has been merged already (#4002). I'm trying to figure out why it fails in the docker CI but not locally.

[dthain]

Ok, I figured out a few things here.

First, there is a minor deficiency in our test scripts -- if the vine manager does not run to completion, then wait_for_file_creation causes the whole script to exit. And so it never gets to the point of displaying the manager and worker logs, which it would normally do if the manager exits with a failure indication. So I just added a quick hack to display manager and worker logs regardless.

Now, the actual symptom is that the manager gets as far as running Task 14 to completion, then Task 15 is submitted and marked as READY.... and then does nothing. The worker, as designed, notices that nothing happens for ten seconds, and then bails out.

Task 14 is ls -lR cctools | wc -l and Task 15 is then wc -l infile where infile is generated by a curl minidisk.

[dthain]

So that makes me think that either the task does not match to the worker in scheduling, or some prior action manipulated the data structures improperly.

One thing that does stand out to me as a potential problem is that the (newly modified) handle_failure treats VINE_MGR_FAILURE as a reason to call remove_worker. I don't think this is right, because a manager failure encompasses things like "Couldn't open an input file locally" and does not imply something is wrong with the worker.

I am not entirely sure that is the source of the problem, but it looks suspicious to me.

@btovar I hope that is helpful and I'll pass the baton back to you on this one.

[btovar]

Related to #4109. Some conditionals just consider worker vs app failures.