vine: properly replicate temp files

[JinZhou5042]

Proposed Changes

Fix #3996

• Update the default values for a set of arguments regarding temp file replication:

  • q->worker_source_max_transfers from 3 to 10
  • max_transfer_procs from 5 to 10
  • I keep q->transfer_replica_per_cycle as is (which is 10), as it limits the number of replications to request per temp file per iteration, and we are not likely to replicate each temp file more than 10 times.

• Improve the logic in the replication process, which helps the manager replicate more efficiently, add more comments

• Provide an API for Python to query the replica count for a file

Merge Checklist

The following items must be completed before PRs can be merged.
Check these off to verify you have completed all steps.

• make test Run local tests prior to pushing.
• make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
• make lint Run lint on source code prior to pushing.
• Manual Update: Update the manual to reflect user-visible changes.
• Type Labels: Select a github label for the type: bugfix, enhancement, etc.
• Product Labels: Select a github label for the product: TaskVine, Makeflow, etc.
• PR RTM: Mark your PR as ready to merge.

[JinZhou5042]

I believe the argument values are definitely not the real issue, the actual issue is probably some logic bugs or a bit tangled if statements in the previous implementation. I've cleaned up the logic, and now the replica count for temp files comes out right at the end of every run.

Before:

After:

[dthain]

Looks good!

[JinZhou5042]

RTM