Unfortunately, I have still not been able to figure this out properly. When we catch the ctrl-break, after we do what we need to do, if we exit 1, then HPC fails us and does not requeue the task and if we exit 0, then HPC thinks we have finished and
does not requeue the task. So it seems that HPC has to be the one to actually terminate the process in order for the task to be requeued. Our non-ideal workaround is after we finish our clean up actions, we go into an intentional infinite loop.
This causes HPC to cancel the task after the grace period duration and requeue the task. The down side is that anytime a task is canceled, it now lasts the entire cancellation grace duration (2 minutes). This isn't a big deal, but wastes
grid resources for that time.
I would appreciate any help on this issue and am surprised that I can't find anything about this. The documentation talks about this cancellation grace period but doesn't mention anything about these complications.