[sihara]

Pull Request Description

Fixes #7993

What this changes

Four error paths in ADIOI_LUSTRE_Exch_and_write() did goto over, exiting the
write loop on a single rank. Because the loop body performs point-to-point
exchanges that every rank must take part in, and the caller ends with a
collective MPI_Allreduce on error_code, one rank leaving early deadlocks all
the others (see linked issue for stacks and a reproducer).

This patch:

1. records the first error in deferred_error instead of jumping out;
2. agrees collectively, once per iteration, whether any rank has failed, and
   breaks on all ranks at the same iteration;
3. restores the deferred error at over:, so the existing post-loop
   MPI_Allreduce reports it on every rank.

Behaviour

	before	after
write fails on one rank	hangs forever, no error	returns MPI_ERR_IO on all ranks
iterations wasted after the error	all remaining	none — all ranks break at the error's own iteration

Design note for reviewers

Step 2 adds one MPI_Allreduce of a single int per iteration. We have not
measured its cost cleanly — our runs are at 512 ranks / 2242 iterations / ~9.6 TB
and this system has a ~10% run-to-run spread, so we can say only that we did not
see an effect that stands out above that, which is a weak bound. On much larger
rank counts you may prefer:

• checking every N iterations (bounds wasted work at N, cuts the collective
  count by N), or
• a non-blocking MPI_Iallreduce overlapped with the following iteration.

Happy to rework it either way. Without step 2 the deadlock is still fixed, but a
failed job runs to the end of the loop before reporting — on our reproducer that
was 1757 further iterations.

Testing

Reproduced and fixed on stock MPICH 5.0.2, built from the release tarball
with --with-file-system=lustre+ufs, against Lustre 2.14.0 (ldiskfs, 8 OSTs).

Reproducer: romio_quota_deadlock.c (attached in the issue) — ~60 lines, no
IOR needed, 2 client nodes. Ranks are interleaved in the file to force
two-phase collective buffering, then a single MPI_File_write_all is issued
while the user's block quota is exhausted partway.

            quota headroom 1 GB, 30 GB interleaved collective write, 32 ranks / 2 nodes

before:     mpirun exit 124   -- hung, killed by a 200 s timeout
after:      mpirun exit 0     -- "write returned an error after 1.7s",
                                 MPI_File_write_all ... failed, Other I/O error

patch -p1 applies cleanly to 5.0.2; make and make install both exit 0.
The same four goto over statements and the post-loop MPI_Allreduce are
present and identical on main (595/634/649/654 and 323), so the patch is
offered against main.

Note one trap for anyone reproducing: the access pattern must be interleaved.
With each rank writing a single contiguous region, ROMIO takes
ADIOI_LUSTRE_IOCONTIG and never enters the affected code path.

Separately, this was originally found at scale on OpenMPI 5.0.9's vendored
romio341 (a MPICH 3.4.1 snapshot) with IOR ior-hard on 512 ranks, where 54 ranks
took the error exit and 458 blocked in MPI_Waitall.

Author Checklist

• Provide Description
  Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
• Commits Follow Good Practice
  Commits are self-contained and do not do two things at once.
  Commit message is of the form: module: short description
  Commit message explains what's in the commit.
• Passes All Tests
  Whitespace checker. Warnings test. Additional tests via comments.
• Contribution Agreement
  For non-Argonne authors, check contribution agreement.
  If necessary, request an explicit comment from your companies PR approval manager.

[wkliao]

This line will add max_ntimes number of calls to MPI_Allreduce(), which can be very expensive.

[hzhou]

@sihara In order for us to accept and properly credit your contribution, could you sign the contribution license agreement http://www.mpich.org/documentation/contributor-docs/?