Skip to content

Bug fix: zero-length write request to a record variable in a collective put API - #239

Merged
wkliao merged 3 commits into
Parallel-NetCDF:masterfrom
wkliao:zero_put_rec
Sep 11, 2026
Merged

Bug fix: zero-length write request to a record variable in a collective put API#239
wkliao merged 3 commits into
Parallel-NetCDF:masterfrom
wkliao:zero_put_rec

Conversation

@wkliao

@wkliao wkliao commented Sep 3, 2026

Copy link
Copy Markdown
Member

This PR fixes the issue reported in #238

@wkliao
Bug fix: check buf == NULL for zero-sized request
4b316eb 
varp is needed to tell if a variable is a record variable.
With this change, varid must be valid when reaching to put_varm().

Another fix is required to return error earlier in dispatcher when
a invalid varid is detected.
@wkliao
Check varid and return immediately if it is invalid
dc29b83 
When writing to a record variable, a collective call to MPI_Allreduce()
is necessary to update and sync-ed the number of records among all
processes. For fix-sized variables, such a call to MPI_Allreduce() is
not necessary.

The argument varid of a put API supplied by the user may point to either
a fix-sized or record variable. Because there is no way to tell if an
invalid varid is a fix-sized variable or a record variable, there is no
consistent if-condition can be devised to determine whether to call
MPI_Allreduce().

The solution adopted by this commit adds a call to MPI_Allreduce() to
check the error returned from sanity_check(), which unfortunately adds
some communication cost. Luckily, this fix is only necessary for
collective blocking put APIs.
@wkliao
tst_zero_len_recvar.c - zero-length request in a collective blocking …
414b7ab 
…put API

This test program was provided by Blaise Bourdin (and revised a little). It
calls a collective blocking put API to write to a record variable where one of
the processes makes a zero-length request.

When writing to a record variable, PnetCDF calls MPI_Allreduce to update
and sync the number of records among all processes. If a subset of the
processes has a zero-length request, these processes may not participate
the call to MPI_Allreduce, causing program to hang.

See PR Parallel-NetCDF#239.
@wkliao
wkliao merged commit 8c05696 into Parallel-NetCDF:master Sep 11, 2026
3 checks passed
@wkliao wkliao mentioned this pull request Sep 11, 2026
Closed
@bourdin

bourdin commented Sep 12, 2026

Copy link
Copy Markdown

@wkliao Thanks for your fix. I can confirm that this closed a long-standing bug in my fracture code. Out of curiosity, do you have a tentative date for your next release? Ideally, I'd like to be able to push a patch to PETSc so that it can use a point release instead of pointing to a commit.

@wkliao

wkliao commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

Sure, I can do that.

I am curious if you would like to give PnetCDF's nonblocking APIs a try.
They will not cause the process hanging problem you are seeing.
In addition, they generally provide a much higher performance
(at least, in most of the cases I have evaluated.)

wkliao added a commit that referenced this pull request Sep 13, 2026
@wkliao
tst_zero_len_recvar2.c - simplified version of tst_zero_len_recvar.c
b497ab8 
It is to test a collective blocking put API that writes to a record variable
where one of the processes makes a zero-length request.

It is used to test PR #239
@bourdin

bourdin commented Sep 13, 2026

Copy link
Copy Markdown

Thing is that I use pnetcdf through exodusII, but if you point me towards an example or some documentation pages, I'd be happy to see if we could update exodusII to use nonblocking IO

@wkliao

wkliao commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

There is a short usage comparison of nonblocking and blocking APIs in
doc/blocking_vs_nonblocking.md

Example programs in C can be found in examples/C/nonblocking_write.c
and examples/tutorial/pnetcdf-write-nb.c

Please let me know if you have questions or need further information.

I recently collected some performance numbers using the I/O kernels from
3 real-world applications. They are available in benchmarks/README.md

@wkliao

wkliao commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

Below is a short code fragment, for illustrating how to call nonblocking APIs.

int req[2], status[2];

ncmpi_iput_vara_double(ncid, varid_0, start, count, buf_0, &req[0]);
ncmpi_iput_vara_double(ncid, varid_1, start, count, buf_1, &req[1]);

ncmpi_wait_all(ncid, 2, req, status);

@bourdin

bourdin commented Sep 14, 2026

Copy link
Copy Markdown

Thanks for the link. "I" have added non-blocking IO to the exodusII library sandialabs/seacas#928

@wkliao

wkliao commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

Hi, @bourdin
Can you do me a favor by testing your cases using PnetCDF 1.15.0 official release?
If using the nonblocking APIs did not causing process to hang, then I would like
to mention this fact in PnetCDF 1.15.1 release note that the hanging bug only
happens to blocking APIs.

@bourdin

bourdin commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown

You are right, the zero-length but does not affect the non-blocking APIs, and the speedup I observed in 1.15.0 is consistent with that of 1.15.1.

@wkliao

wkliao commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

Thanks for confirming it. I will make the new release in a day or two.

@wkliao

wkliao commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

@bourdin FYI. PnetCDF 1.15.1 was released today (Sep. 18, 2026)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wkliao @bourdin