Serialize Nones

[flying-sheep]

• Fixes .write does not save None values #673
• Added tests
• Added release notes

Rendered release notes

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.63%. Comparing base (4bcd989) to head (230469c).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #999      +/-   ##
==========================================
- Coverage   87.08%   84.63%   -2.45%     
==========================================
  Files          40       40              
  Lines        6107     6116       +9     
==========================================
- Hits         5318     5176     -142     
- Misses        789      940     +151     

Files with missing lines	Coverage Δ
src/anndata/_io/specs/methods.py	88.45% <100.00%> (-0.16%)	⬇️
src/anndata/_io/specs/registry.py	95.40% <ø> (-0.63%)	⬇️

... and 6 files with indirect coverage changes

[ivirshup]

As discussed, I don't want to change the on-disk format in the next release. So I'd hold off on this until at least 0.11.

[flying-sheep]

Since 0.10 is released, we can document and then merge this, right?

[ivirshup]

I would like to have proposals for all disk format changes written up and discussed with other implementers before merging them to main.

[flying-sheep]

Sure, we can use this PR as reference implementation.

Which other implementers are there? Is there a channel with the relevant people on some communication medium?

[alam-shahul]

Hi there, I feel that this is an important feature/bug fix, I was wondering if there were plans to merge this anytime soon/if it's possible to help.

[ilan-gold]

Two things:

1. Should we put this feature behind a flag initially? I know we had spoken about that as a way forward before. However, in this case, I'm no really sure what we'd gain by doing that.
2. This PR is simple enough that it can be merged without much comment. I think we should try to make it clear that this PR is planned for the 0.12 release and that we are looking for feedback (specifically from other languages).

[flying-sheep]

1. Should we put this feature behind a flag initially? I know we had spoken about that as a way forward before. However, in this case, I'm no really sure what we'd gain by doing that.

I’m not super hot on the idea myself. The whole idea behind IOSpecs was that people get a nice error message that says: “this file has been written by a newer anndata version, so use that to load it”. Now we have both that and individual feature flags for e.g. nullable arrays. The intended purpose for the latter is that people see the feature but have to opt-in. But then again they already have to (less explicitly) opt in by using df.convert_dtypes() or dtype='string'.

Now that I’m thinking about it, an alternative possible solution could have been compatibility profiles for each AnnData version with new features: adata.save('foo.h5ad', compat='0.10') would e.g. crash if there are nullable arrays in the file. And to convert to an older version, one would create an env with a new anndata version, delete or modify the incompatible entries, and convert it.

2. This PR is simple enough that it can be merged without much comment. I think we should try to make it clear that this PR is planned for the 0.12 release and that we are looking for feedback (specifically from other languages).

I don’t understand the first half: With it being milestone 0.12, we don’t backport its changelog entry to the 0.11.x branch, that’s what makes this change 0.12.

What kind of feedback are we looking for?

[ilan-gold]

What kind of feedback are we looking for?

More just going off of #673 (comment). Maybe we'll learn something but honestly, from what I can tell, this feature should not be a problem for the programming languages that has anndata support (from memory).

I’m not super hot on the idea myself.

Ok, I think since neither of us can come up with an idea, then maybe we just merge this PR and note to the community that this feature is coming, and if they want to try it out, it's on main. We will also have the integration tests to see what's up with other packages. So overall, I think a no-opt-in way forward is fine too.