Discrepancy between DANDI CLI validate and website/archive validate?

[CodyCBakerPhD]

@yarikoptic @waxlamp

Not really sure how this works on the technology side of things, but Dandiset 000689 has a subset of assets claiming a vague validation error but not really explaining why: https://dandiarchive.org/dandiset/000689

Yet these files passed validation at time of upload via the CLI (it was not forced via any overrides)

cc @laurelrr

[bendichter]

Maybe it's expecting the session start time to have a timezone?

[CodyCBakerPhD]

Possibly - for reference, this is what the message on the website looks like

[CodyCBakerPhD]

Note @yarikoptic @waxlamp that this might be some of the first files that leverage NWB's latest change to relax timestamp and timezone requirements from session start time

[laurelrr]

One thing that is unique about this set of files is that I re-ran the conversion for them yesterday using an updated pynwb (version 2.3.3).

[bendichter]

There are lots of ways we can get the session start time. We can pull it from a pickle file, we can pull it from ecephys metadata, or we can pull it from a string in a data directory. Whatever the case, all of these files that are failing validation do have a session start time (here indicated as startDate)

sub-H14/sub-H14_ses-14-2019-08-30-09-21-05-Disc4_ecephys+image.nwb
wasGeneratedBy.0 - {'name': '14-2019-08-30-09-21-05-Disc4', 'schemaKey': 'Session', 'startDate': '2019-08-30T00:00:00', 'identifier': '14-2019-08-30-09-21-05-Disc4', 'description': 'Auto-generated by neuroconv'} is not valid under any of the given schemas
sub-H27/sub-H27_ses-H27-2020-02-20-10-07-01-Disc4-20k_ecephys+image.nwb
wasGeneratedBy.0 - {'name': 'H27-2020-02-20-10-07-01-Disc4-20k', 'schemaKey': 'Session', 'startDate': '2020-02-20T00:00:00', 'identifier': 'H27-2020-02-20-10-07-01-Disc4-20k', 'description': 'Auto-generated by neuroconv'} is not valid under any of the given schemas
sub-H31/sub-H31_ses-H31-2020-04-30-09-56-31-Disc7_ecephys+image.nwb
wasGeneratedBy.0 - {'name': 'H31-2020-04-30-09-56-31-Disc7', 'schemaKey': 'Session', 'startDate': '2020-04-30T00:00:00', 'identifier': 'H31-2020-04-30-09-56-31-Disc7', 'description': 'Auto-generated by neuroconv'} is not valid under any of the given schemas
sub-H41/sub-H41_ses-H41-2020-04-29-16-55-26-Disc5-9_ecephys+image.nwb
wasGeneratedBy.0 - {'name': 'H41-2020-04-29-16-55-26-Disc5-9', 'schemaKey': 'Session', 'startDate': '2020-04-29T16:55:26', 'identifier': 'H41-2020-04-29-16-55-26-Disc5-9', 'description': 'Auto-generated by neuroconv'} is not valid under any of the given schemas
sub-H42/sub-H42_ses-H42-2020-04-29-16-55-04-Disc5-5_ecephys+image.nwb
wasGeneratedBy.0 - {'name': 'H42-2020-04-29-16-55-04-Disc5-5', 'schemaKey': 'Session', 'startDate': '2020-04-29T00:00:00', 'identifier': 'H42-2020-04-29-16-55-04-Disc5-5', 'description': 'Auto-generated by neuroconv'} is not valid under any of the given schemas

Up until recently, PyNWB would by default add the local timezone to any data file that did not have one. This happened a lot, because while we can often pull the session start time from metadata, this data rarely contains the timezone. We stopped doing that, and starting allowing NWB files to have a session start time without a timezone. I am concerned that the errors here are because when DANDI tries to validate the metadata it extracts from the NWB files, it is trying to match the metadata to a regex that contains a timezone, and it is failing to do so.

If that is the case, this particular problem could be solved by adding a timezone to each of the session_start_time's and re-uploading. However, that would mean this problem is likely to show up for a lot more data so we'll need to figure out a way to handle this between NWB and DANDI.

[laurelrr]

I see. Is there a straightforward way to add timezone info to that field, something like using the HD5 context usage as I did to correct the author name?

[bendichter]

yeah, probably. Let me take a look

[bendichter]

I am still trying to download one of your files to make sure, but I think this should work:

import h5py

with h5py.File(fpath, "a") as file:
    file["session_start_time"][()] = file["session_start_time"][()] + b'-07:00'

[satra]

for what it's worth, dandischema validates those asset metadata fine. i suspect this is happening on the server side for some reason.

[CodyCBakerPhD]

OK, I found the source of the issue. There are actually two issues here...

The first is related to the .asf file discussion on Slack; these NWB files have an external file (you can see via neurosift - acquisition / Video : 14_20190830_Disc4) but the dandiset has no such file

Discovered via the following process:

• downloaded one of the currently 'invalid' files from DANDI
• made a dandiset on staging
• attempting to organize, validate, and upload the file to that staging set

Doing

dandi organize "..\sub-H14_ses-14-2019-08-30-09-21-05-Disc4_ecephys+image.nwb" --update-external-file-paths --files-mode copy --media-files-mode copy 

if the video path is absolute causes a printout to stdout

external file .../xyz.asf should be one of: [".mp4", ".avi", ".wmv", ".mov", ".flv", ".mkv"]

but nothing errors or prevents moving forward; dandi validate passes and dandi upload proceeds but silently skips the video file

So I'd say the more prominent issue is that dandi validate or upload do not check and warn/error about non-supported file types for external files

[CodyCBakerPhD]

The first issue is easy for @laurelrr to resolve; just do that .mp4 conversion and then re-organize and re-upload to the video files onto DANDI

The second issue that prompted this discussion; adding a timezone to the datetime does seem to have resolved the asset metadata issue on the landing page of my staging server (even though the asset should still be invalid due to the missing file)

[laurelrr]

This is interesting, though, that dandi ultimately threw an error for the files I processed with the newer version of pynwb. In fact, there were 26 of the 33 files with this incorrect file format and those processed with an earlier pynwb version did not produce the validation issue.