Batch Edit: Support for editing basic fields

[sharadsw]

Fixes #5413
Uses code from #4929

⚠️ Note: This PR affects database migrations. See migration testing instructions.

This PR pretty much has the same code as #4929 but updated to production. To minimize the scope for testing, batch edit for relationships is disabled in this PR and will be enabled in the follow up PR - batch edit for relationships. There are still some merge conflicts resulting from some newer code in prod which will be resolved in the next PR.

Checklist

• Self-review the PR after opening it to make sure the changes look good
  and self-explanatory (or properly documented)
• Add automated tests
• Add relevant issue to release milestone

Testing instructions

• Create a query
• Click on Batch Edit. This will create a new Batch Edit dataset with columns involving relationships disabled (readonly)
• Change values you wish to edit
• Click Validate
• Verify changed values are highlighted as updated cells
• Click Commit. This will apply your changes
• Re-run the original query and verify the values changed as expected
• Click rollback
• Verify values were rolled back to original

[sharadsw]

POTENTIAL IMPROVEMENTS?

• Data Mapper in Batch Edit?
  • Should the Data Mapper be accessible in Batch Edit datasets? Since Batch Edit datasets can only be created from a Query, it seems confusing that users can still access the Data Mapper from a Batch Edit dataset with most of the mapper functionalities being disabled. It seems the mapper is only needed for changing Batch Edit Preferences. Could move the preferences dialog to Editor page instead.

[realVinayak]

Nice observation. My fault for keeping a hidden feature secret: Another, use for data mapper is to control “ignore when blank”-type behavior per field basis. That’s actually supported in that PR ( even for batch edit data mapper, you can modify this per-field setting. Ask grant for more context )

the only things not read only are

1. Batch edit prefs
2. Field behavior (never ignore, …)
3. must match (also, could you leave a note somewhere to make trees must match by default?)

search for usages of “readonlyspec”.

For purposes of this PR, you can theoretically not add data mapper (you also don’t need batch edit prefs, they don’t do anything when there are no relationships)

[sharadsw]

@lexiclevenger Thanks! Age should be readonly now and tree queries hopefully shouldn't have any crashes

[emenslin]

• Verify changed values are highlighted as updated cells
• Re-run the original query and verify the values changed as expected
• Verify values were rolled back to original

I feel like this shouldn't have 'workbench' in it and should just say 'Batch Edit Data Sets'

Search and replace can edit readonly fields. If you upload the changes don't carry over, however, the values do appear to be changed in the data set.

sleDTBw1iN.mp4

Results aren't showing up correctly

I didn't make any changes so I'm not sure why this roll back failed but when a rollback fails it gives an error but then shows a sucess dialog

ywSOdTzZmr.mp4

Also, the rollback success dialog should say "it cannot be edited." instead of "it cannot be edit."

Can't batch edit for taxon as it requires you to add rank -> name but you can't do that (see #5935) and on top of that it requires you to add every single rank so if you have multiple taxon trees it wants every rank from every tree. Based on this pr and what it is supposed to cover it might be worth it to just disable the batch edit button when a tree is the base table.

I'm not sure if this happens elsewhere but in this example on the accession table I had a valid pick list valueIn Process but when I clicked batch edit, in the grid it became inprocess an invalid value

PWNTOImZTg.mp4

[combs-a]

Testing instructions

• Create a query
• Click on Batch Edit. This will create a new Batch Edit dataset with columns involving relationships disabled (readonly)
• Change values you wish to edit
• Click Validate
• Verify changed values are highlighted as updated cells
• Click Commit. This will apply your changes
• Re-run the original query and verify the values changed as expected
• Click rollback
• Verify values were rolled back to original

Agree with @emenslin on the text changes, disabling the batch edit button for taxon for now.

About the rollback not being allowed when you didn't make changes, it might've been something along the lines of a required field being edited from previously having nothing. Not sure why it was missing a required field but 🤷

I was able to recreate the issue with the picklists. Looks like it's pulling from the value field and not the title field for Picklist items.

Also able to recreate the search and replace.

[realVinayak]

1. Eh, don't bother fixing search and replace, it should be clear because of highlighting that it does not actually get changed in the database. Maybe there is a setting in HOT to ignore readonly cells?

2. Ugh, results not showing up correctly might actually be a bug in HOT, I'm not sure. I think there was a related bug, I debugged and wrote it up in 4929 somewhere IIRC.

3. Hmm, if rollback fails, the "best" way to debug that would be to understand how rollback for batch edit works. When, initially, you run the query and make a dataset, the backend secretly, and shamefully, just makes two copies of the dataset. The former is the one that user modifies, and the latter is used for batch edit rollbacks. When you commit your changes, and want to roll it back, the batch edit on the later (used internally) is performed. I wanted to test that more but whatever.

This is where it happens:

specify7/specifyweb/workbench/upload/upload.py

Line 513 in 4d82298

def rollback_batch_edit(

I think what you could do, just to debug this, would be to set a breakpoint here

specify7/specifyweb/workbench/upload/upload.py

Line 588 in 4d82298

if not success:

Then, just find what row has the failure. Me and Grant talked about making the better. Technically, we know exactly what went wrong. Heck, for debugging, can you just add json of failed rows (or first failed row) -- that is, json of upload result -- to the exception it raises (don't do any fancy, just string addition)?

The next lines are needed. Basically, when you have relationships, you could have uploaded a record that you don't need anymore, so you can pretty cutely just run original workbench rollback after this special batch edit rollback.

Another thing I wanted to do (as indicated in "cannot be edited") was to make it readonly after one rollback. Do it simply by removing the backer, and if the batch edit (isupdate = true), just make it readonly (add a flag from the backend?). Theoretically, this is not needed, as the process is idempotent, fancy,

4. I'm not guaranteering any of my code works for multiple trees, I didn't get to finish that up 😉. But, originally, no it doesn't ask for every rank. If you selected species (and if species is the last rank), it will not ask for more ranks. If you selected genus, and there are subspecies and species rank below it, then it would force you to select species and subspecies. Basically, all the ranks below the highest rank need to be selected. It gets complicated in cases of relationships (determinations -> taxon) otherwise.

Also, don't straight up disable in case of taxon. Only do that if there is a rank also selected. It differentiates between the two cases. I'd recommend going back to original PR and maybe ask Grant, if he has time, on how it differentiates them. There's a somewhat generic logic for it, shouldn't be hard to find, in the BatchEdit/index.tsx file.

5. I swear I tested picklists, and I thought it worked! In any case, a somewhat decent way (which could actually fix another issue floating around which asks for seeing title of picklist) would be to add a "case when" here

   specify7/specifyweb/stored_queries/format.py

   Lines 329 to 336 in 4d82298

   	def _fieldformat(self, specify_field: Field,
   	field: Union[InstrumentedAttribute, Extract],
   	format_name: Optional[str] = None):
   	if self.format_agent_type and specify_field is Agent_model.get_field("agenttype"):
   	cases = [(field == _id, name) for (_id, name) in enumerate(agent_types)]
   	_case = case(cases)
   	return blank_nulls(_case) if self.replace_nulls else _case

Specifically, after 4929, you can format agent type as actual text (needed for batch edit). Just add similar code for picklists too (with a prop for "format_picklist"). Lmk if you got a better implementation. You could, theoretically, do it at parsing later on. But, I'd recommend it doing it here.

I'm sorry for putting you through the last one, completely my fault. I really thought that picklist worked.