Skip to content

[WIP] Drop batch processing and store encountered advisories - #445

Closed
sbs2001 wants to merge 1 commit into
aboutcode-org:mainfrom
sbs2001:drop-batch-processing
Closed

[WIP] Drop batch processing and store encountered advisories#445
sbs2001 wants to merge 1 commit into
aboutcode-org:mainfrom
sbs2001:drop-batch-processing

Conversation

@sbs2001

@sbs2001 sbs2001 commented Apr 20, 2021

Copy link
Copy Markdown
Collaborator

This PR simplifies the code to insert the data into db by doing single inserts instead of bulk inserts.
To avoid doing duplicate work, all encountered advisories are stored. See the models for details.

TODOs :

  • Fix tests
  • Detect conflicts and log them
  • Improve speed further, by using batch inserts for inserting the advisory objects. And leverage some heuristics. (This is waaay simpler and low hanging fruit to imporve perf)

Signed-off-by: Shivam Sandbhor shivam.sandbhor@gmail.com

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
@sbs2001 sbs2001 closed this Apr 29, 2021
@sbs2001

sbs2001 commented Apr 29, 2021

Copy link
Copy Markdown
Collaborator Author

This is doing too many things at a time, I;m breaking it into smaller parts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant