Skip to content

Add progress bar while importing data from importers - #419

Closed
Abhra303 wants to merge 1 commit into
aboutcode-org:mainfrom
Abhra303:progress_bar
Closed

Add progress bar while importing data from importers#419
Abhra303 wants to merge 1 commit into
aboutcode-org:mainfrom
Abhra303:progress_bar

Conversation

@Abhra303

@Abhra303 Abhra303 commented Apr 1, 2021

Copy link
Copy Markdown

This fixes the issue #318 . Here I used tqdm package to visualize the process of importing with a progress bar . Here is the screenshot below -

import_runner py - vulnerablecode  Container vulnerablecode_web (_vulnerablecode)  - Visual Studio Code 01-04-2021 15_21_11

@AmitGupta7580

Copy link
Copy Markdown
Contributor

@Abhra303 Try your code on NVD importer. As I mentioned #318 (comment) . You are only showing progress bar for saving data to database. In case of archlinux the downloading time is less (5280 vulnerabilites) thats why it does not make any difference.

@Abhra303

Abhra303 commented Apr 2, 2021

Copy link
Copy Markdown
Author

@AmitGupta7580 you're right. I didn't think about it. Trying if that can be managed.

@sbs2001

sbs2001 commented Apr 5, 2021

Copy link
Copy Markdown
Collaborator

@Abhra303 any issues ?

@Abhra303

Abhra303 commented Apr 5, 2021

Copy link
Copy Markdown
Author

Actually no severe issue.
When we ran the import command, it first fetches the data from the source then store them in the database.
The progress bar that I have implemented is for process_advisory function and this function is used to store data into database. So the progress bar is showing the progress of storing the data into the database not the progress of fetching the data from the source.
As both of these functions use for loops , they takes time to fetch the data or store the data.
So for importers such as nvd , the data they fetch is huge. As a result, when I run import command the progress bar is showing after a significant time.

Signed-off-by: Abhra303 <chakrabortyabhradeep79@gmail.com>
@Abhra303

Abhra303 commented Apr 6, 2021

Copy link
Copy Markdown
Author

User experience for import progress is improved.
Fetching vulnerability data from a source can be time consuming based on user's internet speed. We have nothing to do with it. But the "Collecting data from source..." statement will let the user know that the importer is now fetching vulnerability data from source . After fetching all data , user will see the progress bar that denotes the download of those data into database. Here is a screenshot -

import py - vulnerablecode  WSL_ Ubuntu-18 04  - Visual Studio Code 06-04-2021 14_40_49

@Abhra303

Copy link
Copy Markdown
Author

@sbs2001 can you please review the changes?

@AmitGupta7580

Copy link
Copy Markdown
Contributor

@Abhra303 Instead of just writing Collecting the data from the source, we should also have a progress bar which display the status of downloaded data, As importer is still not fully verbose.

@Abhra303

Copy link
Copy Markdown
Author

@AmitGupta7580 I was thinking of adding progress bar for collecting the data also. To do this , we have to add tqdm function in every importer file manually. Not only this, if in future other contributors make new importer files then they also have to take care of tqdm.
Another point is that if in future any issue generate related to the way importers collect data, then developers also may have to change some code related to tqdm to keep the behaviour of cli same and this have to be done in every importer file.
For these reasons, I thought that it will be better to add the progress bar only for the procress_advisory function.
But yes, If reviewers like @AmitGupta7580 's idea, I will surely add progress bar for collecting the data also..

@sbs2001

sbs2001 commented Apr 19, 2021

Copy link
Copy Markdown
Collaborator

@Abhra303 it seems to me there would be some overhead for importer implementation to get the progress bar working.

IMHO this ticket would be best resolved after we remove the batch processing things and have a uniform way for importer implementation.

@Abhra303

Copy link
Copy Markdown
Author

Thank you for the clarification.

@Abhra303 Abhra303 closed this Apr 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants