[Bug] PDF Reports

[Dexus]

Describe the bug
A upstream bug, that lets not create PDF if you not use the default report settings.

To Reproduce
Steps to reproduce the behavior:
...

Expected behavior
Create PDF with all kind of settings.

Screenshots

WORKING:

NOT WORKING:

Image in use:
current latest

Additional
GB Ticket: AP-1987

[markdesilva]

Actually if I try to show a report in any format (CSV, PDF, XML, TXT, etc) all the reports are empty.

Is this related to the same bug?

[Dexus]

I'm not sure I use in general only PDF Format. Will check on Monday if there are other known issues about this topic.
But to be clear, no it should not as far as I know.

[markdesilva]

I'm not sure I use in general only PDF Format. Will check on Monday if there are other known issues about this topic.

Yes I use PDF only as well, previously with the fix in SSC #249

#!/bin/sh
echo " --> updmap-sys --syncwithtrees"
yes 2> /dev/null | updmap-sys --syncwithtrees > /dev/null 2>&1 > /dev/null
echo " --> updmap-sys"
updmap-sys > /dev/null 2>&1 > /dev/null
echo " --> mktexlsr"
mktexlsr > /dev/null 2>&1 > /dev/null
echo " --> texhash"
texhash > /dev/null 2>&1 > /dev/null
echo " --> fmtutil-sys --all"
fmtutil-sys --all > /dev/null 2>&1 > /dev/null
echo " --> updmap-sys --force"
updmap-sys --force > /dev/null 2>&1 > /dev/null
exit 0

This fixed the PDF reports only, the other reports were working fine. But now all the reports are empty, so I'm wondering if it was related to this upstream bug.

Thanks for checking!

[Dexus]

Are you using the default config on the webpage? Or did you set some different filters?

[markdesilva]

Are you using the default config on the webpage? Or did you set some different filters?

Defaults config.

apply_overrides=1 levels=hml rows=100 min_qod=70 first=1 sort-reverse=severity

Includes Notes and Overrides.

[markdesilva]

Should I run the /opt/setup/script/reportFix.sh?

[Dexus]

Should I run the /opt/setup/script/reportFix.sh?

Never used this script, and have no idea if this will help or for which exact problem it was ever created.

[markdesilva]

Never used this script, and have no idea if this will help or for which exact problem it was ever created.

Its a GVM Reporting fix for lines over a certain amount. It says:

"This tool allows you to modify the max_rows_per_page setting. A larger number will allow you to export more data, but it will make the web UI load much slower. Any scan with more than 15000 results should be broken into multiple scans."

Won't run this then.

[markdesilva]

Have been testing since early this morning. All report formats result in 0byte downloads.

Notifications that are supposed to attach reports don't have any attachments.

My docker run has OPT_PDF="1" which has installed the necessary texlive deps:

ii  texlive-base                2020.20210202-3                all          TeX Live: Essential programs and files
ii  texlive-binaries            2020.20200327.54578-7          amd64        Binaries for TeX Live
ii  texlive-fonts-recommended   2020.20210202-3                all          TeX Live: Recommended fonts
ii  texlive-latex-base          2020.20210202-3                all          TeX Live: LaTeX fundamental packages
ii  texlive-latex-extra         2020.20210202-3                all          TeX Live: LaTeX additional packages
ii  texlive-latex-recommended   2020.20210202-3                all          TeX Live: LaTeX recommended packages
ii  texlive-pictures            2020.20210202-3                all          TeX Live: Graphics, pictures, diagrams

Secure-Compliance-Solutions-LLC/GVM-Docker#249 does not fix the situation.

Thanks.

[Dexus]

can you move in you container to /var/lib/gvm/gvmd/report_formats/11e2bb5a-5dbf-496f-abe6-7ebb301aae68
and make a ls -alsR

root@36f20adb1648:/var/lib/gvm/gvmd/report_formats/11e2bb5a-5dbf-496f-abe6-7ebb301aae68# ls -alsR
.:
total 32
4 drwxr-xr-x 8 gvm gvm 4096 Mar 22 13:20 .
4 drwxr-xr-x 3 gvm gvm 4096 Mar 22 13:20 ..
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 5057e5cc-b825-11e4-9d0e-28d24461215b
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 77bd6c4a-1f62-11e1-abf0-406186ea4fc5
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 a3810a62-1f62-11e1-9219-406186ea4fc5
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 a994b278-1f62-11e1-96ac-406186ea4fc5
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 c1645568-627a-11e3-a660-406186ea4fc5
4 drwxr-xr-x 2 gvm gvm 4096 Apr  1 07:40 c402cc3e-b531-11e1-9163-406186ea4fc5

./5057e5cc-b825-11e4-9d0e-28d24461215b:
total 24
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 .
4 drwxr-xr-x 8 gvm gvm 4096 Mar 22 13:20 ..
8 -rw-r--r-- 1 gvm gvm 6520 Mar 22 13:20 Anonymous_XML.xsl
4 -rwxr-xr-x 1 gvm gvm  904 Mar 22 13:20 generate
4 -rw-r--r-- 1 gvm gvm  394 Mar 22 13:20 report_format.xml

./77bd6c4a-1f62-11e1-abf0-406186ea4fc5:
total 20
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 .
4 drwxr-xr-x 8 gvm gvm 4096 Mar 22 13:20 ..
4 -rwxr-xr-x 1 gvm gvm 1010 Mar 22 13:20 generate
4 -rw-r--r-- 1 gvm gvm 1587 Mar 22 13:20 ITG.xsl
4 -rw-r--r-- 1 gvm gvm  492 Mar 22 13:20 report_format.xml

./a3810a62-1f62-11e1-9219-406186ea4fc5:
total 60
 4 drwxr-xr-x 2 gvm gvm  4096 Mar 22 13:20 .
 4 drwxr-xr-x 8 gvm gvm  4096 Mar 22 13:20 ..
 4 -rwxr-xr-x 1 gvm gvm  1029 Mar 22 13:20 generate
 4 -rw-r--r-- 1 gvm gvm   466 Mar 22 13:20 report_format.xml
44 -rw-r--r-- 1 gvm gvm 41283 Mar 22 13:20 TXT.xsl

./a994b278-1f62-11e1-96ac-406186ea4fc5:
total 16
4 drwxr-xr-x 2 gvm gvm 4096 Mar 22 13:20 .
4 drwxr-xr-x 8 gvm gvm 4096 Mar 22 13:20 ..
4 -rwxr-xr-x 1 gvm gvm  961 Mar 22 13:20 generate
4 -rw-r--r-- 1 gvm gvm  357 Mar 22 13:20 report_format.xml

./c1645568-627a-11e3-a660-406186ea4fc5:
total 32
 4 drwxr-xr-x 2 gvm gvm  4096 Mar 22 13:20 .
 4 drwxr-xr-x 8 gvm gvm  4096 Mar 22 13:20 ..
16 -rw-r--r-- 1 gvm gvm 15120 Mar 22 13:20 CSV_Results.xsl
 4 -rwxr-xr-x 1 gvm gvm   880 Mar 22 13:20 generate
 4 -rw-r--r-- 1 gvm gvm   659 Mar 22 13:20 report_format.xml

./c402cc3e-b531-11e1-9163-406186ea4fc5:
total 84
 4 drwxr-xr-x 2 gvm gvm  4096 Apr  1 07:40 .
 4 drwxr-xr-x 8 gvm gvm  4096 Mar 22 13:20 ..
 4 -rwxr-xr-x 1 gvm gvm  1512 Apr  1 07:40 generate
68 -rw-r--r-- 1 gvm gvm 69517 Mar 22 13:20 latex.xsl
 4 -rw-r--r-- 1 gvm gvm   482 Mar 22 13:20 report_format.xml

[markdesilva]

can you move in you container to /var/lib/gvm/gvmd/report_formats/11e2bb5a-5dbf-496f-abe6-7ebb301aae68
and make a ls -alsR

I have no such directory.

root@c2fa4cdc630f:~ # cd /var/lib/gvm/gvmd/report_formats/11e2bb5a-5dbf-496f-abe6-7ebb301aae68
bash: cd: /var/lib/gvm/gvmd/report_formats/11e2bb5a-5dbf-496f-abe6-7ebb301aae68: No such file or directory
root@c2fa4cdc630f:~ # cd /var/lib/gvm/gvmd/report_formats
root@c2fa4cdc630f:/var/lib/gvm/gvmd/report_formats # ls
root@c2fa4cdc630f:/var/lib/gvm/gvmd/report_formats # ls -ailhR
.:
total 8.0K
1318923 drwxr-xr-x 2 gvm gvm 4.0K Apr 24 18:04 .
1318920 drwxr-xr-x 4 gvm gvm 4.0K Apr 24 18:37 ..
root@c2fa4cdc630f:/var/lib/gvm/gvmd/report_formats #

Have I done something wrong?

[Dexus]

No but there is a difference between the docs and the code. (#15)

Please use the code from https://github.com/DeineAgenturUG/greenbone-gvm-openvas-for-docker/blob/devel/GVMDocker/scripts/sync-all.sh and add it to /opt/setup/scripts/sync-all.sh to fix it please run also the script.

Need later to release the fixes to the latest images.

[markdesilva]

Thanks @Dexus, I’m not at my desk at the moment will come back and do it.

[markdesilva]

I copied the code from https://github.com/DeineAgenturUG/greenbone-gvm-openvas-for-docker/blob/devel/GVMDocker/scripts/sync-all.sh to my container /opt/setup/scripts/sync-all.sh. The only difference between them is:

root@c2fa4cdc630f:/opt/setup/scripts # diff sync-all.sh sync-all.sh.org
2,3c2,3
< set -Eeo pipefail
< if [[ ! "${AUTO_SYNC}" =~ ^(yes|y|YES|Y|true|TRUE)$ ]]; then
---
> set -Eeuo pipefail
> if [[ "${AUTO_SYNC}" != "YES" ]]; then

I ran the script, it did some updates, but my /var/lib/gvm/gvmd/report_formats is still empty.

In my feed status, it says all the feeds are updating. After updating, I tried to generate different reoprts (TXT, CSV, etc) they are all still 0 bytes.

[Dexus]

is the folder still empty?

maybe I found the bug.
It looks like latex exits with error 1 because of

/tmp/tmp.g2YW3e2mf9/report.tex:235: Misplaced \noalign.
\rowcolor ->\noalign 
                     {\ifnum 0=`}\fi \global \let \CT@do@color \CT@@do@color...
l.235 \rowcolor
               {white}{\verb=Services=}\\
/tmp/tmp.g2YW3e2mf9/report.tex:235:  ==> Fatal error occurred, no output PDF file produced!

But this is more a upstream bug, so I'm not able to do here something else as wait... not sure if it is latex bug or one for the greenbone side.

[markdesilva]

Yes, still empty.

root@c2fa4cdc630f:/var/lib/gvm/gvmd/report_formats # ls -ailh
total 8.0K
1318923 drwxr-xr-x 2 gvm gvm 4.0K Apr 24 18:04 .
1318920 drwxr-xr-x 4 gvm gvm 4.0K Apr 24 18:37 ..

I even restarted the container and let it update again.

[Dexus]

what does the gvmd log tells you? /var/log/gvm/gvmd.log`

If there is no error to find, please try it with a complete clean system (all volume should be empty) and change after the feeds are synced and ~10 minutes idle time the database folder to your updated db and restart the container. I see this very very rarely and most when the system was stopped before the sync was done, that files are missing.

[markdesilva]

I see a whole bunch of these messages in /var/log/gvm/gvmd.log, which corresponds to every time my tasks try to generate a report or I manually try to generate a report of any type (TXT, CSV, PDF, etc).

md manage:WARNING:2022-04-25 02h32.42 +08:3897: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 02h33.03 +08:3909: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/a3810a62-1f62-11e1-9219-406186ea4fc5/generate
md manage:WARNING:2022-04-25 02h34.03 +08:4288: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 02h37.03 +08:4364: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/5057e5cc-b825-11e4-9d0e-28d24461215b/generate
md manage:WARNING:2022-04-25 02h37.38 +08:4383: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 02h39.52 +08:4449: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/a3810a62-1f62-11e1-9219-406186ea4fc5/generate
md manage:WARNING:2022-04-25 02h40.28 +08:4589: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 02h40.43 +08:4600: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c1645568-627a-11e3-a660-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h07.03 +08:5275: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h07.59 +08:5481: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h24.32 +08:6354: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h24.55 +08:6370: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/a3810a62-1f62-11e1-9219-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h38.20 +08:1118: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h40.07 +08:1219: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h46.56 +08:2610: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h47.05 +08:2671: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 03h59.42 +08:42514: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h03.50 +08:69296: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h07.00 +08:82231: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h07.47 +08:87860: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h07.49 +08:87823: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h19.21 +08:4127: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h47.12 +08:109642: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h47.42 +08:109550: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 04h53.46 +08:109460: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 05h35.43 +08:100097: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 06h00.56 +08:1879: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 06h22.29 +08:92130: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 06h23.53 +08:99018: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 08h04.21 +08:876576: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 09h52.39 +08:93147: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 11h58.50 +08:25464: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 19h57.19 +08:1121283: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 19h58.30 +08:1121508: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
md manage:WARNING:2022-04-25 19h59.07 +08:1121531: run_report_format_script: No generate script found at /var/lib/gvm/gvmd/report_formats/27127ef5-e2b9-4d04-a1ed-368e97fd8448/a3810a62-1f62-11e1-9219-406186ea4fc5/generate

[markdesilva]

Ok will try this after dinner - thank you!

please try it with a complete clean system (all volume should be empty) and change after the feeds are synced and ~10 minutes idle time the database folder to your updated db and restart the container. I see this very very rarely and most when the system was stopped before the sync was done, that files are missing.

[Dexus]

Ok will try this after dinner - thank you!

please try it with a complete clean system (all volume should be empty) and change after the feeds are synced and ~10 minutes idle time the database folder to your updated db and restart the container. I see this very very rarely and most when the system was stopped before the sync was done, that files are missing.

maybe a gvmd --rebuild-gvmd-data=report_formats help there also

gvmd --help
...
 --rebuild                                     Remove NVT db, and rebuild it from the scanner.
  --rebuild-gvmd-data=<types>                   Reload all gvmd data objects of a given types from feed. The types must be "all" or a comma-separated of the following: "configs", "port_lists" and "report_formats"
  --rebuild-scap                                Rebuild all SCAP data.
...

[markdesilva]

I tried gvmd --rebuild-gvmd-data=report_formats but it didn't work. I even tried gvmd --rebuild and it didn't work also.

But then, I think I found the issue.

The directory name for report generation directory (/var/lib/gvm/gvmd/report_formats/<report_generation_directory>) is stored in the database. When I use the old database, it somehow doesn't create the directory.

I checked my previous version of gvm container and found that the <report_generation_directory> name is the same as what was showing up in the warnings in the gvmd.log in the new container (27127ef5-e2b9-4d04-a1ed-368e97fd8448). I recalled from the SC github pages, some one (was it you or pixel) mentioned that the location of the generate scripts for all the report types is stored in the pg db. So I made a copy of 27127ef5-e2b9-4d04-a1ed-368e97fd8448 and all its sub-directories and contents from my old container and put it into the new container in /var/lib/gvm/gvmd, set the permissions, the owner and group to gvm and now all reports are working properly, including PDF.

Thanks for the help!

[Dexus]

btw. why did you not reuse the volumes from the old one? only the DB volume need to be changed after the update to match PGSql 13.

[markdesilva]

btw. why did you not reuse the volumes from the old one? only the DB volume need to be changed after the update to match PGSql 13.

You’re right! Stupid of me. Sorry to waste your time on this issue, it was not intentional. Completely my fault.

[Dexus]

No problem.