Getting traceback calls & no report is generated.

[kiranravindran90]

Getting traceback calls & no report is generated.

How To Reproduce

While scanning source code files, after encountering a .sh file, this error seemed to be popping out (image below).

Many Traceback calls were followed by a "Multiprocessing Context Timeout Error"

System configuration

• OS - Windows
• ScanCode Version - 30.1.0
• Installation - Source download

[pombredanne]

@kiranravindran90 Thanks: do you have instructions so we can reproduce your setup? including a full trace?

[kiranravindran90]

@pombredanne Find the full trace & instruction details.

---

Scan Instruction :
scancode -clpi -n 2 --timeout 3600 --json .json

Package : (~15GB)
Linux based Flashwares & source code.

File types :
.bin, .bct, .dat, .dtb, .img, .cpio

Trace :
Removing temporary files...done.
Traceback (most recent call last):
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\Scripts\scancode-script.py", line 33, in
sys.exit(load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')())
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\click\core.py", line 1137, in call
return self.main(*args, **kwargs)
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\commoncode\cliutils.py", line 69, in main
return click.Command.main(
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\click\core.py", line 1062, in main
rv = self.invoke(ctx)
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\click\core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\click\decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\src\scancode\cli.py", line 451, in scancode
success, _results = run_scan(
File "c:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\src\scancode\cli.py", line 887, in run_scan
scan_success = run_scanners(
File "c:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\src\scancode\cli.py", line 1125, in run_scanners
scan_success = scan_codebase(
File "c:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\src\scancode\cli.py", line 1236, in scan_codebase
scan_timings) = next(scans)
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\click_termui_impl.py", line 116, in next
return next(iter(self))
File "C:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\lib\site-packages\commoncode\cliutils.py", line 172, in generator
for rv in self.iter:
File "c:................\scancode-toolkit-30.1.0_py39-windows\scancode-toolkit-30.1.0\src\scancode\pool.py", line 52, in wrap
return func(self, timeout=timeout or 3600)
File "C:\Program Files\Python39\lib\multiprocessing\pool.py", line 865, in next
raise TimeoutError from None
multiprocessing.context.TimeoutError

---

[Jeeppler]

This is related to: #2160

[Jeeppler]

I am facing the same issue.

OS: Linux (Debian)
Python: 3.9
Scancode version: 30.1.0
Installation: download (container Dockerfile is public)

Timeout option set to: --timeout 18000 (5 * 3600 = 18000; 3600 seconds are one hour)

However, the scan still stops after 3600 seconds (60 minutes). I looked at https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/pool.py#L52. However, I could not really understand how it works.

The first, what is this line actually doing:

func(self, timeout=timeout or 3600)

What exactly does the statement timeout or 3600 does?

Second, what is the purpose of the wrapped method/function?

[pombredanne]

@Jeeppler there is a new release of ScanCode and I would like to see if you are also experiencing the issue there. Can you try the latest 31.x?
But there is no application archive for Python 3.9 ... only 3.8 for now.
Let me review your Dockerfile at https://github.com/mercedes-benz/sechub/blob/develop/sechub-pds-solutions/scancode/docker/ScanCode-Debian.dockerfile

in https://github.com/nexB/scancode-toolkit/blob/6007301cca4eff424202abf197467ee2f004e139/src/scancode/pool.py#L52
The first, what is this line actually doing:
func(self, timeout=timeout or 3600)
What exactly does the statement timeout or 3600 does?

timeout=timeout or 3600 will assign the timeout argument with the value of the timeout variable if it has a value or with 3600 seconds (aka. 1 hour) otherwise if timeout is None or 0. It essentially provides a default timeout if none was provided.

Second, what is the purpose of the wrapped method/function? in https://github.com/nexB/scancode-toolkit/blob/6007301cca4eff424202abf197467ee2f004e139/src/scancode/pool.py#L39

I reckon it feels a bit arcane. This is a function that returns a function such that calling the inner function will always be subject to a timeout. The code is there because of historical multiprocessing bugs that have been existing in the past and a lot of trial and error to get things working correctly (obviously not enough!):

• https://bugs.python.org/issue8296 now at multiprocessing.Pool hangs when issuing KeyboardInterrupt python/cpython#52543 and was for Python 2.7 that we no longer support
• https://bugs.python.org/issue9205 now at Parent process hanging in multiprocessing if children terminate unexpectedly python/cpython#53451 and was for early Python 3 versions
  that we no longer support
• https://bugs.python.org/issue22393 now at multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly python/cpython#66587 is still an open issue in Python 3.8 and up that we actively support. There some PRs though so I would need to revisit the comments there
• https://bugs.python.org/issue38084 now at multiprocessing cannot recover from crashed worker python/cpython#82265 was for Python 3.7, 3.8 and 3.9 and has been marked as resolved but there seems to be still multiprocessing cannot recover from crashed worker python/cpython#82265 that is not solved yet

... long story made short: there are likely some changes and fixes in the code of Python that mean we may not need anymore some of this machinery. The difficult thing here is that we support Windows, macOS and Linux and they have subtly different bugs in their Python implementations at times and not-so-subtle differences on how they each handle forking and spawing processes. We support all of them :|

If you think you could help track down these issues, and have a reliably failing test case, I could may be try to simplify the code in a branch and let you test this?

[Jeeppler]

Can you try the latest 31.x?
But there is no application archive for Python 3.9 ... only 3.8 for now.

Since Debian 11 comes with Python 3.9 it would be nice to have an archive for Python 3.9 otherwise I would have to install ScanCode via pip. Are there any problems installing ScanCode via pip rather than the archive?

If you think you could help track down these issues, and have a reliably failing test case, I could may be try to simplify the code in a branch and let you test this?

I would love to help fix the bug, it includes coding something as well.

However, I was unable so far to find public test cases/files to test it with. The files I use, I am not allowed to share.
I tried it with ISO images of Linux Distributions, for example with the one from Rocky Linux. Those images are huge, but the scan takes only a few minutes.

In case you have an idea what type of binaries could work let me know. I am willing to test them.

[Jeeppler]

Related to: #2703

[Jeeppler]

Can I disable the timeout if I set it to 0? Or does it set the fallback in this case to 3600 seconds?

[Jeeppler]

I was able to reproduce the issue with the Armbian image of the Jetson Nano:

Armbian_22.08.1_Jetson-nano_jammy_current_5.18.19.img

The image takes more than an hour to scan and on Debian 11 with Scancode-Toolkit version: 30.1.0. I get the following output:

/tools/scancode-toolkit-30.1.0/src/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
 Setup plugin: scan:info...
 Setup plugin: scan:licenses...
 Setup plugin: scan:copyrights...
 Setup plugin: scan:packages...
 Setup plugin: scan:emails...
 Setup plugin: scan:urls...
 Setup plugin: output:spdx-tv...
Collect file inventory...
Scan files for: info, licenses, copyrights, packages, emails, urls with 2 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
  File "/tools/scancode-toolkit-30.1.0/bin/scancode", line 33, in <module>
    sys.exit(load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')())
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/commoncode/cliutils.py", line 69, in main
    return click.Command.main(
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tools/scancode-toolkit-30.1.0/src/scancode/cli.py", line 451, in scancode
    success, _results = run_scan(
  File "/tools/scancode-toolkit-30.1.0/src/scancode/cli.py", line 887, in run_scan
    scan_success = run_scanners(
  File "/tools/scancode-toolkit-30.1.0/src/scancode/cli.py", line 1125, in run_scanners
    scan_success = scan_codebase(
  File "/tools/scancode-toolkit-30.1.0/src/scancode/cli.py", line 1236, in scan_codebase
    scan_timings) = next(scans)
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/click/_termui_impl.py", line 116, in __next__
    return next(iter(self))
  File "/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/click/_termui_impl.py", line 328, in generator
    for rv in self.iter:
  File "/tools/scancode-toolkit-30.1.0/src/scancode/pool.py", line 52, in wrap
    return func(self, timeout=timeout or 3600)
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 865, in next
    raise TimeoutError from None
multiprocessing.context.TimeoutError

I set --timeout 18000 (5 hours). However, the scan only takes a hour and then I get the multiprocessing.context.TimeoutError.

[pombredanne]

@Jeeppler Thanks for digging further.
https://redirect.armbian.com/jetson-nano/Jammy_current is 350MB of xzipped data is massive. So I guess that we should rather skipp scanning such an image for anything but file information. We can do it just based on the size, with a default and a possible new CLI option?

What should be the default max size above which it does not make sense to attempt scanning? 100MB? 200MB?

[pombredanne]

(FWIW, the uncompressed disk image above is 1.7GB ;) )

[pombredanne]

Can I disable the timeout if I set it to 0? Or does it set the fallback in this case to 3600 seconds?

you can but that may make all fail. You can make it super short as 0.001 second if you want

[Jeeppler]

So I guess that we should rather skipp scanning such an image for anything but file information

No, no skipping of the file. The requirement is to scan files in that size or bigger for license information. It takes hours, but Scancode can find license information in files of that size.

In case, you have more questions about the use case you can reach me via email using the email address located in this file: https://github.com/mercedes-benz/sechub/blob/develop/MAINTAINERS.md

[Jeeppler]

The problem is, that even if I set the --timeout above 3600 I get the multiprocessing.context.TimeoutError after exactly one hour (3600 seconds = 1 hour). My expectation, would be that --timeout 18000 will increase the timeout duration to 5 hours and the scan might work.

I encountered the problem under Linux. It apparently works under Windows.