memoryerror in output plugin

[chinyeungli]

Description

scancode fails to write to the output for scanning ~200k files.

ERROR: failed to run output plugin: csv:
Traceback (most recent call last):
  File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 82, in process_codebase
    write_csv(results, csv)
  File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 100, in write_csv
    rows = list(flatten_scan(results, headers))
  File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 145, in flatten_scan
    file_info['scan_errors'] = '\n'.join(errors)
  File "C:\Python27\Lib\collections.py", line 78, in __setitem__
    last[1] = root[0] = self.__map[key] = [last, root, key]
MemoryError

ERROR: failed to run output plugin: json-pp:
Traceback (most recent call last):
  File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 119, in process_codebase
    results = get_results(codebase, as_list=False, **kwargs)
  File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 161, in get_results
    files = OutputPlugin.get_files(codebase, **kwargs)
  File "c:\scancode-toolkit\src\plugincode\output.py", line 104, in get_files
    return map(serializer, resources)
MemoryError

Scanning done.
Summary:        info with 4 process(es)
Errors count:   0
Scan Speed:     73.85 files/sec. 1.50 MB/sec.
Initial counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s)
Final counts:   204473 resource(s): 187137 file(s) and 17336 directorie(s) for 3.72 GB
Timings:
  scan_start: 2020-05-07T025407.619000
  scan_end:   2020-05-07T043955.115000
  inventory: 431.73s
  scan: 2533.86s
  output:csv: 107.15s
  output:json-pp: 55.21s
  output: 162.36s
  total: 6570.06s

How To Reproduce

The command that I used was

scancode -n4 -i <project> --json-pp output.json --csv output.csv