Cannot import istio

[list4nestor]

Full logs here:

# python manage.py import istio
Importing data from istio
Traceback (most recent call last):
  File "/opt/vulnerablecode/vulnerabilities/management/commands/import.py", line 110, in _import_data
    ImportRunner(importer, batch_size).run(cutoff_date=cutoff_date)
  File "/opt/vulnerablecode/vulnerabilities/import_runner.py", line 89, in run
    process_advisories(data_source)
  File "/usr/local/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/opt/vulnerablecode/vulnerabilities/import_runner.py", line 116, in process_advisories
    advisory_batches = chain(data_source.updated_advisories(), data_source.added_advisories())
  File "/opt/vulnerablecode/vulnerabilities/importers/istio.py", line 61, in updated_advisories
    processed_data = self.process_file(f)
  File "/opt/vulnerablecode/vulnerabilities/importers/istio.py", line 89, in process_file
    data = self.get_data_from_md(path)
  File "/opt/vulnerablecode/vulnerabilities/importers/istio.py", line 183, in get_data_from_md
    return saneyaml.load(front_matter)
  File "/usr/local/lib/python3.8/site-packages/saneyaml.py", line 62, in load
    return yaml.load(s, Loader=loader)
  File "/usr/local/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "yaml/_yaml.pyx", line 719, in yaml._yaml.CParser.get_single_node
yaml.composer.ComposerError: expected a single document in the stream
  in "<unicode string>", line 1, column 1
but found another document
  in "<unicode string>", line 7, column 1
Failure to import data from istio. Continuing...
CommandError: 1 failed!: istio

[pombredanne]

Thank you for the report.

[sbs2001]

Can reproduce this when using GH graphql for fetching versions.

If SVN is used this import fails even before this

✦ ❯ python manage.py import istio
Importing data from istio
svn: E170013: Unable to connect to a repository at URL 'https://github.com/istio/istio/tags'
svn: E000111: Error running context: Connection refused
Traceback (most recent call last):
  File "/home/shivam/work/vulnerablecode/vulnerabilities/management/commands/import.py", line 110, in _import_data
    ImportRunner(importer, batch_size).run(cutoff_date=cutoff_date)
  File "/home/shivam/work/vulnerablecode/vulnerabilities/import_runner.py", line 88, in run
    with data_source:
  File "/home/shivam/work/vulnerablecode/vulnerabilities/importers/istio.py", line 52, in __enter__
    self.set_api()
  File "/home/shivam/work/vulnerablecode/vulnerabilities/importers/istio.py", line 55, in set_api
    asyncio.run(self.version_api.load_api(["istio/istio"]))
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/shivam/work/vulnerablecode/vulnerabilities/package_managers.py", line 74, in load_api
    await asyncio.gather(
  File "/home/shivam/work/vulnerablecode/vulnerabilities/package_managers.py", line 467, in fetch
    tags_xml = check_output(["svn", "ls", "--xml", f"{endpoint}/tags"], text=True)
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['svn', 'ls', '--xml', 'https://github.com/istio/istio/tags']' returned non-zero exit status 1.
Failure to import data from istio. Continuing...
CommandError: 1 failed!: istio

SVN seems to be causing problems in more cases than expected.

[sbs2001]

Fixed by #570