Improve tests and other refactorings

[pombredanne]

This PR started with a focus on tests enhancements but does a few more things.

• The test code has been streamlined and updated. In particular:

  • test files are now grouped by test module under a testdata/test_xxx directory.
  • All the unused test files have been removed.
  • All expectedFailure and FAILING_test_xxx tests have been either removed or updates except for one test that does live API call and is marked as skipped for now.
  • All tests now run in the CI, including doctests
  • new tests have been added including
    • validation of all ABOUT files
    • help text tests

• Command line module has been streamlined and updated:

  • cmd.cli() has been renamed to cmd.about()
  • Options definitions have been streamlined and standardized including help and METAVAR
  • the command help has been reviewed and updated
  • We now use a validation callback functions for several options
  • command line error reporting functions have been refactored and made simpler.
  • The display of message at the start and end of a command run have been standardized
  • to avoid error.log file overwrite with When running multiple "inventory" command in //, error log file will be mixing several inventory errors or be deleted.  #363, it is now named after the "OUTPUT" argument as
    <OUTPUT>-error.log such that each command creates a mostly unique error log filename.
  • a new parse_key_values function is used to parse options that contain key=value such as --filter
  • internally the use_mapping and mapping_file variables and processing code common to many functions has been updated. Early in subcommands, the --mapping and --mapping-file options are evaluated. They are mutually exclusive options and a check is now done early and an error is returned if both options are used. Then the proper mapping file is used and passed around to functions as a single and optional mapping_file argument.

• inventory subcommand:

  • The --filter option parsing has been streamlined and this option is now validated early.

• attrib subcommand:

  • the syntax of Jinja template is now checked early before generating any attribution and attribution fails if the syntax is not correct.
  • internal error handling has been updated such that "generation" functions return Error objects on failure like other functions elsewhere.
  • The --mapping-output option has been removed. Mapping cannot be applied on outputs Using mapping of keys should be limited only to CSV #361
  • The --vartext option code has been streamlined and this option is now validated early. Internally it has been renamed to variable. How to make the Title of a generated Attribution file a variable #332

• gen subcommand:

  • The --license-notice-text-location option has been renamed to --reference Rename gen --license-notice-text-location #367

• Other:

  • some u'' unicode string prefixes have been removed when not needed.
  • some formatting has been applied as needed.
  • several TODO: and FIXME: comments have been added as reminders.
  • the util module has been refactored and updated. Some shared functions have been moved there and non-shared functions moved out where they are used.
  • some NOQA and pragma: nocover comment tags have been added toward future coverage and code style validation

[pombredanne]

All green and merging in develop after a chat with @chinyeungli
The saneyaml and csv transform hare part of the branch now too.