From 50f61b502bfe4118b33445cdd4c8c522d4da87f5 Mon Sep 17 00:00:00 2001 From: mariojmdavid Date: Wed, 24 Nov 2021 16:45:50 +0000 Subject: [PATCH] issue #67, substitute SHALL by MUST --- content/06.quality_criteria.md | 301 ++++++++++++++------------------- 1 file changed, 125 insertions(+), 176 deletions(-) diff --git a/content/06.quality_criteria.md b/content/06.quality_criteria.md index 1d264fa..47307da 100644 --- a/content/06.quality_criteria.md +++ b/content/06.quality_criteria.md @@ -14,59 +14,59 @@ reducing the likelihood of service disruption. ## 4.1. Code Accessibility [QC.Acc] -* **[QC.Acc01]** Following the open-source model, the source code being produced MUST be open and - publicly available to promote the adoption and augment the visibility of the software +* **[QC.Acc01]** Following the open-source model, the source code being produced **MUST** be open + and publicly available to promote the adoption and augment the visibility of the software developments. -* **[QC.Acc02]** Source code MUST use a Version Control System (VCS). +* **[QC.Acc02]** Source code **MUST** use a Version Control System (VCS). - * **[QC.Acc02.1]** It is RECOMMENDED that all software components delivered by the same project - agree on a common VCS. + * **[QC.Acc02.1]** It is **RECOMMENDED** that all software components delivered by the same + project agree on a common VCS. -* **[QC.Acc03]** Source code produced within the scope of a broader development project SHOULD +* **[QC.Acc03]** Source code produced within the scope of a broader development project **SHOULD** reside in a common organization of a version control repository hosting service. ## 4.2. Licensing [QC.Lic] -* **[QC.Lic01]** As open-source software, source code MUST adhere to an open-source license to be - freely used, modified and distributed by others. +* **[QC.Lic01]** As open-source software, source code **MUST** adhere to an open-source license to + be freely used, modified and distributed by others. * **[QC.Lic01.1]** Non-licensed software is exclusive copyright by default. -* **[QC.Lic02]** License MUST be compliant with the Open Source Definition +* **[QC.Lic02]** License **MUST** be compliant with the Open Source Definition [@https://opensource.org/osd]. - * **[QC.Lic02.1]** RECOMMENDED licenses are listed in the Open Source Initiative portal under the - Popular Licenses category [@https://opensource.org/licenses]. + * **[QC.Lic02.1]** **RECOMMENDED** licenses are listed in the Open Source Initiative portal under + the Popular Licenses category [@https://opensource.org/licenses]. -* **[QC.Lic03]** Licenses MUST be physically present (e.g. as a LICENSE file) in the root of all the - source code repositories related to the software component. +* **[QC.Lic03]** Licenses **MUST** be physically present (e.g. as a LICENSE file) in the root of all + the source code repositories related to the software component. ## 4.3. Code Style [QC.Sty] Code style requirements pursue the correct maintenance of the source code by the common agreement of a series of style conventions. These vary based on the programming language being used. -* **[QC.Sty01]** Each individual software product MUST comply with community-driven or de-facto code - style standards for the programming languages being used. +* **[QC.Sty01]** Each individual software product **MUST** comply with community-driven or de-facto + code style standards for the programming languages being used. - * **[QC.Sty01.1]** Compliance with multiple complementary standards MAY exist. + * **[QC.Sty01.1]** Compliance with multiple complementary standards **MAY** exist. -* **[QC.Sty02]** Custom code style guidelines SHOULD be avoided, only considered in the hypothetical - event of programming languages without existing community style standards. +* **[QC.Sty02]** Custom code style guidelines **SHOULD** be avoided, only considered in the + hypothetical event of programming languages without existing community style standards. - * **[QC.Sty02.1]** Custom styles MUST be documented by defining each convention and its expected - output. + * **[QC.Sty02.1]** Custom styles **MUST** be documented by defining each convention and its + expected output. - * **[QC.Sty02.2]** Custom styles SHOULD evolve over time towards a more consistent definition. + * **[QC.Sty02.2]** Custom styles **SHOULD** evolve over time towards a more consistent definition. * **[QC.Sty03]** Exceptions of individual conventions from the main definition are allowed but - SHOULD be avoided + **SHOULD** be avoided - * **[QC.Sty03.1]** Absence of standard conventions SHOULD be justified and tracked. + * **[QC.Sty03.1]** Absence of standard conventions **SHOULD** be justified and tracked. -* **[QC.Sty04]** Code style compliance testing MUST be automated and MUST be triggered for each - candidate change in the source code. +* **[QC.Sty04]** Code style compliance testing **MUST** be automated and **MUST** be triggered for + each candidate change in the source code. ## 4.4. Code metadata [QC.Met] @@ -75,8 +75,8 @@ software citation viable [@doi:10.7717/peerj-cs.86]. It allows the assignment of Identifier (DOI) and is key towards preservation, discovery, reuse, and attribution of the software component. -* **[QC.Met01]** A metadata file SHOULD exist along side the code, under its VCS. The metadata file - SHOULD be updated when needed, as is the case of a new version. +* **[QC.Met01]** A metadata file **SHOULD** exist along side the code, under its VCS. The metadata + file **SHOULD** be updated when needed, as is the case of a new version. ## 4.5. Unit Testing [QC.Uni] @@ -84,24 +84,22 @@ Unit testing evaluates all the possible flows in the internal design of the code behavior becomes apparent. It is a key type of testing for early detection of failures in the development cycle. -* **[QC.Uni01]** Minimum acceptable code coverage threshold SHOULD be 70%. +* **[QC.Uni01]** Minimum acceptable code coverage threshold **SHOULD** be 70%. - * **[QC.Uni01.1]** Unit testing coverage SHOULD be higher for those sections of the code + * **[QC.Uni01.1]** Unit testing coverage **SHOULD** be higher for those sections of the code identified as critical by the developers, such as units part of a security module. - * **[QC.Uni01.2]** Unit testing coverage MAY be lower for external libraries or pieces of code not - maintained within the product’s code base. + * **[QC.Uni01.2]** Unit testing coverage **MAY** be lower for external libraries or pieces of code + not maintained within the product’s code base. -* **[QC.Uni02]** Units SHOULD reside in the repository code base but separated from the main code. +* **[QC.Uni02]** Units **SHOULD** reside in the repository code but separated from the main code. -* **[QC.Uni03]** Unit testing coverage MUST be checked on change basis. +* **[QC.Uni03]** Unit testing coverage **MUST** be checked on change basis. -* **[QC.Uni04]** Unit testing coverage MUST be automated. +* **[QC.Uni04]** Unit testing coverage **MUST** be automated. * **[QC.Uni04.1]** When working on automated testing, the use of testing doubles is - RECOMMENDED to mimic a simplistic behavior of objects and procedures (c.f. section 4.6.). - -## **PROPOSAL** + **RECOMMENDED** to mimic a simplistic behavior of objects and procedures (c.f. section 4.6.). ## 4.6. Test Harness [QC.Har] @@ -136,46 +134,16 @@ purposes. There are various kinds of double [@isbn:9780131495050]: As such the following criteria is defined for Test Harness: -* **[QC.Har01]** When working on automated testing, the use of Test Doubles is RECOMMENDED to mimic - a simplistic behavior of objects and procedures. +* **[QC.Har01]** When working on automated testing, the use of Test Doubles is **RECOMMENDED** to + mimic a simplistic behavior of objects and procedures. -* **[QC.Har02]** Test Doubles SHOULD reside in the software component repository code base but +* **[QC.Har02]** Test Doubles **SHOULD** reside in the software component repository code base but separated from the main code. -* **[QC.Har03]** Regression testing, that checks the conformance with previous tests, is covered at - this stage by executing the complete set of Test Doubles available. - -* **[QC.Har04]** Test Doubles and regression, MUST be checked on change basis. - -## 4.XX. Functional Testing [QC.Fun] - **TO BE REMOVED/CAN BE REMOVED** - -Functional testing is a type of black-box testing. -It involves the verification of the software component’s -identified functionality, based on requested requirements and agreed design -specifications. This type of software testing focus on the evaluation of the -functionality that the software component exposes, leaving apart any internal -design analysis or side-effects to external systems. - -* **[QC.Fun01]** Functional testing SHOULD tend to cover the full set of functionality that - the software component claims to provide. **[SvcQC.Fun01]** - -* **[QC.Fun02]** Functional tests SHOULD be checked automatically with the exception of those - functionality that require human interaction, such as Graphical User - Interfaces (GUI). **[SvcQC.Fun02]** - -* **[QC.Fun03]** When working on automated testing, the use of testing doubles is RECOMMENDED - to mimic a simplistic behavior of objects and procedures. - **TEST HARNESS** - -* **[QC.Fun04]** Functional tests SHOULD reside in the software component repository code - base but separated from the main code. **[SvcQC.Fun03]** **TEST HARNESS** - -* **[QC.Fun05]** Regression testing, that checks the conformance with previous tests, is - covered at this stage by executing the complete set of functional tests available. **TEST HARNESS** +* **[QC.Har03]** Regression testing, that checks the conformance with previous tests, **SHOULD** be + covered at this stage by executing the complete set of Test Doubles available. -* **[QC.Fun06]** Functional and regression testing MUST be checked on change basis. **TEST HARNESS** - -* **[QC.Fun07]** Functional and regression testing coverage MAY NOT be suitable for automated testing. **REMOVE** +* **[QC.Har04]** Test Doubles and regression, **MUST** be checked on change basis. ## 4.7. Test-Driven Development [QC.Tdd] @@ -184,33 +152,33 @@ requirements being converted to test cases before software is fully developed, a software development by repeatedly testing the software against all test cases. This is opposed to software being developed first and test cases created later. -* **[QC.Tdd01]** Software requirements SHOULD be converted to test cases, and these test cases - SHOULD be checked automatically. +* **[QC.Tdd01]** Software requirements **SHOULD** be converted to test cases, and these test cases + **SHOULD** be checked automatically. ## 4.8. Documentation [QC.Doc] -* **[QC.Doc01]** Documentation MUST be treated as code. +* **[QC.Doc01]** Documentation **MUST** be treated as code. - * **[QC.Doc01.1]** Version controlled, it MAY reside in the same repository where the source code - lies. + * **[QC.Doc01.1]** Version controlled, it **MAY** reside in the same repository where the source + code lies. -* **[QC.Doc02]** Documentation MUST use plain text format using a markup language, such as Markdown - or reStructuredText. +* **[QC.Doc02]** Documentation **MUST** use plain text format using a markup language, such as + Markdown or reStructuredText. - * **[QC.Doc02.1]** It is RECOMMENDED that all software components delivered by the same project - agree on a common markup language. + * **[QC.Doc02.1]** It is **RECOMMENDED** that all software components delivered by the same + project agree on a common markup language. -* **[QC.Doc03]** Documentation MUST be online and available in a documentation repository. +* **[QC.Doc03]** Documentation **MUST** be online and available in a documentation repository. - * **[QC.Doc03.1]** Documentation SHOULD be rendered automatically. + * **[QC.Doc03.1]** Documentation **SHOULD** be rendered automatically. -* **[QC.Doc04]** Documentation MUST be updated on new software versions involving any substantial or - minimal change in the behavior of the application. +* **[QC.Doc04]** Documentation **MUST** be updated on new software versions involving any + substantial or minimal change in the behavior of the application. -* **[QC.Doc05]** Documentation MUST be updated whenever reported as inaccurate or unclear. +* **[QC.Doc05]** Documentation **MUST** be updated whenever reported as inaccurate or unclear. -* **[QC.Doc06]** Documentation MUST be produced according to the target audience, varying according - to the software component specification. The identified types of documentation and their +* **[QC.Doc06]** Documentation **MUST** be produced according to the target audience, varying + according to the software component specification. The identified types of documentation and their RECOMMENDED content are: * **[QC.Doc06.1]** README file @@ -252,7 +220,7 @@ software being developed first and test cases created later. * Public API documentation. * Command Line Interface (CLI) reference. -* **[QC.Doc07]** Documentation MUST be checked on change basis. +* **[QC.Doc07]** Documentation **MUST** be checked on change basis. ## 4.9. Security [QC.Sec] @@ -260,114 +228,94 @@ Security assessment is essential for any production **Software**. An effective i security requirements applies to every stage in the software development life cycle (SDLC), especially effective at the source code level. -* **[QC.Sec01]** Secure coding practices SHALL be applied into all the stages of a software +* **[QC.Sec01]** Secure coding practices ****MUST**** be applied into all the stages of a software component development lifecycle. * **[QC.Sec01.1]** Compliance with Open Web Application Security Project (OWASP) secure coding guidelines [@https://owasp.org/www-project-secure-coding-practices-quick-reference-guide/migrated_content] - is RECOMMENDED, even for non-web applications. + is **RECOMMENDED**, even for non-web applications. -* **[QC.Sec02]** Source code SHALL use automated linter tools to perform static application security - testing (SAST) [@https://owasp.org/www-community/Source_Code_Analysis_Tools] that flag common - suspicious constructs that may cause a bug or lead to a security risk (e.g. inconsistent data - structure sizes or unused resources). +* **[QC.Sec02]** Source code ****MUST**** use automated linter tools to perform static application + security testing (SAST) [@https://owasp.org/www-community/Source_Code_Analysis_Tools] that flag + common suspicious constructs that may cause a bug or lead to a security risk (e.g. inconsistent + data structure sizes or unused resources). * **[QC.Sec03]** Security code reviews [@https://owasp.org/www-project-code-review-guide/migrated_content] - for certain vulnerabilities SHOULD be done as part of the identification of potential security - flaws in the code. Inputs SHOULD come from automated linters and manual penetration testing + for certain vulnerabilities **SHOULD** be done as part of the identification of potential security + flaws in the code. Inputs **SHOULD** come from automated linters and manual penetration testing results. -* **[QC.Sec04]** World-writable files or directories MUST NOT be present in the product’s +* **[QC.Sec04]** World-writable files or directories **MUST NOT** be present in the product’s configuration or logging locations. -**TO BE REMOVED - moved/added/adapted in the Service QA** - -* **[QC.Sec03]** Dynamic application security testing (DAST) - [@https://www.owasp.org/index.php/Category:Vulnerability_Scanning_Tools] - SHALL be performed from the outside, to software components in an operating state, to look for - security vulnerabilities (e.g. SQL injection, cross-site scripting). - -* **[QC.Sec04]** Interactive Application Security Testing (IAST) - [@https://www.veracode.com/security/interactive-application-security-testing-iast], - analyzes code for security vulnerabilities while the app is run by an automated test. IAST SHOULD - be performed to software components in an operating state, while Functional Testing **[QC.Fun]**, - is running. - -* **[QC.Sec05]** Manual penetration testing MAY be part of the application security verification effort. - -* **[QC.Sec08]** World-writable files SHOULD NOT be created while the service is in - operation. Whenever they are required, the relevant files MUST be documented. - -* **[QC.Sec09]** World-readable files MUST NOT contain passwords. - -* **[QC.Sec10]** The services delivered SHALL adhere to any extra security policies or - requirements set at the operational level. - ## 4.10. Code Workflow [QC.Wor] A change-based approach is accomplished with a branching model. -* **[QC.Wor01]** The main branch in the source code repository MUST maintain a working state version - of the software component. +* **[QC.Wor01]** The main branch in the source code repository **MUST** maintain a working state + version of the software component. - * **[QC.Wor01.1]** Main branch SHOULD be protected to disallow force pushing, thus preventing + * **[QC.Wor01.1]** Main branch **SHOULD** be protected to disallow force pushing, thus preventing untested and un-reviewed source code from entering the production-ready version. - * **[QC.Wor01.2]** New features SHOULD only be merged in the main branch whenever the SQA criteria - is fulfilled. + * **[QC.Wor01.2]** New features **SHOULD** only be merged in the main branch whenever the SQA + criteria is fulfilled. -* **[QC.Wor02]** New changes in the source code MUST be placed in individual branches. +* **[QC.Wor02]** New changes in the source code **MUST** be placed in individual branches. - * **[QC.Wor02.1]** It is RECOMMENDED to agree on a branch nomenclature, usually by prefixing, to - differentiate change types (e.g. feature, release, fix). + * **[QC.Wor02.1]** It is **RECOMMENDED** to agree on a branch nomenclature, usually by prefixing, + to differentiate change types (e.g. feature, release, fix). * **[QC.Wor03]** The existence of a secondary long-term branch that contains the changes for the - next release is RECOMMENDED. + next release is **RECOMMENDED**. - * **[QC.Wor03.1]** Next release changes come from the individual branches. + * **[QC.Wor03.1]** Next release changes **SHOULD** come from the individual branches. * **[QC.Wor03.2]** Once ready for release, changes in the secondary long-term branch are merged into the main branch and versioned. At that point in time, main and secondary branches are - aligned. This step SHOULD mark a production release. + aligned. This step **SHOULD** mark a production release. ## 4.11. Semantic Versioning [QC.Ver] -* **[QC.Ver01]** Semantic Versioning [@https://semver.org] specification is RECOMMENDED for tagging - the production releases. +* **[QC.Ver01]** Semantic Versioning [@https://semver.org] specification is **RECOMMENDED** for + tagging the production releases. ## 4.12. Code Management [QC.Man] * **[QC.Man01]** An issue tracking system facilitates structured software development. Leveraging - issues to track down both new enhancements and defects (bugs, documentation typos) is RECOMMENDED. + issues to track down both new enhancements and defects (bugs, documentation typos) is + **RECOMMENDED**. * **[QC.Man01.1]** In addition to monitoring the internal development, issues are the best - means for supporting users. External users SHOULD be able to create issues based on the + means for supporting users. External users **SHOULD** be able to create issues based on the operational performance of the software. - * **[QC.Man01.2]** The description of an issue SHOULD be concise and state clearly the problem. It - is RECOMMENDED to add any reference to the actual problem. In the case of bugs, the issue SHOULD - be accompanied by the relevant debug information. - * The usage of templates for the issue description is RECOMMENDED. + * **[QC.Man01.2]** The description of an issue **SHOULD** be concise and state clearly the + problem. It is RECOMMENDED to add any reference to the actual problem. In the case of bugs, the + issue **SHOULD** be accompanied by the relevant debug information. + * The usage of templates for the issue description is **RECOMMENDED**. * **[QC.Man02]** In social coding environments, pull or merge requests represent the cornerstone of collaboration. A pull or merge request provides a place for review and discussion of the changes proposed to be part of an existing version of the code. - * **[QC.Man02.1]** Pull/Merge requests SHOULD be used for every change in the codebase. + * **[QC.Man02.1]** Pull/Merge requests **SHOULD** be used for every change in the codebase. - * **[QC.Man02.2]** A software project SHOULD be open to external collaboration through pull/merge - requests. + * **[QC.Man02.2]** A software project **SHOULD** be open to external collaboration through + pull/merge requests. - * **[QC.Man02.3]** A pull/merge request description SHOULD be concise and state clearly its + * **[QC.Man02.3]** A pull/merge request description **SHOULD** be concise and state clearly its purpose (e.g. if it is fixing an observed bug or adding a new feature). - * **[QC.Man02.4]** The usage of templates for the pull/merge request's description is RECOMMENDED. + * **[QC.Man02.4]** The usage of templates for the pull/merge request's description is + **RECOMMENDED**. + + * **[QC.Man02.5]** It is **RECOMMENDED** to use pull/merge requests to address open issues. - * **[QC.Man02.5]** It is RECOMMENDED to use pull/merge requests to address open issues. The - pull/merge request description SHOULD make reference to the identifiers of the issues it is - fixing (to eventually close them, either manually or automatically). + * **[QC.Man02.6]** The pull/merge request description **SHOULD** make reference to the identifiers + of the issues it is fixing (to eventually close them, either manually or automatically). ## 4.13. Code Review [QC.Rev] @@ -376,8 +324,8 @@ source code [@https://owasp.org/www-project-code-review-guide/migrated_content]. last step in the change management pipeline, once the candidate change has successfully passed over the required set of change-based tests. -* **[QC.Rev01]** Code reviews MUST be done in the agreed peer review tool within the project, with - the following RECOMMENDED functionality: +* **[QC.Rev01]** Code reviews **MUST** be done in the agreed peer review tool within the project, + with the following RECOMMENDED functionality: * **[QC.Rev01.1]** Allows general and specific comments on the line or lines that need to be reviewed. @@ -387,12 +335,13 @@ the required set of change-based tests. * **[QC.Rev01.3]** Allows to prevent merges of the candidate change whenever not all the required tests are successful. Exceptions to this rule cover the third-party or upstream contributions which MAY use the existing mechanisms or tools for code review provided by the target software - project. This exception MUST only be allowed whenever the external revision lifecycle does not - interfere with the project deadlines. + project. This exception **MUST** only be allowed whenever the external revision lifecycle does + not interfere with the project deadlines. -* **[QC.Rev02]** Code reviews MUST be open and collaborative, allowing external experts revisions. +* **[QC.Rev02]** Code reviews **MUST** be open and collaborative, allowing external experts + revisions. -* **[QC.Rev03]** Code reviews SHOULD be concise and use neutral language. The areas where the +* **[QC.Rev03]** Code reviews **SHOULD** be concise and use neutral language. The areas where the reviewers MAY focus are: * **[QC.Rev03.1]** Message description: commit message is clear, self-explanatory and describes @@ -407,46 +356,46 @@ the required set of change-based tests. * **[QC.Rev03.4]** Review of required tests: current battery of tests is sufficient for validation. - * **[QC.Rev03.5]** Review of documentation: whether the change SHOULD bring along a corresponding - update in the documentation. + * **[QC.Rev03.5]** Review of documentation: whether the change **SHOULD** bring along a + corresponding update in the documentation. -* **[QC.Rev04]** Code reviews MUST be checked on change basis. +* **[QC.Rev04]** Code reviews **MUST** be checked on change basis. -* **[QC.Rev05]** Code reviews SHOULD assess the inherent security risk of the changes, ensuring that - the security model has not been downgraded or compromised by the changes. +* **[QC.Rev05]** Code reviews **SHOULD** assess the inherent security risk of the changes, ensuring + that the security model has not been downgraded or compromised by the changes. ## 4.14. Automated Delivery [QC.Del] Automated delivery comprises the build of Software into an artifact, its upload/registration into a public repository of such artifacts and notification of the success of the process. -* **[QC.Del01]** Production-ready code SHALL be built as an artifact that can be executed on a +* **[QC.Del01]** Production-ready code **MUST** be built as an artifact that can be executed on a system. -* **[QC.Del02]** The builded artifact SHALL be uploaded and registered into a public repository of - such artifacts. +* **[QC.Del02]** The builded artifact **MUST** be uploaded and registered into a public repository + of such artifacts. -* **[QC.Del03]** Upon success of the previous (**QC.Del02**) process, a notification SHALL be sent - to pre-defined parties such as the main developer or team. +* **[QC.Del03]** Upon success of the previous (**QC.Del02**) process, a notification **MUST** be + sent to pre-defined parties such as the main developer or team. ## 4.15. Automated Deployment [QC.Dep] -* **[QC.Dep01]** Production-ready code SHALL be deployed as a workable system with the minimal user +* **[QC.Dep01]** Production-ready code **MUST** be deployed as a workable system with the minimal user or system administrator interaction leveraging software configuration management (SCM) tools. * **[QC.Dep02]** A SCM module is treated as code. - * **[QC.Dep02.1]** Version controlled, it SHOULD reside in a different repository than the source - code to facilitate the distribution. + * **[QC.Dep02.1]** Version controlled, it **SHOULD** reside in a different repository than the + source code to facilitate the distribution. -* **[QC.Dep03]** It is RECOMMENDED that all software components delivered by the same project agree - on a common SCM tool. +* **[QC.Dep03]** It is **RECOMMENDED** that all software components delivered by the same project + agree on a common SCM tool. - * **[QC.Dep03.1]** However, software products are not restricted to provide a unique solution for - the automated deployment. + * **[QC.Dep03.1]** However, software products **MAY** not be restricted to provide a unique + solution for the automated deployment. -* **[QC.Dep04]** Any change affecting the application’s deployment or operation MUST be subsequently - reflected in the relevant SCM modules. +* **[QC.Dep04]** Any change affecting the application’s deployment or operation **MUST** be + subsequently reflected in the relevant SCM modules. -* **[QC.Dep05]** Official repositories provided by the manufacturer SHOULD be used to host the SCM - modules, thus augmenting the visibility and promote external collaboration. +* **[QC.Dep05]** Official repositories provided by the manufacturer **SHOULD** be used to host the + SCM modules, thus augmenting the visibility and promote external collaboration.