Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create 'SARIF Validation Rule Authoring Principles' doc. #1906

Merged
merged 29 commits into from
Jun 24, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
a820ed5
Create 'SARIF Validation Rule Authoring Principles' doc.
Jun 4, 2020
ddd3c69
Add periods to sentences; complete incomplete principle.
Jun 4, 2020
955db4b
Rewrite after discussion with Michael and team.
Jun 6, 2020
505f32e
Fix typos.
Jun 6, 2020
7fe5524
Fix a typo.
Jun 6, 2020
e21954d
First half of rule factoring: existing rules.
Jun 8, 2020
db2696a
Correct description of rule number ranges.
Jun 8, 2020
4095e34
Fix a typo.
Jun 8, 2020
f14ae8f
Finish first draft.
Jun 8, 2020
db757f4
Add missing separator.
Jun 8, 2020
5d52039
Wordsmith a heading.
Jun 8, 2020
145b299
Changes per HK review.
Jun 8, 2020
f6e1123
Finish HK review.
Jun 8, 2020
dd6e2b2
Incomplete 'Contributing a rule' doc, derived from BinSkim.
Jun 13, 2020
82b0121
Merge branch 'master' into users/lgolding/rule-authoring-principles
Jun 15, 2020
703ed94
Start rule factoring spreadsheet.
Jun 15, 2020
43f8df4
Merge branch 'master' into users/lgolding/rule-authoring-principles
Jun 15, 2020
67150b0
Finish first cut at rule factoring spreadsheet.
Jun 15, 2020
9143d7a
Fix typos.
Jun 15, 2020
735e6a6
Update rules spreadsheet per MF review.
Jun 18, 2020
e0d3cda
Update rules spreadsheet; add Rule Messages doc.
Jun 20, 2020
c3c7aa4
Finish strings for 2009; add 'Message status' column.
Jun 20, 2020
0fde47f
Incorporate Introduction.
Jun 22, 2020
e87e490
Update Rule Messages with Introduction.
Jun 22, 2020
838cd68
Remove redundant file; rename real one.
Jun 22, 2020
639ff13
Message refinements.
Jun 23, 2020
97dd4d6
Rename 'Producing' doc and tweak intro.
Jun 23, 2020
fcc4be4
Author messages for SARIF2002.UseMessageArguments.
Jun 23, 2020
3f716f2
Author messages for SARIF1012.MessagePropertiesMustBeConsistent.
Jun 23, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions docs/Contributing a SARIF Validation Rule.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Contributing a SARIF Validation Rule

1. File a (or locate an existing) sarif-sdk issue that serves as the specification for the rule.
2. Create a shell rule.
3. Prepare test assets for the rule.
4. Author standard functional tests.
5. Implement the analysis.
6. Review the impact of the new analysis on the test assets and update "expected outputs" if necessary.
----
## File or locate a rule request.

1. Create a "Validation rule request" issue in the `sarif-sdk` repository by clicking **New issue** and selecting the **Validation rule request** template (or just click [here](https://github.com/microsoft/sarif-sdk/issues/new?assignees=&labels=rule-request&template=validation-rule-request.md&title=%5BRULE+REQUEST%5D+Concise+description+of+new+analysis)).

**CONCERN** As written, that URL requires the user to have "add labels" permission.

- The issue template contains additional guidance for authoring a complete rule specification.

- See the [CET shadow stack compatibility](https://github.com/microsoft/sarif-sdk/issues/277) rule specification for an example.

**TODO** Update to a real example ASAP.

2. Work with a repository maintainer to iron out details in implementation, _etc._

3. Rules marked with the `approved` label are ready to implement.

## Create shell rule for implementation.

1. Create a constant for the rule id in [RuleId.cs](https://github.com/microsoft/sarif-sdk/blob/master/src/Sarif.Multitool/Rules/RuleId.cs). The name of the constant must match the rule friendly name, _e.g._, `ReferToFinalSchema`, and its value must be the rule identifier, _e.g._, `"SARIF1020"`.

2. Open [RuleResources.resx](https://github.com/microsoft/sarif-sdk/blob/master/src/Sarif.Multitool/Rules/RuleResources.resx) and create new user-facing strings for the rule description and for the result. See The string values should be retrieved from the approved rule specific/issue. You can start rules development with placeholder values, obviously.

3. Add the rule description from the specification issue to a string named as RULEID_FRIENDLYNAME_Decription, e.g., `BA2025_EnableControlEnforcementTechnologyShadowStack_Description`. Again, you can use a placeholder if this text is still under refinement.

4. Open the shell rule starter code in [BAXXXX.RuleFriendlyName.cs]() and save a copy to either the [ELFRules] or [PERules] directory, depending on the target platform for the rule. Rename the file on save to conform to the actual assigned rule id and friendly rule name, e.g., `BA2025.EnableControlEnforcementTechnologyShadowStack`.

5. Find/Replace all occurrences of `BAXXXX` in this file with the rule id.

6. Find/Replace all occurrences of `RULEFRIENDLYNAME` in this file with the rule name.



----
This section originally appeared in the "principles" document. Integrate it into this section.

- The rule description resource string MUST be named `<ruleId>_<ruleName>`, _e.g._, `SARIF1020_ReferToFinalSchema`.

- Result message string resource strings MUST named `<ruleId>_<messageStringId>`, e.g., `SARIF1020_SchemaReferenceMissing`. These names must be concise and informative.

- Every result message string resource MUST have use a "real" friendly name, even if there is only one in the analyzer. Do NOT use a generic name like "default" for a singleton message string.

- Result message string resource names MUST be PascalCase (except for the underscore between the ruleId and the friendly name).
---
## Prepare test assets.

- Every rule should be tested against one SARIF file that explicitly violates the rule (covering every check that the rule performs), and one that conforms to the rule.
- Both passing and failing SARIF files should conform to all other . I.e., the goal is for test binaries to be entirely clean in the `pass` case and to only fire results for the new rule in the `rule` case.
- Create a directory in the rules functional test directory that matches the rule id and friendly name, separated with a dot character, e.g. [BA2025.EnableControlEnforcementTechnologyShadowStack](https://github.com/microsoft/binskim/tree/master/src/Test.FunctionalTests.BinSkim.Rules/FunctionalTestsData/BA2025.EnableControlEnforcementTechnologyShadowStack).
- Create directories named [Pass](https://github.com/microsoft/binskim/tree/master/src/Test.FunctionalTests.BinSkim.Rules/FunctionalTestsData/BA2025.EnableControlEnforcementTechnologyShadowStack/Pass) and [Fail](https://github.com/microsoft/binskim/tree/master/src/Test.FunctionalTests.BinSkim.Rules/FunctionalTestsData/BA2025.EnableControlEnforcementTechnologyShadowStack/Fail) in this directory and copy relevant secure and vulnerable test binaries to their respective location.
By convention, test binary names indicate their language, bittedness/processor, toolchain, and kind, with each attribute separated by an underscore. `Native_x64_VS2019_Console.exe`, for example, indicates a C++ Intel 64-bit console application compiled by the Microsoft Visual Studio 2019 toolchain.
- In some cases, it may be useful to create a specific binary to test proper return of the BinSkim `notApplicable` result (which indicates that the binary itself is not a relevant candidate for analysis). For many rules, the standard BinSkim "zoo" of test binaries can be used to verify proper enforcement of applicability.

## Author functional tests that use the new test assets


## Implement the analysis

There's not much more to say about this.

## Review the impact of the new analysis on the test assets and update "expected outputs" if necessary

Since the test assets for the existing rules were authored before you created the new rule, it's possible that some of them will violate the new rule. You need to correct any new violations in existing test assets so that tests for the existing rules continue to pass
Loading