Skip to content

Privacy DataClassification fixes + keep Label-scope findings at minor#102

Merged
JesperSchulz merged 2 commits into
mainfrom
fix/privacy-data-classification-default
Jul 14, 2026
Merged

Privacy DataClassification fixes + keep Label-scope findings at minor#102
JesperSchulz merged 2 commits into
mainfrom
fix/privacy-data-classification-default

Conversation

@gggdttt

@gggdttt gggdttt commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Two data-grounded fixes from the online-eval false-positive analysis, targeting the worst-scoring cited articles.

1. Privacy article: privacy/data-classification-required-on-pii-fields.md

This article had 6 findings, 0 TP / 6 FP.

a. Correct the platform default (factual fix). The Description claimed the default value is SystemMetadata. In AL, when DataClassification is omitted, the platform applies ToBeClassified (a "not yet reviewed" placeholder), not SystemMetadata. The wrong statement made the model fabricate contradictory consequences across findings (one claimed the default was SystemMetadata, another ToBeClassified), undermining credibility. Now states: omission → ToBeClassified; and both leaving PII as ToBeClassified and setting SystemMetadata to silence the requirement are under-classifications.

b. Broaden the classification taxonomy (scope fix). Best Practice only listed personal-PII values. Real findings flagged organization/account identifiers (a BE company VAT/registration number; a Swiss QR-Bill IBAN) with no correct value to point at. Added: company registration / VAT number → OrganizationIdentifiableInformation; bank account number / IBAN → AccountData.

2. Style skill: skills/review/al-style-review.md (correction to the B1 severity-calibration paragraph)

The B1 calibration listed "a Label declared at local instead of object scope" among conventions "a formal analyzer already flags", routing it to info (below the engine's Medium floor → dropped). Verified against the eval data this is wrong:

  • No analyzer enforces label scope. The labels-declared-at-object-scope.md article cites no CodeCop rule (the other calibrated items all cite AA0248 / AA0074 / AA0218 / AA0002); its rationale is translation-pipeline fragility, a tooling concern the compiler does not check.
  • It is the highest-value style rule in the dataset: 17 findings, 5 TP, 1 thumbs-up, 4 applied by developers. Downgrading it to info would suppress genuinely valuable, adopted findings.

Fix: move label-scope out of the analyzer-flagged (info) list and into the minor list ("concrete downstream impact the analyzer does not catch"), with a note that no analyzer enforces label scope. The other three calibrated items (AA0248/AA0218/AA0002) are correctly analyzer-enforced and stay at info.

Scope

Privacy article: Description + Best Practice only (Anti Pattern and AL samples unchanged). Style skill: one sentence in the severity-calibration paragraph.

@gggdttt gggdttt changed the title Fix DataClassification default fact and broaden classification taxonomy Privacy DataClassification fixes + keep Label-scope findings at minor Jul 14, 2026
@gggdttt gggdttt marked this pull request as ready for review July 14, 2026 12:18
@JesperSchulz JesperSchulz merged commit 809af97 into main Jul 14, 2026
5 checks passed
@gggdttt gggdttt requested a review from Copilot July 14, 2026 13:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates BCQuality’s AL guidance content to reduce false positives and improve severity calibration by correcting DataClassification defaults/coverage in the privacy knowledge article and ensuring label-scope findings remain actionable (minor) rather than suppressed (info) in the AL style review skill.

Changes:

  • Corrected the privacy article to state that omitting DataClassification defaults to ToBeClassified, and clarified that both ToBeClassified and SystemMetadata can be under-classifications when the field contains sensitive data.
  • Expanded the privacy article’s taxonomy guidance to include organization identifiers (OrganizationIdentifiableInformation) and financial account identifiers (AccountData).
  • Updated AL style severity calibration so “Labels declared procedure-local instead of object scope” is treated as minor (explicitly noting no analyzer enforces label scope), rather than being grouped with analyzer-enforced info items.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
microsoft/skills/review/al-style-review.md Adjusts severity calibration text so label-scope findings remain minor (non-analyzer-enforced but with downstream translation impact).
microsoft/knowledge/privacy/data-classification-required-on-pii-fields.md Fixes DataClassification default behavior and broadens recommended classifications to cover org/account identifiers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants