feat(ingest-metrics): Ensure billing metrics consumer avoids outcome double-billing

[klochek]

With relay producing billing outcomes here, introduce and check for a new tag ("billing_outcome_accepted") to avoid double-billing.

I checked for prior string existence with:

SELECT string, count(*) 
FROM sentry_{perf,}stringindexer 
WHERE 1=1
    AND string IN (
        'billing_outcome_accepted'
    ) 
GROUP BY string;

Guarded by a new temporary flag organizations:relay-generate-billing-outcome

[sentry]

Bug: The billing metrics consumer unconditionally suppresses outcomes without checking the organizations:relay-generate-billing-outcome feature flag.
_{Severity: HIGH}

Suggested Fix

Before checking the billing_outcome_accepted tag, retrieve the organization associated with the metric. Wrap the suppression logic in a feature flag check, such as if features.has("organizations:relay-generate-billing-outcome", organization):, to ensure it only executes when the feature is enabled for that organization.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/sentry/ingest/billing_metrics_consumer.py#L84-L86

Potential issue: The billing metrics consumer at
`src/sentry/ingest/billing_metrics_consumer.py` unconditionally suppresses billing
outcomes if the `billing_outcome_accepted` tag is `"true"`. This logic should only
execute when the `organizations:relay-generate-billing-outcome` feature flag is enabled
for the specific organization. Because the feature flag check is missing, billing
metrics will be incorrectly dropped for organizations where the feature is not yet
active, leading to inaccurate billing data and potential revenue loss.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[Dav1dde]

That's totally fine and we should not check the feature flag.

Logical this is two separate components:

• Consumer can skip metrics
• Feature flag to control logic in Relay

[Dav1dde]

The feature flag also needs to be exposed to Relay, search for EXPOSABLE_FEATURES

[Dav1dde]

That's totally fine and we should not check the feature flag.

Logical this is two separate components:

• Consumer can skip metrics
• Feature flag to control logic in Relay

[jjbayer]

Suggested change

@with_feature("organizations:relay-generate-billing-outcome")

Mentioning the feature here should not be necessary because the consumer-side behavior is determined by the tag, not the feature flag, right? This might also be what confused sentry[bot] in their comment above.

[jjbayer]

Not sure if you already bikeshedded on the name but billing_outcome_emitted would make slightly more sense to me (the outcome is Accepted but the thing relay does is emitting it).