Skip to content

no-jira: test/monitoring: increase load balancer readiness and curl connection timeout#30994

Open
tthvo wants to merge 2 commits intoopenshift:mainfrom
tthvo:lb-timeout
Open

no-jira: test/monitoring: increase load balancer readiness and curl connection timeout#30994
tthvo wants to merge 2 commits intoopenshift:mainfrom
tthvo:lb-timeout

Conversation

@tthvo
Copy link
Copy Markdown
Member

@tthvo tthvo commented Apr 10, 2026

This PR increases the timeout for load balancer readiness and curl connection to address slow DNS propagation on cloud provider, for example, AWS EU Sovereign Cloud.

Background

AWS Load balancer DNS SOA records for AWS ELB in EUSC region show a 900s (15-minute) retry interval, which can cause the load balancer hostname resolution (if first hit NXDOMAIN) to take longer than the previous 10-minute timeout. Additionally, it can take a few more minutes for DNS to fully propagate to its nameserver.

$ nslookup -q=SOA eusc-de-east-1.elb.amazonaws.eu
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
eusc-de-east-1.elb.amazonaws.eu
	origin = ns-501.awsdns-eusc-62.fr
	mail addr = awsdns-hostmaster.amazon.com
	serial = 1
	refresh = 7200
	retry = 900
	expire = 1209600
	minimum = 86400

References

We are seeing a similar issue in CCM hairpin tests: openshift/cluster-cloud-controller-manager-operator#449.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot
Copy link
Copy Markdown

@tthvo: This pull request explicitly references no jira issue.

Details

In response to this:

The service-type-load-balancer-availability monitor test was timing out during preparation when waiting for the load balancer to become reachable.

DNS SOA records for AWS ELB in EUSC region show a 900s (15-minute) retry interval, which can cause the load balancer hostname resolution to take longer than the previous 10-minute timeout.

Increase the timeout from 10 to 20 minutes to accommodate slow DNS propagation in regions with high TTL values.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 10, 2026
@openshift-ci openshift-ci bot requested review from deads2k and p0lyn0mial April 10, 2026 02:52
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 10, 2026

Walkthrough

Increased the initial HTTP reachability check timeout from 10 to 20 minutes in the load-balancer disruption test; added WithTimeout(time.Duration) to BackendSampler and applied a 120s connection timeout to the backend samplers used in that test.

Changes

Cohort / File(s) Summary
Network Disruption Service Load Balancer Test
pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go
Increased the testReachableHTTPWithMinSuccessCount pre-collection timeout from 10 * time.Minute to 20 * time.Minute. Introduced connectionTimeout := 120 * time.Second and applied it to backend samplers via .WithTimeout(connectionTimeout) (after .WithExpectedBody("hello")) for the service-load-balancer-with-pdb-new-connections and service-load-balancer-with-pdb-reused-connections samplers.
Backend Disruption Sampler
pkg/monitor/backenddisruption/disruption_backend_sampler.go
Added func (b *BackendSampler) WithTimeout(timeout time.Duration) *BackendSampler to set an overrideable timeout (stored as *time.Duration) used when deriving HTTP client and phase timeouts.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 10
✅ Passed checks (10 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately reflects the main changes: increasing the load balancer readiness timeout from 10 to 20 minutes and adding a curl connection timeout of 120 seconds.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Stable And Deterministic Test Names ✅ Passed The PR changes timeout values and adds a WithTimeout builder method without modifying any Ginkgo test names or titles.
Test Structure And Quality ✅ Passed PR modifies monitor test framework utilities and backend disruption sampling infrastructure with no Ginkgo test code present.
Microshift Test Compatibility ✅ Passed PR modifies monitor test infrastructure with timeout changes and builder methods, not Ginkgo e2e tests. Existing MicroShift protection already present in code.
Single Node Openshift (Sno) Test Compatibility ✅ Passed This PR does not add any new Ginkgo e2e tests. Changes only modify timeout configuration values and add a helper method to existing code.
Topology-Aware Scheduling Compatibility ✅ Passed The pull request introduces only timeout value adjustments and a new builder method for configuring HTTP connection timeouts in test/monitoring infrastructure code. No new scheduling constraints are introduced.
Ote Binary Stdout Contract ✅ Passed The PR modifies timeout configuration values and adds a builder method to BackendSampler; these changes do not involve stdout writes in process-level code.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR does not add new Ginkgo e2e tests. It only modifies existing monitor test timeouts and adds a utility method. The check applies only to new Ginkgo test functions, which are absent here.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 10, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tthvo
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/monitor/backenddisruption/disruption_backend_sampler.go`:
- Around line 235-237: Validate the timeout in BackendSampler.WithTimeout and
refuse non-positive values: check if timeout <= 0 and if so do not set b.timeout
(return b unchanged) so callers cannot accidentally disable timeouts; otherwise
set b.timeout = &timeout and return b. This ensures BackendSampler.WithTimeout
enforces a positive time.Duration (use the existing BackendSampler and
WithTimeout identifiers and the b.timeout field).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 304a4f0e-faeb-43e6-99fa-921457c4fef5

📥 Commits

Reviewing files that changed from the base of the PR and between f454514 and 60e1b17.

📒 Files selected for processing (2)
  • pkg/monitor/backenddisruption/disruption_backend_sampler.go
  • pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go
✅ Files skipped from review due to trivial changes (1)
  • pkg/monitortests/network/disruptionserviceloadbalancer/monitortest.go

Comment thread pkg/monitor/backenddisruption/disruption_backend_sampler.go
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@tthvo
Copy link
Copy Markdown
Member Author

tthvo commented Apr 10, 2026

/test e2e-aws-ovn-fips

The service-type-load-balancer-availability monitor test was timing out
during preparation when waiting for the load balancer to become reachable.

DNS SOA records for AWS ELB in EUSC region show a 900s (15-minute) retry
interval, which can cause the load balancer hostname resolution to take
longer than the previous 10-minute timeout.

Increase the timeout from 10 to 20 minutes to accommodate slow DNS
propagation in regions with high TTL values.
@tthvo
Copy link
Copy Markdown
Member Author

tthvo commented Apr 16, 2026

/retitle no-jira: test/monitoring: increase load balancer readiness and curl connection timeout

@openshift-ci openshift-ci bot changed the title no-jira:test/monitoring: increase load balancer readiness timeout to 20 minutes no-jira: test/monitoring: increase load balancer readiness and curl connection timeout Apr 16, 2026
The service-load-balancer-with-pdb disruption test was experiencing
DNS resolution failures when connecting to the load balancer during
test execution. AWS ELB DNS zones have an SOA TTL of 900 seconds,
which means DNS propagation can take up to 15 minutes for newly
created load balancers.

This change adds a WithTimeout() method to BackendSampler and sets
a 120-second timeout for the service load balancer disruption samplers,
allowing sufficient time for DNS retries during the propagation period.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@tthvo
Copy link
Copy Markdown
Member Author

tthvo commented Apr 17, 2026

A couple of /testwith including this PR showed that these monitoring tests passed in EUSC:

I think the slow DNS propagation can be delegated to AWS to improve their infrastructure in the near future (this EUSC cloud is net new - Jan 2026). However, I am taking the proactive approach here and this should help better avoid flakes 🤔

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@tthvo
Copy link
Copy Markdown
Member Author

tthvo commented Apr 17, 2026

/test e2e-gcp-csi
/test e2e-gcp-ovn

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 17, 2026

@tthvo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere-ovn-upi ec8a551 link true /test e2e-vsphere-ovn-upi

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants