Skip to content

HDDS-15614. Remove Datanode download/pull container replication#10584

Open
errose28 wants to merge 9 commits into
apache:masterfrom
errose28:worktree-remove-download-replicator
Open

HDDS-15614. Remove Datanode download/pull container replication#10584
errose28 wants to merge 9 commits into
apache:masterfrom
errose28:worktree-remove-download-replicator

Conversation

@errose28

@errose28 errose28 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Datanodes have used push replication to move containers by default since early 2023. There is no reason to continue using the pull model which is now disabled behind a feature flag. We can safely remove it, which will be easier to do before ZDU is merged into master.

  • On the ZDU replication path, both the client and server need to operate on the same apparent version to provide agreement on the API while datanodes are in mixed versions. Tracking this version requires twice the work when there are two protocols that have different client and server roles.
  • By removing the API before ZDU we do not need to do any compatibility handling, since the cluster must be upgraded to the first ZDU enabled version before it can use ZDU for upgrades going forward.
    • If we remove it after ZDU, we will need to take precautions to make sure the replications don't fail as datanodes are being upgraded.

Changes generated by Claude Code using various models with reviews and modifications by me.

Implementation Notes

  • Compression was only used for pull replication where it was disabled by default. This was never used for push replication so it is now removed entirely.
  • To exclude decommissioning and maintenance datanodes as replication targets, AbstractReplicationTask#shouldOnlyRunOnInServiceDatanodes could block pull replication from running on these nodes.
    • With push replication the switch was always false since a decom/maintenance node will need to push its replicas to other datanodes before it can be removed. Therefore the switch was removed.
  • Freon's ClosedContainerReplicator was removed, since it only worked for pull replication and there is no equivalent for push replication.
    • This tool worked when Freon was both the client and the target, and it could download from existing Datanodes that were the server and the source.
    • With push replication, Freon remains the client but it most also be the source, despite having no container data since that only exists in running datanodes.
    • Since this is not a CLI used on the production code path it was removed entirely, although we could optionally deprecate it and make it a no-op.

Reviewer Notes

This change is fairly large and I could not find a good way to break it down, although I am open to suggestions. One alternative is to iteratively review the prod changes first, and once those look good move on to the test changes.

What is the link to the Apache JIRA

HDDS-15614

How was this patch tested?

  • Tests specifically for pull that already had push equivalents were removed.
  • Tests that were only using pull were migrated to push equivalents:
    • TestOzoneContainerWithTLS#downloadContainer cannot be cleanly switched to a pull model. See HDDS-15644 for details and a plan to add tests for TLS + container tokens on the replication path as a larger follow-up change after this PR.
    • TestReplicationSupervisor#testReplicationImportReserveSpace was removed, since the reserved space check is relevant on the target only, and ReplicationSupervisor is now always the source.
      • TestSendContainerRequestHandler#testNoSpaceOnTargetVolume was added as a new test to cover this functionality. SendContainerRequestHandler is the receiving class on the target node in the push model.
  • ReplicateContainerCommand#forTest was no longer relevant. Tests now call toTarget directly.

Green run on my fork

@errose28 errose28 changed the title Worktree remove download replicator HDDS-15614. Remove Datanode download/pull container replication Jun 22, 2026
@errose28 errose28 marked this pull request as ready for review June 23, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant