Skip to content

Scope gRPC dependencies and rely on grpc-bom for single version#4784

Merged
merlimat merged 2 commits into
apache:masterfrom
merlimat:wt/charming-wiles-8aa002
May 9, 2026
Merged

Scope gRPC dependencies and rely on grpc-bom for single version#4784
merlimat merged 2 commits into
apache:masterfrom
merlimat:wt/charming-wiles-8aa002

Conversation

@merlimat
Copy link
Copy Markdown
Contributor

@merlimat merlimat commented May 9, 2026

Summary

  • Drop grpc-all (with its long exclusion list in the parent pom) and depend on the specific io.grpc:* artifacts each module actually uses; grpc-bom continues to enforce a single version across them.
  • Drop grpc-protobuf from stream-storage-proto — dead after the LightProto migration; the generated code uses Netty ByteBuf only and no Java code anywhere imports io.grpc.protobuf.*.
  • Drop pointless protobuf-java usages so statelib no longer needs the dependency:
    • statelib/Bytes.toLong(ByteString) was a dead overload — every caller uses the byte[] variant.
    • statelib/MVCCStoreImpl used TextFormat.escapeBytes once for an error message — replaced with JDK 17 HexFormat.of().formatHex.
    • bookkeeper-server/BookieImplTest had two no-op protobuf round-trips (ByteString.copyFrom(bytes).toByteArray() and unsafeWrap(...).asReadOnlyByteBuffer()) — replaced with plain byte[] / ByteBuffer.wrap.
  • Update LICENSE-{all,server,bkctl}.bin.txt to match the new bundled jar set: drop entries for grpc-all, grpc-alts, grpc-auth, grpc-grpclb, grpc-opentelemetry, grpc-protobuf, grpc-protobuf-lite, grpc-services, grpc-testing, grpc-xds, grpc-rls, grpc-gcp-csm-observability, plus transitively-removed proto-google-common-protos, protobuf-java-util, api-common, google-http-client*, google-auth-library-*, auto-value-annotations, re2j, opencensus-*, conscrypt; clean up orphaned source-URL footnotes.

Module-by-module pom changes

Module Before After
parent pom.xml grpc-bom import + grpc-all dependencyManagement entry with 8 exclusions grpc-bom import only
stream/common grpc-all, grpc-netty-shaded grpc-api, grpc-stub, grpc-core, grpc-netty-shaded; grpc-inprocess, grpc-util (test)
stream/tests-common grpc-all, grpc-netty-shaded, grpc-testing grpc-api, grpc-stub, grpc-netty-shaded
stream/proto + grpc-protobuf (dropped)
stream/clients/java/base (silent transitive grpc-all) explicit grpc-inprocess, grpc-netty-shaded; grpc-util (test)
metadata-drivers/etcd grpc-all, grpc-netty-shaded grpc-api, grpc-stub, grpc-netty-shaded

Net effect on the BK distribution

  • gRPC: bookkeeper-dist/{all,server,bkctl} now ships 7 io.grpc:* jars (grpc-api, grpc-stub, grpc-core, grpc-context, grpc-inprocess, grpc-netty-shaded, grpc-util) — down from 18.
  • protobuf: ships 1 protobuf-java jar — down from 3 (protobuf-java, protobuf-java-util, proto-google-common-protos). The remaining protobuf-java is now sourced only via prometheus-metrics-exposition-formats-no-protobuf, not via gRPC.
  • metadata-drivers/etcd (which legitimately needs grpc-protobuf / protobuf-java for jetcd) is not pulled into the BK distribution or bookkeeper-server, so its protobuf-java usage doesn't affect dist licensing.

Test plan

  • mvn -pl stream/common,stream/clients/java/base,stream/statelib -DstreamTests test — 229 tests pass (stream/common 41, clients/java/base 70, statelib 118).
  • mvn -pl bookkeeper-server test -Dtest=BookieImplTest — 3 tests pass (covers the protobuf-removal refactor).
  • mvn -DskipTests -pl '!native-io,!bookkeeper-dist/{all,server,bkctl}' --fae install — full reactor builds and installs.
  • mvn -pl bookkeeper-dist/all dependency:tree — verified no io.grpc:grpc-all / grpc-protobuf / proto-google-common-protos / protobuf-java-util / google-auth-library-* / opencensus-* / conscrypt in the dist tree.
  • CI to validate broader test surface.

Replace the brittle `grpc-all` dependency (with its long exclusion list in
the parent pom) with the specific gRPC artifacts each module actually
uses. The `grpc-bom` import in dependencyManagement continues to enforce
a single version across all `io.grpc:*` artifacts.

Module changes:
- pom.xml: drop the `grpc-all` dependencyManagement block; the BOM alone
  manages versions
- stream/common: `grpc-all` -> `grpc-api`, `grpc-stub`, `grpc-core`
  (compile); `grpc-inprocess`, `grpc-util` (test)
- stream/tests-common: `grpc-all` -> `grpc-api`, `grpc-stub`; drop unused
  `grpc-testing` (no `io.grpc.testing.*` imports anywhere)
- stream/proto: drop `grpc-protobuf` (dead after the LightProto
  migration; generated code uses Netty `ByteBuf` only)
- stream/clients/java/base: add explicit `grpc-inprocess`,
  `grpc-netty-shaded`, `grpc-util` that were silently inherited
- metadata-drivers/etcd: `grpc-all` -> `grpc-api`, `grpc-stub`

Drop pointless protobuf-java usages so statelib no longer needs it:
- statelib `Bytes.toLong(ByteString)` was a dead overload (every caller
  uses the `byte[]` variant); removed
- statelib `MVCCStoreImpl` used `TextFormat.escapeBytes` once for an
  error message; replaced with JDK 17 `HexFormat.of().formatHex`
- bookkeeper-server `BookieImplTest` had two no-op protobuf
  round-trips (`ByteString.copyFrom(bytes).toByteArray()` and
  `unsafeWrap(...).asReadOnlyByteBuffer()`); replaced with plain
  `byte[]` and `ByteBuffer.wrap`

Update LICENSE-{all,server,bkctl}.bin.txt to match the new bundled jar
set: drop entries for grpc-all/-alts/-auth/-grpclb/-opentelemetry
/-protobuf/-protobuf-lite/-services/-testing/-xds/-rls
/-gcp-csm-observability and the transitively-removed proto-google-common
-protos, protobuf-java-util, api-common, google-http-client*,
google-auth-library-*, auto-value-annotations, re2j, opencensus-*,
conscrypt; clean up the orphaned source-URL footnotes.

Net dist change: 7 grpc jars (down from 18) and 1 protobuf jar
(down from 3, now sourced only via prometheus-metrics-exposition-formats).
@merlimat merlimat requested a review from lhotari May 9, 2026 00:05
CI's `dev/check-all-licenses` flagged remaining stale references that
were not caught in the previous commit:

- NOTICE-{all,server,bkctl}.bin.txt: drop the unbundled `grpc-all`,
  `grpc-auth`, `grpc-protobuf`, `grpc-protobuf-lite`, `grpc-testing`
  entries; add the previously-missing `grpc-api`, `grpc-inprocess`,
  `grpc-util` to the gRPC list (these have always been bundled).
- LICENSE-{all,server,bkctl}.bin.txt: drop
  `opentelemetry-gcp-resources-1.56.0-alpha.jar` (was transitive via the
  removed `grpc-gcp-csm-observability`, no longer bundled).
- LICENSE-bkctl.bin.txt: drop the entire OpenTelemetry SDK list and the
  protobuf-java entry; bkctl never bundled any of these (pre-existing
  staleness, hidden in CI by `set -e` stopping on the first dist error).
- bookkeeper-dist/src/assemble/{bin-all,bin-server,bkctl}.xml: drop the
  `deps/google-auth-library-credentials-0.20.0/LICENSE` include from
  every assembly (orphaned now that the jar is no longer bundled), and
  drop the `deps/protobuf/LICENSE` include from the bkctl assembly
  (bkctl doesn't ship protobuf-java).
- Delete the orphaned
  `bookkeeper-dist/src/main/resources/deps/google-auth-library-credentials-0.20.0/`
  directory.

Verified locally: `dev/check-all-licenses` passes against freshly-built
all/server/bkctl tarballs.
@merlimat merlimat merged commit 8b6671f into apache:master May 9, 2026
36 of 38 checks passed
@merlimat merlimat deleted the wt/charming-wiles-8aa002 branch May 9, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants