fix(storage): Invocation ID is not retained on multipart upload retries#8190
fix(storage): Invocation ID is not retained on multipart upload retries#8190thiyaguk09 wants to merge 17 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements persistent invocationId tracking across the upload lifecycle and refactors the StorageTransport.makeRequest method to improve project ID resolution and header management. Feedback focuses on critical issues in the makeRequest refactor, including a bug where the request body is incorrectly assigned to an undefined property, thread-safety risks from unconditionally clearing shared interceptors, and an inconsistent response handler that mutates data and breaks return type expectations.
| this.gaxiosInstance.interceptors.request.clear(); | ||
| if (reqOpts.interceptors) { | ||
| this.gaxiosInstance.interceptors.request.clear(); | ||
| for (const inter of reqOpts.interceptors) { | ||
| this.gaxiosInstance.interceptors.request.add(inter); | ||
| } | ||
| } |
There was a problem hiding this comment.
The interceptors.request.clear() call is now executed unconditionally for every request. This will remove any default interceptors set on the gaxiosInstance even if no new interceptors are provided in reqOpts. Furthermore, clearing and adding interceptors on a shared instance is not safe for concurrent requests. It should at least be wrapped in the if (reqOpts.interceptors) block as it was before, although a better approach would be to avoid modifying the shared instance's interceptors entirely.
| this.gaxiosInstance.interceptors.request.clear(); | |
| if (reqOpts.interceptors) { | |
| this.gaxiosInstance.interceptors.request.clear(); | |
| for (const inter of reqOpts.interceptors) { | |
| this.gaxiosInstance.interceptors.request.add(inter); | |
| } | |
| } | |
| if (reqOpts.interceptors) { | |
| this.gaxiosInstance.interceptors.request.clear(); | |
| for (const inter of reqOpts.interceptors) { | |
| this.gaxiosInstance.interceptors.request.add(inter); | |
| } | |
| } |
There was a problem hiding this comment.
required for retry operations fix in conformance Test PR
There was a problem hiding this comment.
What is the retry operations fix and why do we need to make this change in the current PR ? Same for the other comments below.
There was a problem hiding this comment.
To support custom headers required by conformance tests without creating race conditions where concurrent async requests overwrite each other's configuration on a shared instance, we modified makeRequest to isolate custom interceptors using a fresh, request-scoped Gaxios instance, while explicitly stripping the adapter property during cloning to prevent infinite recursion and heap out-of-memory crashes between GoogleAuth and Gaxios.
| const responseHandler = (resp: GaxiosResponse<T>) => { | ||
| // eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
| const data = resp.data as any; | ||
| if (data !== null && typeof data === 'object') { | ||
| data.headers = resp.headers; | ||
| data.status = resp.status; | ||
| return data; | ||
| } | ||
| return resp; | ||
| }; |
There was a problem hiding this comment.
The responseHandler introduces an inconsistency in the return type of makeRequest. If the response data is an object, it returns the data mutated with headers and status. If the response data is not an object (e.g., a string or null), it returns the entire GaxiosResponse object. This violates the expected return type T and will likely cause runtime errors in callers expecting raw data. Additionally, mutating the data object directly is risky as it might be a frozen object or a class instance that doesn't support these extra properties. It is recommended to return resp.data consistently as before.
const responseHandler = (resp: GaxiosResponse<T>) => resp.data;There was a problem hiding this comment.
required for retry operations
There was a problem hiding this comment.
Please elaborate as stated in the above comment.
There was a problem hiding this comment.
To preserve backward compatibility for legacy SDK methods that expect resource metadata objects to carry .headers and .status properties without introducing runtime errors or mutating binary data, we added a strict validation check to ensure these properties are strictly injected into plain JSON response objects and never onto Buffer or Stream payloads.
There was a problem hiding this comment.
I agree with using isPlanObject to avoid mutating Buffer or Stream payloads.
However the issue mentioned above refers to the fallback "return resp;" statement. If the payload is a Buffer or Stream and the check fails, returning the entire GaxiosResponse wrapper instead of the raw data might break backward compatibility for binary downloads, since it previously resolved with just the data.
What do you think about returning data instead in that fallback case to keep the return types consistent?
There was a problem hiding this comment.
Great catch. Simply returning data broke resumable uploads, which rely on resp.headers.location when the API returns an empty string. I've updated the logic to handle all three scenarios correctly:
- Plain objects: Mutate and return
data. - Buffers/Streams (non-plain objects): Return raw
datato protect binary downloads. - Primitives (empty strings/null): Return the full
respwrapper so session URIs can still be read from the headers.
da2ee2d to
b4ee48b
Compare
b4ee48b to
b2e12e7
Compare
Hoists the generation of `persistentInvocationId` to the beginning of the upload process in `Bucket.upload` and `File.save`. This ensures that retried multipart upload attempts reuse the same invocation ID in the `x-goog-api-client` header, rather than generating a new one for each attempt.
5b69a10 to
e232f4e
Compare
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
011f5b5 to
1ae557f
Compare
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
1ae557f to
cc411a0
Compare
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
cc411a0 to
9010041
Compare
…ipart-invocation-id
…sport (googleapis#8283) - Remove Service.ts and common.ts files from handwritten/storage - Migrate remaining functionality to StorageTransport - chore(ci): upgrade conformance tests to Node 18
9010041 to
0e8f067
Compare
…ipart-invocation-id
…ion flag to storage transport
| reqOpts.queryParameters?.ifGenerationMatch !== undefined || | ||
| reqOpts.queryParameters?.ifMetagenerationMatch !== undefined || | ||
| reqOpts.queryParameters?.ifSourceGenerationMatch !== undefined || | ||
| bodyStr.includes('"etag"') |
There was a problem hiding this comment.
On the bodyStr.includes("etag") check
- Is it possible that this could cause false positive incase a user uploads the word "etag"
- If reqOpts.data is a object, bodyStr defaults to '', so it is possible that we would miss an existing etag entirely.
There was a problem hiding this comment.
Spot on. The .includes('"etag"') check was brittle. I've updated this to properly inspect the payload as an object (safely parsing it first if it's a stringified JSON) to check for an actual etag key. This eliminates both false positives on raw text and false negatives on object payloads.
And as per the comment below, this refactor was removed here and added to PR #8235.
|
|
||
| const bodyStr = typeof reqOpts.body === 'string' | ||
| ? reqOpts.body | ||
| : (typeof reqOpts.data === 'string' ? reqOpts.data : ''); |
There was a problem hiding this comment.
these changes are present both in this pr and PR #8235. Since these changes look retry related are they required in this pr ?
There was a problem hiding this comment.
You're completely right. I added these temporarily during refactoring to keep the conformance tests green, but missed removing them before updating this PR. I have removed them now!
0e8f067 to
72c17d7
Compare
…ipart-invocation-id
Hoists the generation of
persistentInvocationIdto the beginning of theupload process in
Bucket.uploadandFile.save. This ensures that retried multipart upload attempts reuse the same invocation ID in thex-goog-api-clientheader, rather than generating a new one for each attempt.Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> 🦕