Customize Headers in network search remote requests by joneubank · Pull Request #1066 · overture-stack/arranger

joneubank · 2026-05-20T23:35:27Z

Summary

Introduces a mechanism to customize the network requests made to remote nodes during network search. With this initial setup, we have added the ability to add custom headers to outgoing requests to each remote node sent during network aggregation searches.

When the arranger graphql-router is created, an optional function can be provided as part of the network config. This CustomizeRemoteRequestFn will receive the Arranger context (including incoming request headers), as well as the config for a remote node. This function can then determine whether to inject headers into the outgoing request.

The search-server has used this to enable a new network configuration: passthroughHeaders. If the network configuration file includes a passthroughHeaders property (string[]), then headers of the given name will be taken from the incoming search query and added to the outgoing request to every remote node.

This functionality is requested to enable transporting Authorization credentials from the requesting user through to the other nodes in the network which have implemented a custom authorization scheme in front of the arranger servers.

Description of Changes

modules/graphql-router

Adds to the ArrangerBaseContext information about the incoming request
- headers are made available with this change. Future additions could expand this.
Added to graphqlRouter the customizeRemoteRequest optional config property.
- This takes a function that can add properties to each outgoing request to a remote node in a network query.
- The output of this function can supply headers to add to the outgoing request
- This function is called once for each remote node, so that custom logic can be provided for modifying each node's request.

apps/search-server

network config now accepts a passthroughHeaders property with an array of string values
- headers with a name matching a passthroughHeaders value will be copied from the incoming request into each of the outgoing requests to a remote node
- Updated the example config files to demonstrate the usage of this new property

Readiness Checklist

Self Review
- I have performed a self review of code
- I have run the application locally and manually tested the feature
- I have checked all updates to correct typos and misspellings
Formatting
- Code follows the project style guide
- Autmated code formatters (ie. Prettier) have been run
Local Testing
- Successfully built all packages locally
- Successfully ran all test suites, all unit and integration tests pass
Updated Tests
- Unit and integration tests have been added that describe the bug that was fixed or the features that were added
Documentation
- All new environment variables added to .env.schema file and documented in the README
- All changes to server HTTP endpoints have open-api documentation
- All new functions exported from their module have TSDoc comment documentation

Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* Use SearchQueryResponse * createElasticClient * Types file updated * createOpenSearch updates * Use overture-stack/types import * SharedIndicesCreateParams type change * Remove yarn.lock * Update parameter types * Update Response Types * Minor edits * Search Params & Body updates, removed instance of Prettify, Update Import names * Revise Options & Config types * Minimal Search Options type * Consistent Naming

* Rename network config as nodes, remove config property to enable network search * search-server remove env flag for network search * Types clean up and improved nodes data reporting in network query - nodes hits now return if no aggregations are requested - nodes that failed to connect on startup are included in the nodes data query - overhauled the Result type to allow different data to be returned for each result case - created a seaprate config type for local vs remote nodes, not yet used * WIP: Enhanced types for graphql-router and network search on local node - Next Step is to define local node configs and build the local query from those - Converted all graphql-router/schema files to typescript - Added Context as a generic type for many of the Resolver types and related functions. This is used to pass information to the resolvers and custom resovler functions. At the moment this is only the server side filter function, but passing custom context to this is important. * Working network search within a catalog * Move shared types into dedicated files * Update config schema to match new config format * validateConfigs will list all missing properties from the config

joneubank · 2026-05-20T23:36:42Z

 The `SearchClient` abstraction already exists as the right boundary — the migration should align the types and default configuration to OpenSearch while preserving compatibility for ES users.

-**What's already done:** The integration test suite (`integration-tests/server`) already supports both engines via a `SEARCH_ENGINE` env var (defaults to `'elasticsearch'`). `buildSearchClient` accepts a `client` type parameter mapped to `SupportedSearchClients`. The architecture is ready; the missing pieces are the OpenSearch client dependency and a running OpenSearch instance in CI.
+**What's already done:** The integration test suite (`integration-tests/server`) already supports both engines via a `SEARCH_ENGINE` env var (defaults to `'elasticsearch'`). `buildSearchClient` accepts a `client` type parameter mapped to `SupportedClientTypes`. The architecture is ready; the missing pieces are the OpenSearch client dependency and a running OpenSearch instance in CI.


There is no type named SupportedSearchClients, I believe this is leftover before a name change. I have replaced all instances if this with SupportedClientTypes and this SHOULD work as intended. If needed, suggest the desired name and we can make this consistent before merging (or resolve in a separate PR).

I strongly prefer the more descriptive name I had applied (this needs to be read as a configuration option, not a code compliance thing), but we can redo that change once this is merged (along with a couple other name changes decided elsewhere), so your PR's scope doesn't grow. 👍

joneubank · 2026-05-20T23:37:21Z

+ *   - Ensures every entry in `table.defaultSorting` has `desc` explicitly set,
+ *     defaulting to `false` when the field is absent.
+ */
+const normalize = (fileDataJSON: any) => {


I don't think this is the best function name, but I'm trying to refactor only lightly as I add features.

joneubank · 2026-05-20T23:38:33Z

+export type ExternalNetworkConfig = Partial<{
+	[serverNetworkConfigExtendedProperties.PASSTHROUGH_HEADERS]: string[];
+}> &
+	NetworkConfig<ArrangerBaseContext>;


Config file types and graphql-router types are diverging, which is to be expected. It would be worthwhile doing a dedicated task on separating the types for these two contexts, and to provide validation schemas (zod) for the config files.

justincorrigible · 2026-05-21T13:34:29Z


 	// Network Search Nodes
 	"network": {
+		"passthroughHeaders": ["Authorization"],


I see what you're doing here, and could use some documentation (even if only as a comment in the line above)

Yeup, I owe better (read: some) documentation for this feature

justincorrigible · 2026-05-21T13:36:15Z

@@ -1,3 +1,13 @@
 {
-	"network": []
+	"network": {


justincorrigible · 2026-05-21T13:47:40Z

 /**
- * Normalizes raw config file JSON for use by the search server.
+ * This will assume the network config provided has the expected format, it is not fully validated.
+ * TODO: Fully validate config file structure


nice! added an item to this effect in the roadmap, to do it both her and in the core module, with proper error feedback, and to add the corresponding tests

justincorrigible · 2026-05-21T15:00:12Z

-		"test": "NODE_OPTIONS=\"$NODE_OPTIONS --experimental-vm-modules\" jest --config=jest.config.ts"
-	},
-	"type": "module"
+		"test": "jest --config=jest.config.ts"


did you test this? I think it breaks compatibility on some imports, hence why I added it. seems beyond the scope of this PR tbh

Another merge conflict issue, will restore the script from main.

justincorrigible · 2026-05-21T15:03:57Z

-		[configOptionalProperties.GRAPHQL_MAX_ALIASES]: number;
-		[configOptionalProperties.GRAPHQL_MAX_DEPTH]: number;


please readd these. looks like merge conflict mistake

may I ask to confirm the related logic wasn't also removed?

justincorrigible · 2026-05-21T15:13:39Z

+				const headers = new Headers();
+				for (const [key, value] of Object.entries(req.headers)) {
+					if (value !== undefined) {
+						const valueAsString = Array.isArray(value) ? value.join(', ') : value;
+						headers.set(key, valueAsString);
+					}
+				}
+				const request: RequestContextProps = { headers };
+
+				// Add to context based on external parameters


while I of course may be misreading the passthrough logic overall, it looks like this applies the same headers to every remote node in the network. i.e. there's no per-node control.

for instance If remote nodes have different trust levels, e.g. one is internal, another is a third-party node, forwarding Authorization to all of them may leak credentials.

... either the docs/config schema should explicitly warn that this affects all nodes equally, or better yet, this should be an opt-in kind of deal so we don't go against OWASP A01: Broken Access Control recommendations.

justincorrigible · 2026-05-21T15:18:43Z

+		headers: {
+			'Content-Type': 'application/json',
+			'X-REQUEST-TYPE': 'GraphQL',
+			...(customRequestProps?.headers || {}),


hm yeah, here it is... the passthrough thing is non-discriminate and could leak credentials. thinking the configuration should state explicitly which nodes to send which headers to, somehow (granted, that sounds complex)

justincorrigible · 2026-05-21T15:20:36Z

-const isFieldAggregationSupported = (fieldObject: AggregationField): fieldObject is SupportedAggregationField => {
+export const isFieldAggregationSupported = (
+	fieldObject: AggregationField,
+): fieldObject is SupportedAggregationField => {


i really need to wrap my head around this "is" thing. I forget it exists and can be so useful

I actually strongly discourage using type predicates now. TS will infer the predicate logic if the function returns boolean and narrows the type of the input variable. If TS can't calculate that the type is actually narrowed correctly then manually declaring the type predicate is about as risky as using any type assertion.

justincorrigible · 2026-05-21T15:22:27Z

 			if (response.success) {
 				const documentName = config.documentType;
-				const responseData = response.data[documentName];
+				// TODO: Response content is not validated, we expect the return structure based on the GraphQL query we requested


this sounds easier than it may actually be, from a model-agnostic perspective, considering an aggregator may not be fully aware of the nodes' model and harmonization (or at least some mapping) may be necessary

justincorrigible · 2026-05-21T15:23:48Z

 		[facetsProperties.AGGS]: [],
 	},
 	[configRootProperties.MATCHBOX]: [],
-	[configRootProperties.NETWORK_AGGREGATION]: { [configArrangerNetworkProperties.REMOTE_NODES]: [] },


surprised at first, but i guess it makes sense in the decoupling of "core" from "aggregator" types 👍

This is because I updated the config type to make the remoteNodes property optional so there is no need to set an empty list by default.

justincorrigible

Great work overall (as usual)! 👏
two items of "concern" I'd like to surface here:

the OWASP a01 thing I pointed out, passing headers to all nodes.
the possible regression on GraphQL security functionality (thanks for putting up with the Git shenanigans)

@justincorrigible

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

@justincorrigible

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

justincorrigible and others added 21 commits March 5, 2026 15:04

add SQON module, define schema and types

753de6c

abstract server and types from main Arranger module

408bd6d

add Arranger Charts

b60de0a

Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

restore integration tests functionality

0035c8b

add better CORS management

3d53b80

add introspection endpoints

901fc9a

add turborepo for multiversion pipeline support

aa815c2

fix build integration issues

4919f73

Merge branch 'multiversion' into es-multiversion

ff24372

Merge branch 'searchengines' into 3.1

695a404

add MCP server foundation

c74cc0c

add multicatalog functionality (#1049)

c5d5551

improve multicatalog scaffolding, add testing for it

f03738a

Merge branch 'release-charts' into 3.1

ffc1f14

remove baseUrl from tsconfigs for TS@7

7415c9c

#1055: add "select all" button to facet panel (#1056)

c2435d3

Optional customizable headers in remote node network requests

9c4f52a

Optional customizable headers in remote node network requests

56a176e

Merge branch 'main' into feat/custom-remote-node-requests

42c1bab

joneubank requested review from demariadaniel and justincorrigible May 20, 2026 23:35

joneubank commented May 20, 2026

View reviewed changes

joneubank added 3 commits May 20, 2026 19:39

Remove unintended logging text changes

da84267

Restore unintentional formatting change

6f6f423

Fixing merge errors

b48c448

justincorrigible reviewed May 21, 2026

View reviewed changes

Comment thread apps/search-server/src/configs/fromFiles/normalize.ts Outdated

justincorrigible reviewed May 21, 2026

View reviewed changes

Comment thread apps/search-server/src/configs/fromFiles/normalize.ts

add TODO comment

d43d605

justincorrigible reviewed May 21, 2026

View reviewed changes

Comment thread apps/search-server/configTemplates/network.json Outdated

justincorrigible reviewed May 21, 2026

View reviewed changes

justincorrigible requested changes May 21, 2026

View reviewed changes

joneubank and others added 3 commits May 21, 2026 14:46

passthroughHeaders template as empty array

3206dd0

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

Remove unnecessary undefined check

42b67ad

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

Fix merge conflicts

587535f

		[configOptionalProperties.GRAPHQL_MAX_ALIASES]: number;
		[configOptionalProperties.GRAPHQL_MAX_DEPTH]: number;

Conversation

joneubank commented May 20, 2026

Summary

Description of Changes

modules/graphql-router

apps/search-server

Readiness Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

justincorrigible May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

justincorrigible May 21, 2026 •

edited

Loading

justincorrigible May 21, 2026 •

edited

Loading

justincorrigible May 21, 2026 •

edited

Loading

justincorrigible May 21, 2026 •

edited

Loading

justincorrigible May 21, 2026 •

edited

Loading

justincorrigible May 21, 2026 •

edited

Loading