Skip to content

feat(agent-builder): author, save, and dry-run custom tools#3237

Open
dmarticus wants to merge 4 commits into
mainfrom
posthog-code/custom-tool-authoring-dry-run
Open

feat(agent-builder): author, save, and dry-run custom tools#3237
dmarticus wants to merge 4 commits into
mainfrom
posthog-code/custom-tool-authoring-dry-run

Conversation

@dmarticus

Copy link
Copy Markdown
Contributor

What

Wires the backend's custom-tool authoring loop (posthog#66584) into PostHog Code. Until now the agent builder could only read tool source (tools/<id>/source.ts, read-only) — this adds the client + a minimal GUI to create, save, and dry-run a custom tool end to end, behind the existing agent-platform flag.

This is v0: correctness over polish. No capabilities panel beyond a save-time readout, no monaco intellisense, no per-team quota UX — those are v1.

How

Part A — API client + types (hand-written; agent-platform types are not generated from OpenAPI)

  • @posthog/shared/agent-platform-types: ToolCompileError/ToolCompileErrorKind, ToolCapabilities, WriteToolRequest/WriteToolResult, DryRunToolRequest/DryRunToolEnvelope/DryRunToolResult.
  • posthog-client.ts:
    • putRevisionTool — 422 returns a typed compile-failed result (rendered inline); other non-2xx (400/409 sealed/…) still throw.
    • deleteRevisionTool — 404 (tool_not_found) treated as success.
    • dryRunRevisionTool — reads ok from the body (a throwing tool is HTTP 200 with ok:false); 200 and 500 both return the envelope; 429 → throttled (with max_concurrent), 503 → unavailable. Never thrown, never retried.
    • parseFailedRequest helper recovers status + body from the shared fetcher's Failed request: [status] body throw.

Part B — hooks

  • useSaveRevisionTool (invalidates the bundle(...) query only when the compile persisted), useDeleteRevisionTool (invalidates bundle), useDryRunRevisionTool (no invalidation). All expose isPending.

Part C — UI (agent builder)

  • ToolSourceEditor — draft-only editable source with compile-on-save, inline ToolCompileError diagnostics (kind + message + 1-based line/column), Revert, double-submit guard, and a secret_refs / dynamic_secret_refs readout from the save response.
  • ToolDryRunPanel — args JSON editor, optional mock_secrets key/value rows, Test button, and a result area handling success (result + duration_ms), tool failure (error.code/message), 429 busy, and 503 unavailable.
  • ToolBody gates edit + dry-run behind AGENT_PLATFORM_FLAG; non-draft / non-custom tools stay read-only.

Testing

  • pnpm typecheck (23/23 packages) and pnpm biome check on all touched files pass.
  • New client tests: 200/422/409 writes, delete 404-as-success, dry-run 200-ok / 200-fail / 500 / 429 / 503 + mock_secrets passthrough. New hook test for the invalidate-only-when-persisted branch. @posthog/api-client 78 pass, @posthog/ui 1325 pass.

Note

End-to-end dry-run requires the deployed backend (janitor + main app) to include posthog#66584 with the sandbox backend configured. Until then dry-run returns 503, which is the expected "not available here" state this UI is built against.

🤖 Generated with Claude Code

Wire the backend's custom-tool authoring loop (posthog#66584) into PostHog
Code so an internal user can create, save, and dry-run a custom tool end to
end, behind the existing agent-platform feature flag. v0: correctness over
polish.

API client + types (hand-written; agent-platform types are not generated):
- Add ToolCompileError(Kind), ToolCapabilities, WriteToolRequest/Result,
  DryRunToolRequest/Envelope/Result to @posthog/shared.
- Add putRevisionTool (422 -> typed compile-failed result, inline; other
  non-2xx throw), deleteRevisionTool (404 -> success), and dryRunRevisionTool
  (reads `ok` from the body; 200 and 500 return the envelope; 429 -> throttled;
  503 -> unavailable; never thrown, never retried). A parseFailedRequest helper
  recovers status + body from the shared fetcher's throw.

Hooks:
- useSaveRevisionTool (invalidates the bundle only when the compile persisted),
  useDeleteRevisionTool, useDryRunRevisionTool. All expose isPending.

UI (agent builder):
- ToolSourceEditor: draft-only editable source with a compile-on-save,
  inline ToolCompileError diagnostics (kind + message + 1-based line/column),
  double-submit guard, and a capabilities readout from the save response.
- ToolDryRunPanel: args JSON editor, optional mock_secrets rows, Test button,
  and a result area handling success / tool failure / 429 busy / 503 unavailable.
- Gate edit + dry-run behind AGENT_PLATFORM_FLAG; non-draft/non-custom stays
  read-only.

Tests: client cases for 200/422/409 writes, delete 404-as-success, and dry-run
200-ok/200-fail/500/429/503 + mock_secrets passthrough; a hook test for the
invalidate-only-when-persisted branch.

Note: end-to-end dry-run needs the deployed backend (posthog#66584) with the
sandbox configured; until then it returns 503, the expected "not available
here" state.

Generated-By: PostHog Code
Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba
@trunk-io

trunk-io Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

React Doctor found no issues in the changed files. 🎉

Reviewed by React Doctor for commit 89510b2.

@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff (1)

  1. packages/ui/src/features/agent-applications/components/ToolSourceEditor.tsx, line 982 (link)

    P2 Uncleaned timeout on unmount

    setTimeout is started but never cleared, so if the component unmounts within the 2-second window (e.g. the user navigates away right after saving), the captured setJustSaved closure fires on a stale closure. React 18 makes this a silent no-op, but the pattern accumulates stale closures. Store the timer ID in a useRef and clear it in a cleanup callback or on the next onSave call.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (1): Last reviewed commit: "feat(agent-builder): author, save, and d..." | Re-trigger Greptile

Comment thread packages/api-client/src/posthog-client.ts
dmarticus added 2 commits July 7, 2026 09:38
…e timer

- Make DryRunToolResult.throttled `max_concurrent` optional and omit it on a
  429 without a numeric count, so the panel no longer renders a misleading
  "0 runs in flight". The dry-run panel shows a count only when present.
- Clear the "saved" tick setTimeout in ToolSourceEditor on unmount and before
  the next save (via a ref) so a stale timer can't setState after unmount.
- Add a client test for a 429 with no max_concurrent.

Generated-By: PostHog Code
Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba
- ToolDryRunPanel: give mock-secret rows a stable id (ref counter) and key/
  update/remove by it, instead of the array index.
- ToolSourceEditor: make the source textarea uncontrolled (defaultValue + ref)
  and track only a `dirty` flag, dropping the prop-derived useState and the
  in-render reconcile. The parent keys the editor by revision+tool so switching
  either remounts it with a fresh buffer.
- Key each compile-error row by its kind/line/column/message instead of the
  array index.

Generated-By: PostHog Code
Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba
@dmarticus dmarticus requested a review from a team July 7, 2026 17:10
The tool detail view showed an always-open source textarea and an always-
expanded dry-run panel on draft tools, which made the node look like a code
editor bolted into the builder. Extract a ToolSourcePanel that defaults to a
collapsed, read-only view:

- Resting state is a `source · <path>` header plus a "Show source" toggle
  (read-only CodeBlock when expanded) — no editor, no open dry-run panel.
- "Edit source" (draft-only) is an explicit toggle that swaps in the inline
  editor; "Test tool" is an explicit toggle that reveals the dry-run panel,
  with a "Hide test" to collapse it.
- Keyed by revision+tool so switching either resets the toggles and reseeds
  the editor buffer.

ToolSourceEditor/ToolDryRunPanel are unchanged — just moved behind the toggles.

Generated-By: PostHog Code
Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant