Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,16 @@
.env.*
!.env.sample

# Documentation and repos
# Documentation, scratch, and repos (not needed to build/run the server)
docs/
repos/
scratch/
queries/
packs/
*.md

# Local tool downloads (the ./bun wrapper caches Bun here)
download/

# CLI build artifacts
packages/cli/dist/
Expand Down
48 changes: 40 additions & 8 deletions .env.sample
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,30 @@

# PostgreSQL connection string for the application database. One database holds
# the auth + core control plane and every per-space me_<slug> schema.
DATABASE_URL=postgres://postgres:postgres@localhost:5432/me
DATABASE_URL=postgres://postgres:postgres@localhost:5432/postgres

# Public base URL for OAuth callbacks
API_BASE_URL=http://localhost:3000

# OpenAI API key (or compatible provider) for generating embeddings
EMBEDDING_API_KEY=

# better-auth signing secret: signs session cookies and encrypts the JWKS keys.
# Required — the server refuses to boot without it. Use a long random value,
# e.g. `openssl rand -base64 32`.
BETTER_AUTH_SECRET=

# -----------------------------------------------------------------------------
# Optional — Docker Compose (self-host)
# -----------------------------------------------------------------------------

# Password for the bundled Postgres container in compose.yaml. Required by
# `docker compose up` (the server's DATABASE_URL is built from it); ignored by
# host-run dev (`./bun run server`), which uses DATABASE_URL above. Use a
# URL-safe value (letters/digits) so it needs no encoding in the connection
# string. See SELF_HOST.md.
# POSTGRES_PASSWORD=

# -----------------------------------------------------------------------------
# OAuth Providers
# -----------------------------------------------------------------------------
Expand Down Expand Up @@ -47,22 +63,37 @@ GOOGLE_CLIENT_SECRET=
# AUTH_SCHEMA=auth
# CORE_SCHEMA=core

# Cron schedule for cleaning up expired device authorizations (UTC)
# DEVICE_FLOW_CLEANUP_CRON=*/15 * * * *
# Cron schedule for the auth cleanup job, e.g. expired device authorizations
# (UTC). The legacy name DEVICE_FLOW_CLEANUP_CRON is still honored.
# AUTH_CLEANUP_CRON=*/15 * * * *

# Directory of the built web UI to serve at root `/` (the Docker image bakes
# this in; override only for host runs with a non-default build location).
# WEB_DIST=packages/web/dist

# Extra origins allowed to make cookie-authenticated requests (CSRF gate),
# comma-separated. The API_BASE_URL origin is always allowed; add others here
# (e.g. to permit both api.* and app.* during a cutover).
# WEB_ALLOWED_ORIGINS=

# -----------------------------------------------------------------------------
# Optional — Telemetry
# -----------------------------------------------------------------------------

# Logfire token for OpenTelemetry export (omit to disable)
# Logfire token for OpenTelemetry export. Default: unset (no export). The token
# is project-scoped, so data only appears in that token's own Logfire project.
# LOGFIRE_TOKEN=

# LOGFIRE_ENVIRONMENT=prod
# Deployment environment label for telemetry. Default: unset.
# Example: LOGFIRE_ENVIRONMENT=prod
# LOGFIRE_ENVIRONMENT=

# Print spans to console (useful for local development)
# Print spans to the console. Default: off — set to true to enable
# (useful for local development).
# LOGFIRE_CONSOLE=true

# Disable scrubbing of sensitive fields (content, embeddings, tokens)
# Scrubbing of sensitive fields (content, embeddings, tokens) is ON by default.
# Set to false to disable.
# LOGFIRE_SCRUBBING=false

# -----------------------------------------------------------------------------
Expand Down Expand Up @@ -142,7 +173,8 @@ GOOGLE_CLIENT_SECRET=
# The embedding worker uses a dedicated pool. Each setting defaults to the
# corresponding application-pool value (or DATABASE_URL) when unset.

# WORKER_DATABASE_URL=postgres://postgres:postgres@localhost:5432/me
# WORKER_DATABASE_URL=postgres://postgres:postgres@localhost:5432/postgres
# Defaults to max(WORKER_COUNT, 1), not a fixed 2.
# WORKER_DB_POOL_MAX=2
# WORKER_DB_POOL_IDLE_REAP_SECONDS=300
# WORKER_DB_POOL_MAX_LIFETIME=0
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Always use the `./bun` wrapper script (auto-installs the pinned Bun version):
**Important — verification runs against the local Postgres**: after making
code changes, run `./bun run check` (fast, no DB). Before committing, run
`./bun run check:full` — it defaults to the `me-postgres` Docker container
(if it isn't running: `docker start me-postgres || ./bun run pg`). Only run
(if it isn't running: `docker start me-postgres || ./bun run pg:docker`). Only run
against ghost when explicitly asked to test against ghost. CI is the strict
gate: it runs every suite with `TEST_CI=1`, which disables conditional skips
— any new `describe.skipIf` gate **must** include `!process.env.TEST_CI` in
Expand All @@ -114,7 +114,7 @@ its condition (pattern: `packages/embedding/generate.test.ts`,
`*.integration.test.ts` files run against a real PostgreSQL 18 with the
required extensions (citext, ltree, pgvector, pg_textsearch). Everything
defaults to the **local `me-postgres` Docker container** at 127.0.0.1:5432
(same image CI builds; `./bun run pg` creates it). `test:db` is the focused
(same image CI builds; `./bun run pg:docker` creates it). `test:db` is the focused
variant: it first reclaims orphaned test schemas, then runs **every**
`*.integration.test.ts` under `packages/` (the auth/core/space migration
suites plus the engine/server/worker suites), `--parallel=2`, 30s timeout:
Expand Down
70 changes: 56 additions & 14 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ memory-engine itself as that can be confusing for the model.

```bash
./bun install
./bun run pg
./bun run pg:docker
# configure .env (see below)
./bun run setup
./bun run server
Expand All @@ -63,10 +63,10 @@ memory-engine itself as that can be confusing for the model.
The project requires PostgreSQL 18 with three extensions: `pgvector`, `pg_textsearch`, and `ltree`. The included Dockerfile builds a pre-configured image:

```bash
./bun run pg
./bun run pg:docker
```

This builds the Docker image and starts a container named `me-postgres` on `localhost:5432` with trust authentication (no password).
This builds the Docker image and starts a container named `me-postgres` on `localhost:5432` with trust authentication (no password). It serves the default `postgres` database, which is the one `.env.sample` targets.

Other database commands:

Expand All @@ -88,7 +88,7 @@ cp .env.sample .env
**Database connection** — one database holds the `auth` + `core` control plane and every per-space `me_<slug>` schema:

```
DATABASE_URL=postgres://postgres@localhost:5432/memory_engine
DATABASE_URL=postgres://postgres@localhost:5432/postgres
```


Expand Down Expand Up @@ -144,7 +144,7 @@ GITHUB_CLIENT_SECRET=...

```bash
# Database
DATABASE_URL=postgres://postgres@localhost:5432/memory_engine
DATABASE_URL=postgres://postgres@localhost:5432/postgres

# Server
API_BASE_URL=http://localhost:3000
Expand Down Expand Up @@ -173,19 +173,58 @@ specified in milliseconds, for example `RPC_DB_STATEMENT_TIMEOUT_MS=30000`.
./bun run setup
```

This is idempotent (safe to run multiple times) and will:

1. Create the `accounts` and `shard1` databases if they don't exist
2. Run account schema migrations
3. Create and activate an encryption data key
4. Bootstrap the engine database (extensions and roles)
This is idempotent (safe to run multiple times). It reads `DATABASE_URL` and
creates that database if it doesn't already exist (everything else —
bootstrap, migrations, encryption keys — happens automatically at server
startup). When `DATABASE_URL` targets `/postgres` (the `me-postgres`
container's default database, as in `.env.sample`), this is effectively a
no-op since that database already exists — running it is still harmless.

### 5. Start the server

```bash
./bun run server
```

### 5b. Run the server in Docker (optional)

To exercise the actual production image locally — the multi-stage
`packages/server/Dockerfile` that CI builds and deploys — use the `server:*`
scripts (the Docker counterparts to `pg:*`):

```bash
./bun run server:docker # build the image + run in the foreground (me-server)
./bun run server:build # build the image only
./bun run server:rm # force-remove the container (only needed if it leaks)
```

`server:docker` runs the container in the **foreground** (`--rm -t`): logs
scroll in the terminal, `Ctrl+C` triggers the server's graceful shutdown, and
the container is removed on exit. It publishes the server on `127.0.0.1:3000`
and wires it to the `me-postgres` container. Prerequisites:

- Postgres running (`./bun run pg:docker`).
- A populated `.env` (the container is started with `--env-file .env`, so it
reads `EMBEDDING_API_KEY`, the OAuth credentials, telemetry, etc. from there).

How it reaches Postgres: the script overrides `DATABASE_URL` to
`postgres://postgres@host.docker.internal:5432/postgres` (and adds
`--add-host=host.docker.internal:host-gateway` so the hostname resolves on
Linux too). A container can't reach the host's `localhost`, so this override
replaces the `localhost` value your `.env` uses for host-run development.
`-e` takes precedence over `--env-file`, so the rest of `.env` still applies.

Caveats:

- The script hardcodes port `3000`. If you set a different `PORT` in `.env`,
edit the `server:docker` script's published port and keep `API_BASE_URL`
consistent.
- The override only covers `DATABASE_URL`. If you uncomment
`WORKER_DATABASE_URL` in `.env`, it must also use `host.docker.internal`.
- `docker run --env-file` parses literal `KEY=VALUE` lines — no quotes, no
`$VAR` expansion (unlike Bun's `.env` loader). `.env.sample` is already
compatible.

### 6. Test with the CLI

In another terminal:
Expand All @@ -207,11 +246,14 @@ After login, the server URL is stored as the default in `~/.config/me/credential

| Command | Description |
|---|---|
| `./bun run server` | Start the server |
| `./bun run setup` | Create databases, run migrations, bootstrap engine |
| `./bun run pg` | Build and start PostgreSQL in Docker |
| `./bun run server` | Start the server (on the host) |
| `./bun run setup` | Ensure the `DATABASE_URL` database exists |
| `./bun run pg:docker` | Build and start PostgreSQL in Docker |
| `./bun run pg:rm` | Stop and remove the PostgreSQL container |
| `./bun run psql` | Connect to PostgreSQL with psql |
| `./bun run server:build` | Build the server Docker image (`me-server`) |
| `./bun run server:docker` | Build + run the server in Docker (vs `me-postgres`) |
| `./bun run server:rm` | Stop and remove the server container |
| `./bun run test` | Run all package tests (unit + integration, vs local Postgres by default) |
| `./bun run check` | Fast inner loop: typecheck + lint + unit tests (no database) |
| `./bun run check:full` | Everything: check + full suite + e2e (vs local Postgres by default) |
Expand Down
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@ Memory Engine runs as an MCP server that AI agents connect to over stdio. Each a
- **tstzrange** for temporal queries
- **Tree-scoped access grants** evaluated in the search SQL (no RLS)

## Self-hosting

Want to run your own Memory Engine backend? See **[Self-Hosting](SELF_HOST.md)**
— a Docker Compose stack (server + PostgreSQL), built from a tagged release,
plus building the `me` CLI from source to connect to it.

## Documentation

- [Getting Started](docs/getting-started.md) -- install, login, first memory
Expand Down
Loading