v1.1.0-beta.1 — in-app scheduler, Python 3.13, per-node Sentry env, optional parallel fan-out#61
Open
NikolaosSokos wants to merge 51 commits into
Open
v1.1.0-beta.1 — in-app scheduler, Python 3.13, per-node Sentry env, optional parallel fan-out#61NikolaosSokos wants to merge 51 commits into
NikolaosSokos wants to merge 51 commits into
Conversation
…d to database name
Changes: - Added Docker memory limits. - Externalized configuration. - Implemented Redis-powered breadth check. - Applied .limit() to MongoDB. - Added protection tests. - Updated .gitignore to exclude AI metadata and locks.
Brings in 5 upstream commits not yet on this branch: 9d0a188 fix: make sentry configuration optional to support legacy config.py 3346ed0 Merge branch 'master' of NikolaosSokos/ws-availability 6ac2e31 docs: update README with troubleshooting section 5d65168 fix: resolve v1.0.4 installation issues 71e9374 fix: resolve v1.0.4 installation issues Required before opening the v1.1.0-beta.1 PR against EIDA/master. # Conflicts: # .gitignore # README.md # apps/globals.py # docker-compose.yml # pyproject.toml # start.py # uv.lock
The repository was tracking ws-availability-issue60-briefing.md, an internal handover document not meant for end users. This commit untracks the file (it remains on disk for the maintainer) and extends .gitignore to cover common AI/agent tooling directories (.claude/, .cursor/, .aider*, CLAUDE.md, AGENTS.md, plans/) plus future briefing-style files (*-briefing.md, *briefing*.md). The file remains accessible in git history for anyone who needs the context.
Bump the api and cacher container base images from python:3.10-slim to python:3.13-slim (Python 3.10 reaches end-of-life in October 2026). Loosen the dependency pins in pyproject.toml and regenerate uv.lock: - flask 2.3.2 -> 3.1.3 - gunicorn 20.1.0 -> 26.0.0 - obspy 1.3.1 -> 1.5.0 (1.3.1 does not import on Python 3.13) - pymongo 3.12.3 -> 4.17.0 - redis 4.4.4 -> 7.4.0 - requests 2.31.0 -> 2.34.2 requires-python is bumped to >=3.13 to match the container base. Full test suite (35 tests, excluding the pre-existing test_restriction.py import error) passes on the upgraded stack. The api code uses only stable APIs that did not change between pymongo 3 and 4 (find().limit() with projection kwarg), flask 2 and 3 (request, make_response), and redis 4 and 5 (get/set/setex with connection pool). No application-code changes were required for the upgrade.
Sentry events from every deployment were previously indistinguishable in
the dashboard. Add a SENTRY_ENVIRONMENT configuration field that tags
each event with a node-specific value (e.g. noa_production, noa_staging,
local_development) so operators can filter by deployment.
The field is read from Config.SENTRY_ENVIRONMENT in config.py.sample,
backed by the SENTRY_ENVIRONMENT environment variable, and passed to
sentry_sdk.init(environment=...) at both initialization sites
(start.py for the api, apps/sentry.py for the cacher).
To avoid breaking existing deployments with older config.py files, both
init sites read the attribute defensively via
getattr(Config, "SENTRY_ENVIRONMENT", None) and fall back to
"local_development" if unset.
The docker-compose.yml exposes the variable on both services with the
${SENTRY_ENVIRONMENT:-local_development} pattern, so operators set
the per-node value once in their shell or .env file before running
docker compose up.
Two tests broke when Issue EIDA#60 (commit 2179a8f) added a .limit() call to the MongoDB cursor: the tests set find().return_value to a list, but the new code calls find().limit() on that result, raising AttributeError on the list. The tests were silently failing. Replace the list return value with a MagicMock whose .limit() returns the desired iterable, restoring both tests to working order. No production code is touched. Affected: test_filter_invalid_data, test_mongo_request_query_construction.
For long-range /query and /extent requests, the existing mongo_request()
function issues one sequential MongoDB cursor with maxPoolSize=1. The
wall-clock is dominated by cursor round-trips to MongoDB even when each
batch is small. Sentry traces of multi-month single-channel requests
showed >80% of latency was spent waiting on cursor fetches.
This commit adds an opt-in code path that splits the [start, end] range
into day-aligned windows of FANOUT_WINDOW_DAYS each and runs them as
concurrent cursors on a ThreadPoolExecutor. The compound index
{net,sta,loc,cha,ts,te} (recommended in the README) makes each shard a
clean index range scan with no overlap, and pymongo releases the GIL
during socket I/O so the threads naturally interleave their work.
Four new settings (apps/settings.py):
FANOUT_ENABLED default False — master switch
FANOUT_MAX_WORKERS default 4 — also bumps Mongo maxPoolSize
FANOUT_MIN_DAYS default 7 — skip fan-out for short ranges
FANOUT_WINDOW_DAYS default 30 — target shard size
When FANOUT_ENABLED=false, mongo_request takes the legacy single-cursor
path unchanged. Response bodies are byte-identical between the two
paths for any request (modulo the JSON "created" timestamp) — verified
by a new unit test (test_fanout_result_equals_single_shot). The
existing MAX_DATA_ROWS cap and Issue EIDA#60 broad-query rejection still
apply per shard.
Sentry instrumentation: mongo_request opens a parent span "db.mongo",
and each fan-out shard opens a child "db.mongo.shard" span tagged with
its ts_start/ts_end window so the parallelism is visible in the trace.
Two new tests verify (a) one find() call when fan-out is off; (b) N
non-overlapping shard calls tiling [start, end) when fan-out is on.
README.md documents all four env vars and operational guidance for
enabling fan-out safely.
Bump pyproject.toml version to 1.1.0b1 (PEP 440 pre-release marker; docker image tag will be 1.1.0-beta.1 via the CI workflow's semver metadata-action). Bump apps/globals.py VERSION to "1.1.0-beta.1" so the /version endpoint reports the beta tag. Add BETA.md, a complete deployment and testing guide for EIDA node operators. The guide covers pre-flight checks, three deployment paths (fresh install via pre-built ghcr images, upgrade from v1.0.5, side-by-side test on a different port), a configuration reference table for all environment variables (including the new SENTRY_ENVIRONMENT and FANOUT_* family), enabling fan-out, post-deploy monitoring, what to test specifically, rollback procedures (to v1.0.5 images or feature disable without redeploy), an issue reporting template, and known issues. After the v1.1.0 GA release, BETA.md should be removed or replaced with the upstream release notes.
APScheduler's default misfire_grace_time is 1 second. If a scheduled job fires even slightly late — another job is still running, GIL contention, brief host pause — the run is silently skipped and the next fire is pushed to tomorrow. This was observed in production today (2026-05-23): the cacher container was restarted manually at 02:00:21 UTC, both startup runs succeeded, the 03:00 inventory rebuild ran fine, but the 06:00 materialized-view update fired 1.6 s late and was skipped. The function never executed, so the @sentry_sdk.monitor decorator never emitted a check-in, and Sentry alerted "missed check-in" for update-availability-view. APScheduler log line that exposed it: Run time of job "Builds the daily_streams aggregation into the availability materialized view (...)" was missed by 0:00:01.636588 Apply two changes per job: misfire_grace_time = 300 # 5 minutes; absorbs realistic jitter coalesce = True # collapse a backlog into a single run The 5-minute window is conservative — short enough that operators notice if something keeps the scheduler hung for that long, long enough that GIL/scheduler latency under any plausible load fits. coalesce=True prevents catchup storms after long downtime: if the cacher is offline overnight, only one make-up run fires when it returns, not eight. Fixes Sentry monitor 2691465f-cf32-41fa-836d-5d28811203d8.
Previously SENTRY_ENVIRONMENT was only sourced from the Config class attribute, which required every operator to add a line to their local config.py. Since docker-compose.yml already plumbs the env var into the container, allow start.py and apps/sentry.py to read it from os.environ as the second-priority source, before the local_development default. Resolution order is now: 1. Config.SENTRY_ENVIRONMENT (config.py) 2. os.environ["SENTRY_ENVIRONMENT"] (docker-compose / shell) 3. "local_development" default This means an operator can tag a deployment by either editing config.py or just exporting SENTRY_ENVIRONMENT before docker compose up — no file edit required.
Previously run_materialized_view_update() called updater.run_updates()
with no arguments, which defaults to processing only yesterday. If the
cacher container missed a daily fire (and Sentry caught one such miss
on 2026-05-23 due to a 1.6s misfire), the day's data would not appear
in the availability view until someone manually back-processed.
The legacy host cron on the deployment server runs the equivalent
JavaScript view-update at 01:00 daily with `daysBack=4`, exactly to
provide this safety net. The in-app scheduler must do the same so the
host cron can be retired without losing that guarantee.
Switch to:
updater.run_updates(days_back=4)
The MongoDB pipelines use `$merge whenMatched: "replace"`, so
re-processing the same 4 days each day rewrites the same documents
idempotently — no duplicates, no corruption. The aggregation runs in
seconds even at production volume.
After this lands, the host cron entry on the deployment server can be
removed (single source of truth for view updates is now the in-app
scheduler).
Operators of other EIDA nodes need to follow a single canonical path
to upgrade, not pick between branches. Rewrite §5 to be a numbered
recipe with no decision points in the happy path:
- §5.1 sets two shell variables once ($WSAVAIL_DIR and $NODE_TAG)
- §5.2 backs up config.py, the crontab, and the current commit hash
- §5.3 checks out v1.1.0-beta.1 with a verify step
- §5.4 writes a .env file with SENTRY_ENVIRONMENT=${NODE_TAG}_production
- §5.5 builds api + cacher images
- §5.6/§5.7 swap api then cacher one at a time, each with verification
- §5.8 removes the legacy host cron (if the node ever had one)
- §5.9 runs the same 5-check smoke test as a fresh install
- §5.10 confirms Sentry tags are flowing through
- §5.11 sets expectations for the first 24 h
Add a dedicated Path-B rollback block at the end of §5 with exact
copy-paste commands. The general §11 rollback still exists for
later post-deploy issues.
Every step has a verify command with the expected output, so an
operator who skips ahead can still detect a problem at the next
checkpoint.
Tested end-to-end on eida.gein.noa.gr today (Path B applied to the
ws-availability2025 deployment).
Cut the risk tables, "what's new" exposition, fan-out section, post-deploy monitoring, what-to-test scenarios, known issues, and feedback sections. They were useful context but not what an EIDA node operator needs at install time. Final structure: Prerequisites, Install (8 numbered steps with copy-paste commands), Verify (4 curls), Rollback. 506 lines -> 110 lines.
…split Drop $WSAVAIL_DIR and $NODE_TAG. Operators just cd into their existing checkout and run the commands as-is. config.py step is now idempotent: keep existing, otherwise copy from sample. No verb branching between "fresh install" and "upgrade".
Operators set everything in one place (config.py) again. Previously
SENTRY_ENVIRONMENT was plumbed through docker-compose.yml from a .env
file, which split configuration across two files and risked the
compose default (local_development) silently overriding a value set
elsewhere.
Changes:
- docker-compose.yml: remove the SENTRY_ENVIRONMENT env line from both
services so Compose no longer injects it. config.py is now the sole
source (the code's os.environ fallback stays as a harmless option).
- config.py.sample: SENTRY_ENVIRONMENT is now a plain, must-edit value
with a loud placeholder "{{node}}_production" and a REQUIRED banner,
so an operator who forgets gets an obviously-broken tag in Sentry
instead of silently sharing "local_development" with other nodes.
- BETA.md: drop the `echo > .env` step; instead set SENTRY_ENVIRONMENT
in config.py next to SENTRY_DSN. Verify step reads it back via
Config rather than the container env. Rollback no longer rm's .env.
The application code (start.py, apps/sentry.py) is unchanged: its
resolution order Config -> os.environ -> "local_development" already
makes Config authoritative now that Compose stops injecting the var.
Most EIDA nodes run v1.0.3 or v1.0.4, whose config.py predates some or all of the Sentry settings. The previous BETA.md only told operators to add SENTRY_ENVIRONMENT, which is correct for v1.0.4/v1.0.5 but leaves a v1.0.3 node missing SENTRY_DSN and SENTRY_TRACES_SAMPLE_RATE too. Rewrite step 2 to branch by source version: - from v1.0.5/v1.0.4: add SENTRY_ENVIRONMENT only - from v1.0.3 or earlier: add all three SENTRY_* lines (with the exact os.environ.get(...) snippets to paste into the try: block) - fresh install: all three already present, just replace the placeholder Confirmed by diffing config.py.sample keys across tags: the only config delta from v1.0.3 -> beta is the three Sentry keys; MONGODB_*, CACHE_*, and FDSNWS_STATION_URL are unchanged. Add a copy-paste diff one-liner so an operator can list exactly which keys their config.py is missing versus the shipped sample.
The README had drifted from the codebase: it referenced `pip install -r requirements.txt` (the project uses uv), a non-existent `tests/performance/` directory, `unittest discover` (now pytest), had duplicated `docker start` lines, and stale "after upgrading to v1.0.5" phrasing. Rewrite into a clean reference (418 -> 156 lines): - Architecture table of the four components + the deployment diagram. - Endpoints table (/query, /extent, /version, /application.wadl, /). - Full configuration reference: config.py keys plus the env-var-only settings (MAX_DATA_ROWS, MAX_STREAMS, FANOUT_*). - Deployment via docker-compose + Apache reverse-proxy snippet. - Materialized view: initial build, daily refresh, the recommended compound index. - Operations: the in-app APScheduler jobs (03:00 inventory, 06:00 view update with the 4-day reprocessing window), misfire behaviour. - Performance: workers, maxPoolSize, opt-in fan-out, thread limits. - Development with uv; tests with pytest. - Points beta testers to BETA.md at the top.
Reorganize so the common case is front and center and one-time/greenfield setup is fenced off: - Open with a one-paragraph what-it-is + the three containers. - "Run it" section first: clone, config.py, docker-compose up. That's the whole install for a node with a populated WFCatalog. - Config reference trimmed to the keys operators actually set. - "What runs daily" summarizes the scheduler (03:00 inventory, 06:00 view update) in three lines. - "First-time database setup" (view build + index) is now a clearly fenced section that says "skip if you already run ws-availability". - Reverse proxy demoted to a short "Serving publicly" note (it's node infra, not an app install step). - Tuning (workers, caps, fan-out) condensed to three bullets. 156 -> ~95 lines.
- Deployment now offers two paths: Option A build locally (docker-compose up -d --build), Option B pull pre-built GHCR images via a docker-compose.override.yml. Operators choose based on whether they want to build or just download. - Move "First-time database setup" (materialized view + index) to the last content section, fenced as fresh-DB-only, linked from Deployment. - Reverse-proxy guidance stays as "Serving publicly" (it was in the original README; just optional node infra, not an install step).
The one-liner assumed the reader already knew what fan-out was. Expand it into its own subsection that explains the problem (single cursor = sequential DB round-trips on long ranges), what fan-out does (concurrent day-aligned windows, merged into a byte-identical result), and the practical rules: off by default, only engages for ranges >= FANOUT_MIN_DAYS, applies to both endpoints, and the FANOUT_MAX_WORKERS / FANOUT_WINDOW_DAYS tunables with their connection-budget implication.
…ut vars - Configuration table gains a Default column with the real production defaults from config.py.sample (incl. SENTRY_TRACES_SAMPLE_RATE = 1.0, the cache TTLs, MongoDB defaults). Operators see what each key falls back to without opening the sample. - Remove the "Serving publicly" reverse-proxy section — it's node infrastructure, not part of installing the service. - Present the FANOUT_* tunables as a table with defaults and effects, matching the config-table style.
The old README documented daily materialized-view appension via a host cron (0 6 * * * mongosh ... main.js). That is obsolete in the in-app-scheduler model and was removed from the NOA deployment — do not reintroduce it. Keep the genuinely useful part the scheduler does NOT cover: ad-hoc back-processing of a specific historical range or stream subset via views/main.js parameters. Add two representative examples (a month, a network/station range) instead of the old wall of seven. Note explicitly that no host cron is needed.
Re-add the troubleshooting guidance that was dropped during the README simplification. De-stale it (was "after upgrading to v1.0.5" -> generic "after an upgrade") and include the config.py-vs-sample diff one-liner so operators can see exactly which keys they're missing. Three checks: logs, config completeness, database reachability.
test_restriction.py imported `from restriction import ...` and patched `restriction.redis`, which fails because the package is `apps.restriction` — the module was never collectible, so the file silently dropped out of every test run. Add the repo-root sys.path shim the other test modules use, switch the import to `apps.restriction`, and point both `@mock.patch` targets at `apps.restriction.redis`. The full suite now runs clean: `pytest tests/` -> 47 passed (was 38 with this file erroring out of collection).
The install steps reference the EIDA clone URL and the v1.1.0-beta.1 tag, which only exist there after the PR merges. During the beta the code lives on the contributor fork, so add a note at the top and switch the clone command to the fork URL. Reverts to the EIDA URL once merged.
The beta now ships as the beta/v1.1.0-beta.1 branch on EIDA/ws-availability rather than a fork tag, and without pre-built images (branch-only — no tag-triggered publish, avoids a beta PyPI publish). - BETA.md: clone from EIDA, `git checkout beta/v1.1.0-beta.1` (branch, not tag), drop the "clone the fork during beta" note. Build locally. - README: Option B (pull images) is now version-generic and notes images exist only for tagged releases; the branch beta uses Option A.
Forward-port the five master commits that landed after the beta branched: 8dcf9ae fix(EIDA#18): MONGODB_AUTH_SOURCE configurable (auth db can differ from data db; falls back to MONGODB_NAME) e41307d docs: MONGODB_AUTH_SOURCE row in the config table a449f47 test: backward-compat tests for MONGODB_AUTH_SOURCE f8fe83f fix(EIDA#51): reorder mongo_request fields to match the compound index {net, sta, loc, cha, ts, te} so the planner uses it 4d6c682 feat(db): auto-create the compound index on get_db_client() startup; First-time setup section in README drops the manual createIndex step Conflict resolution kept the beta's fan-out work intact: - get_db_client(): fan-out-aware maxPoolSize + new authSource fallback + the auto-index block. - mongo_request(): EIDA#51 field order propagated into _build_query (was a no-op before — the helper kept the old order). - settings.py: union of fan-out + Issue EIDA#60 fields with mongodb_auth_source. - tests/test_settings.py: kept master's two new tests on top of Issue EIDA#60. 49/49 tests pass.
…'s LMU bug)
docker-compose.yml passed MONGODB_HOST/PORT/USR/PWD/NAME and
FDSNWS_STATION_URL into the container with hardcoded defaults
(${MONGODB_HOST:-127.0.0.1}, etc.). apps/settings.py then dropped any
config.py value whose key was present in os.environ — but those keys
were ALWAYS present, because docker-compose set them unconditionally.
Net result: editing config.py did nothing. The container connected to
the docker-compose defaults instead.
NOA didn't notice because Mongo on NOA happens to be at 127.0.0.1
(host networking, same host). LMU has Mongo on a separate machine, so
the container hit "Connection refused" — Tobias reported this during
beta testing.
Fix:
* docker-compose.yml — remove MONGODB_*/FDSNWS_STATION_URL from the api
service environment. config.py is now the only place those settings
live. Side effect: the leaked NOA credentials (`wfrepouser`,
`2023wf`) and the NOA-internal station URL are gone from the file.
* apps/settings.py — extract the precedence logic into a testable
build_settings(legacy, env=None). Change the override check from
`if key in os.environ` to `if env.get(key)` (truthy), so an
empty-string env no longer clobbers config.py either.
Tests added:
* tests/test_settings.py::TestConfigPyVsEnvPrecedence — the four cases
(config wins when env unset; env wins when set; empty env doesn't
override; None legacy keeps Pydantic default).
* tests/test_docker_compose.py — static YAML guard so re-adding any of
the config.py-only keys to the api/cacher env block fails CI;
separately asserts the NOA credentials never come back.
* tests/test_wfcatalog_query.py::TestBuildQueryFieldOrder — locks in
the EIDA#51 field order in _build_query so a future refactor can't
silently break MongoDB index usage.
pyproject.toml adds pyyaml to dev deps for the static compose test.
58 passed, 0 skipped.
could make an actual difference if the desired value is set as an OS environment variable and if that value evaluates to False with bool() E.g. if trying to set an empty string or the number zero it might inadvertantly get overridden by the default value from config (cherry picked from commit 5ea288d)
Static check: regex over config.py.sample fails if anyone reintroduces the `os.environ.get(...) or DEFAULT` shape that PR EIDA#62 replaced. The `or` form falls back to DEFAULT when env is the empty string (and, in numeric contexts, when env is "0"), silently ignoring an operator who genuinely wants to set that value.
BETA.md updates from Tobias's beta review: - Step 2 install: `cp -n config.py.sample config.py` (no-clobber) replaces the two-line `[ -f config.py ] || cp ...` conditional. - Sentry example for v1.0.3 upgraders uses the PR EIDA#62 safer pattern `os.environ.get("X", default)` (we don't want operators copying the old leaky shape we just removed). - One-sentence callout at the top of Install explaining config.py is now the only place for MongoDB / FDSNWS-Station / Sentry settings — paired with the runtime change from commit 30c86b5. Regression tests in tests/test_docker_compose.py: - BETA.md must contain `cp -n config.py.sample config.py` and must NOT contain the conditional pattern. - BETA.md code snippets must NOT use `os.environ.get(...) or X`. 61/61 tests pass.
Tobias's beta review: "I do firmly believe the number of workers should be settable in the config, as opposed to having to patch the actual code — even if it's in a yaml." Adds gunicorn.conf.py which reads workers from Config.GUNICORN_WORKERS, defaulting to 1 when config.py isn't present (gunicorn must always be able to boot). docker-compose.yml's api command now invokes gunicorn with -c gunicorn.conf.py instead of a hardcoded --workers; same change on Dockerfile.api's CMD so the image works without compose too. config.py.sample gets GUNICORN_WORKERS = 1 in both RUNMODE blocks plus an int-coerced env override using the PR EIDA#62 safer pattern. README.md gains a config-table row and the Tuning section now points at config.py. Tests (tests/test_gunicorn_conf.py): - workers reads from Config.GUNICORN_WORKERS when present - falls back to 1 when GUNICORN_WORKERS missing - falls back to 1 when config.py itself is missing - coerces string env value to int - docker-compose api command uses -c gunicorn.conf.py, no hardcoded --workers - Dockerfile.api COPYs gunicorn.conf.py into the image 67/67 tests pass.
These files predate the move to uv. Both Dockerfiles install via `uv sync --frozen` from uv.lock; nothing in the repo references the text pins (verified: `grep -rE 'requirements-(api|cacher)\.txt'` returns no consumers). The pin set was last touched in 2023 (Flask 2.3.2, gunicorn 20.1, obspy 1.3.1, redis 4.4.4) — all years behind the current uv.lock and the source of duplicated Dependabot alerts that artificially doubled the open-alert count on EIDA/ws-availability. Added `tests/test_docker_compose.py::test_no_stale_requirements_files` so a stray re-commit fails fast. 68/68 tests pass.
Outcome of the Step-7 audit: every package the EIDA repo's Dependabot currently flags is *already* at a patched version in this branch's uv.lock (refreshed during the Python 3.13 upgrade earlier in the beta cycle). The 23 unique uv.lock alerts upstream scan EIDA/master, not this branch — they should resolve themselves when the beta merges. Ran `uv lock --upgrade` anyway to pick up newer patch releases: coverage 7.14.0 → 7.14.1 idna 3.16 → 3.17 redis 7.4.0 → 8.0.0 (major; ours is the redis-py client only) sentry-sdk 2.60.0 → 2.61.0 sqlalchemy 2.0.49 → 2.0.50 Verified after `uv sync`: - 68/68 tests pass with the upgraded venv - apps.redis_client.RedisClient imports and instantiates against redis 8.0 - No API surface changes our usage relies on Per-alert verification table (current vs first patched): idna 3.17 ≥ 3.15 ✓ urllib3 2.7.0 ≥ 2.7.0 ✓ pillow 12.2.0 ≥ 12.2.0 ✓ lxml 6.1.1 ≥ 6.1.0 ✓ python-dotenv 1.2.2 ≥ 1.2.2 ✓ pytest 9.0.3 ≥ 9.0.3 ✓ pygments 2.20.0 ≥ 2.20.0 ✓ requests 2.34.2 ≥ 2.33.0 ✓ werkzeug 3.1.8 ≥ 3.1.6 ✓ flask 3.1.3 ≥ 3.1.3 ✓ gunicorn 26.0.0 ≥ 22.0.0 ✓ pymongo 4.17.0 ≥ 4.6.3 ✓
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Beta release bundling the changes NOA has been running in production.
Deploy/upgrade instructions for node operators: see
BETA.md.What's included
In-app scheduler (replaces host cron)
apps/scheduler.py): inventory refresh at 03:00 UTC, availability-view update at 06:00 UTC, plus a run on startup. No host cron required.misfire_grace_time=300so a brief delay doesn't skip a run; the view update reprocesses the last 4 days so missed days self-heal.Issue #60 — memory-exhaustion protection
.limit(), broad-query rejection,MAX_DATA_ROWS/MAX_STREAMSguards, containermem_limit.Python 3.13 + dependency refresh
python:3.10-slim→python:3.13-slim. Bumps: Flask 2.3→3.1, gunicorn 20→26, pymongo 3.12→4.17, redis 4.4→7.4, obspy 1.3→1.5. No application-code changes required.Per-node Sentry environment tagging
SENTRY_ENVIRONMENTinconfig.py(e.g.noa_production) so each node's events are distinguishable. Sentry Cron monitoring on the scheduled jobs; GDPR-oriented PII scrubbing.Optional parallel MongoDB fan-out
FANOUT_ENABLED(default false) splits long-range queries into concurrent day-aligned cursors. When off, the request path is byte-identical to the single-cursor flow.Docs
BETA.mdinstall/upgrade guide.Validation
eida.gein.noa.grin production; both scheduled jobs firing on time; restriction inventory + materialized view current.uv run pytest tests/→ 47 passed.Notes
--workers 1 --timeout 600gunicorn config is unchanged from v1.0.x; separate hardening (more workers / shorter timeout) is tracked outside this PR.