Support free-threaded (nogil) CPython — blocked on lancedb free-threaded wheels

[HumanBean17]

Context

Follow-up to #447. A consumer on free-threaded CPython 3.14 (cp314t) hit:

Cannot install java-codebase-rag… no matching distribution for lancedb

pip install backtracks through every release and fails on all of them.

Root cause

java-codebase-rag itself is pure Python (py3-none-any.whl) and runs fine on free-threaded builds. The vector stack on free-threaded, component by component:

dep	cp313t/cp314t wheel?
cocoindex	✅ yes
torch (via sentence-transformers)	✅ yes (2.13+)
sentence-transformers	✅ pure Python
lancedb	❌ none, in any version — sole blocker

lancedb is load-bearing (it is the vector store — search_lancedb.py + cocoindex[lancedb]), so there's no drop-in swap.

Why we can't fix it today

1. Can't loosen the pin — no lancedb version ships a free-threaded wheel.
2. Can't marker-gate it — no pip-supported marker detects free-threaded Python. PEP 780's sys_abi_features would express it, but it's still Draft and unimplemented in pip/packaging (unknown marker → hard parse failure for everyone).
3. Can't swap lancedb.

Decision (2026-07-14)

Doc-only stopgap: documented the limitation in the README install section and the pyproject.toml vector-stack comment (branch docs/free-threaded-cpython-note), and advised consumers to use standard CPython (3.11–3.14, not the t build).

We deliberately did not make the vector stack an optional [vector] extra: it would change the headline install command for every standard-Python user, and still wouldn't give free-threaded users vectors (lancedb blocks it) — only graph-only.

Trigger to act

Either of:

• lancedb ships cp313t/cp314t wheels → tracked upstream at Publish free-threaded Python wheels lancedb/lancedb#3453 (open) and CI work Split Python test CI build/test steps and add free-threaded 3.14t build lancedb/lancedb#3456
• PEP 780 sys_abi_features lands in pip/packaging → https://peps.python.org/pep-0780/ (Draft)

When unblocked

• Lift the doc note in README.md (Install section) and the pyproject.toml comment.
• Verify pip install java-codebase-rag resolves on python3.14t (Apple Silicon arm64 is the reproduction path from pip install --upgrade error #447).
• If lancedb ships wheels but PEP 780 still isn't out, no marker is needed — it'll just work.

/kind enhancement
/area dependencies