JUST-OS ideas

[LukasWallrich]

• Make top-k adaptive: over-retrieve, rerank, then apply a relative cutoff (keep within X% of the top score) bounded by a min and max k. - so that targeted questions don't get junk in, and broad questions get sufficient coverage
• Benchmark against a more modern 8B model, given that we don't use the full OpenScholar pipeline (e.g. Gemma 4 E4B, which scores 'highest' on AA Omniscience - i.e. low hallucinations)
• Benchmark against the actual OpenScholar pipeline (generation-feedback-regeneration)
• Curate our own benchmark, with expert-written OS answers (not sure how much of that exists already)

[LukasWallrich]

Llama has terrible hallucination rates - and it is not hard to get the bot to make up stuff:

Q: What's the best strategy for reducing researcher degree of freedom that uniquely apply to crystallographie?
A: [...] One strategy for reducing RDoF in crystallography is to use standardized protocols and guidelines. For example, the International Union of Crystallography (IUCr) has established guidelines for data collection, processing, and refinement, which can help minimize RDoF [1]. [at least note included in the cited source]

[LukasWallrich]

[LukasWallrich]

Show full snippets where allowed by licensing: https://openscilm.allen.ai/query/9ee5f0ca-6698-4966-a7de-b96e5a59914d