Seven open-source repos to build a production stack around GPT-6 Astra
The author lists seven infrastructure layers, from inference serving and routing to memory, guardrails and observability, to put in place before running Astra in production.
GPT 6 Astra needs a f...cking operating system around it 7 GitHub repos cover serving, routing, memory, guardrails and proof 01 vLLM ▸ https://github.com/vllm-project/vllm → serves high throughput inference for real traffic 02 llama.cpp ▸ https://github.com/ggerganov/llama.cpp → keeps a local fallback alive when the cloud path dies 03 RouteLLM ▸ https://github.com/lm-sys/RouteLLM → sends easy prompts to cheaper models automatically 04 Portkey Gateway ▸ https://github.com/Portkey-AI/gateway → one gateway with retries, keys and guardrails 05 Zep ▸ https://github.com/getzep/zep → assembles long running agent context without stuffing the prompt 06 LLM Guard ▸ https://github.com/protectai/llm-guard → scans inputs and outputs before anything leaves the system 07 Opik ▸ https://github.com/comet-ml/opik → traces production runs and catches quality drops early A stronger model does not replace an operating system Astra should decide, route, remember and ship through dedicated layers Swap the model tomorrow and the company stack still stands
This post quotes another post: x.com/marlenuii/status/2109673372091126158