AI Glossary · LLM Integration
Streaming Responses
Returning tokens as they are generated, drastically improving perceived latency for chat and agent-assist experiences.
Related terms in LLM Integration
- API Gateway (LLM)
- A managed proxy that routes model calls, enforces quotas, redacts PII, logs prompts, and applies policy across multiple LLM providers.
- Batch Inference
- Running many model calls asynchronously at lower cost — used for backfills, document processing and offline analytics.
- Caching (Prompt/Response)
- Storing repeat prompts or key/value tensors to cut latency and cost; supported natively by most frontier providers.
- Context Assembly
- The pipeline that gathers system prompt, retrieved chunks, tool schemas and history into a single request within the context window.
- Fallback Routing
- Failing over to a secondary model or provider when the primary is slow, rate-limited or degraded — table stakes for production.
- Model Router
- A component that picks the right model per request based on cost, latency, capability or policy — often small model first, large model on fallback.