LLM integration that holds up in production.

• Choose an LLM integration when a feature must read or write language.

  Classifying messages, extracting fields from documents, drafting text and answering from content are what a language model adds to an application.

• Choose an ordinary integration when the data is already structured.

  When fields already arrive in a known format, a rule or an API integration is cheaper, faster and fully predictable, and no model is needed.

• Choose a chatbot integration when the feature is a conversation.

  A public assistant needs a scope, approved content, a handoff and conversation logging, which the AI chatbot integration service covers.

• Choose an agent only when several steps and tools are genuinely needed.

  An agent adds planning, tool access and more ways to fail. When one model call or a fixed workflow does the job, that simpler design is the better one.

Tool calling lets a language model ask the application to do something, such as look up an order or create a ticket, by returning a structured request that names a tool and its arguments. The application checks the request, performs the action if it is allowed and returns the result to the model.

The mechanism is similar across several providers. Tool calling, also known as function calling or tool use, gives a model a way to interface with external systems and access data outside its training data. For tools the application defines, the model returns a structured call that the application executes, and some providers also offer server tools that run on the infrastructure of the provider instead.

Whichever provider is used, each tool is described precisely, receives only the arguments it needs and runs with the permissions of the user it acts for. A model that asks for the wrong thing is refused by the application rather than obeyed, and every call is logged.

The AI service layer is server code between the feature and the model provider. It holds the API keys, builds the prompt and context, limits input size and request rate, calls the model, validates the response, retries or falls back when the call fails and logs what happened, so none of that is repeated in each feature.

Rate limits are part of the provider contract. A request over a temporary rate limit typically returns an HTTP 429 error, which can include a Retry-After header indicating how long to wait. That header is treated as a minimum wait, exponential backoff with jitter is used when the header is missing, and quota and billing errors are not retried at all.

Long answers can be streamed. Streaming lets an application start showing or processing the beginning of a response while the model is still generating it, although streamed content is harder to moderate. A streamed response is commonly delivered incrementally using server-sent events. Every call also has a timeout, so a slow provider cannot hold a page open.

The contents of a system prompt should not be treated as a secret or used as a security control, and credentials should not be placed in the prompt at all. Keys live in server configuration, and anything a user must not see is enforced in code rather than in the prompt.

Prompt injection is handled by assuming it will be attempted. Instructions and untrusted content, such as documents, emails and web pages, are kept apart and screened; the model gets only the tools and data the task needs; its output is validated before anything acts on it; and actions with real consequences need a person to approve them.

Prompt injection is the name for prompts that alter the behavior or output of a model in unintended ways, whether a user types them directly into the feature or they are carried in indirectly by external content such as websites or files.

Defenses are layered: screening user input, filtering it for known injection patterns, limiting the access the model has to sensitive data and actions, screening tool outputs before the model acts on them, and red-teaming the feature. The model is treated as any other user, with a zero-trust approach, and its responses are validated before they reach backend functions.

Running costs are kept under control by measuring them per request during development, sending only the context a task needs, choosing the smallest model that passes the evaluation set, caching repeated work, batching tasks that can wait, and setting limits and alerts on usage, so an unexpected rise is noticed early.

Several providers offer both prompt caching and batch processing. Prompt caching reuses work when requests share the same prompt prefix, so a long, stable block of instructions is placed first and reused. Batch APIs handle asynchronous requests whose results are returned later rather than at once, suited to work that does not need an immediate response, such as classifying a backlog of tickets.

Input is validated strictly, so no request exceeds reasonable size limits, and rate limiting with user quotas caps the number of requests that a single source can make in a given period. On a public feature, those limits protect the budget as much as the service.

Prices and discounts change, so none is quoted here. The cost per request is measured on the chosen model during testing and recorded with the design.

A model is chosen by running the same evaluation set, real examples with expected results, against candidate models and comparing their results, latency and measured cost per request. The cheapest and fastest model that passes is preferred, and because the set is kept, a newer model can be compared the same way later.

Model names, prices and context limits change often, so they are checked on the day a decision is made and recorded in the documentation rather than fixed in the design. Where data must not be used to improve a provider product, the plan and terms of each candidate are part of the comparison.

The evaluation set also protects the feature after launch. A prompt change or a model update runs against it first, and a drop in results stops the change before users see it.

• LLM integrations
• AI model API integrations
• Model selection and fallback
• Prompt design and evaluation
• Cost, latency and rate-limit controls

Every model call in a feature takes the same path, so limits, logging and validation are applied in one place.

• Output the application can check

  Validated, structured results mean the feature acts on data it has checked, not on free text.

• Failures that degrade gracefully

  When the provider is slow or unavailable, the feature falls back to a defined behavior instead of breaking.

• Costs measured before launch

  The cost per request is known from testing, so a budget can be set against real volumes.

• Room to change models

  Prompts, schemas and provider calls live in one layer, so another model can be evaluated without rewriting the feature.

• Support triage

  Incoming tickets are classified by topic and urgency into a validated schema, and the helpdesk routes them by the result.

• Document extraction

  Fields are extracted from uploaded forms into a schema, checked against rules and sent to review when a check fails.

• In-product writing help

  A SaaS application drafts a reply or a summary that the user edits before anything is sent.

• The provider and model are chosen for each feature against an evaluation set, and the choice is documented.
• Model output is treated as untrusted input: validated against a schema, and encoded or parameterized before it reaches a page or a database.
• Tools run with the minimum permissions the feature needs, and actions that change records or spend money can require approval.
• Content from documents, emails and web pages is kept apart from instructions and screened before the model sees it.
• Rate limits are handled by honoring the wait the provider signals, backing off with jitter, capping retries and falling back.
• Context is kept to what each task needs, and caching or batch processing is used where the task allows it.
• API keys stay on the server, scoped and rotated, and never appear in browser code, prompts or logs.