Ollama API: Troubleshooting Common Mistakes

Understanding the 100k Context Window

The first sentence of every section answers the question in its heading.

A 100,000 token context window sounds massive, but it divides equally between input prompts and output completions. When you send a long system prompt alongside a lengthy conversation history, you consume that budget faster than you might expect. Many developers assume the limit applies only to the input, which leads to truncated responses or silent failures when the model hits its ceiling.

Consider a scenario where you pass a 50,000-token document as context. You only have 50,000 tokens left for the actual generation. If you do not manage this balance carefully, the model will cut off mid-sentence. Always monitor the total token count sent in each request to avoid unexpected truncation.

Max Tokens vs. Default Output Length

Every section must answer the question in its heading.

Unless you specify a limit, most APIs default to a modest output length, often around 2,048 tokens. This default is useful for short queries but disastrous for code generation or long-form writing. If you need the model to produce a full script or a detailed analysis, you must explicitly set the max_tokens parameter.

Setting this value too high can also impact latency and cost. You are paying for every token generated, whether useful or not. A pragmatic approach is to set a reasonable upper bound that matches your use case, rather than leaving it to the default or an arbitrary infinity.

Handling Streaming SSE Chunks Correctly

Streaming requires careful handling of Server-Sent Events to avoid data loss.

When you enable streaming, the API returns a series of small chunks rather than a single complete response. Each chunk contains a delta of text, but crucial details like token usage statistics are often only present in the final chunk. If you stop parsing too early, you miss the usage data needed for billing or rate-limit tracking.

Additionally, streaming introduces complexity in error handling. A network interruption might leave you with partial text. You need a robust parser that can reconstruct the full message from these fragments. This is especially true when integrating with UI components that render text in real-time.

Function Calling Tool Definition Errors

Function calling fails when tool schemas do not match the model's expectations.

The model expects a specific JSON structure for tool definitions. If you omit required fields or use incorrect types, the model may ignore the tool entirely or return a malformed response. This is a common pain point for developers building agents that need to interact with external APIs.

Always validate your tool schema against the API documentation before sending requests. Even small deviations, like using an integer instead of a string for a parameter, can break the function call. Debugging this requires inspecting the raw JSON response to see if the model attempted to call the tool at all.

JSON Mode Response Format Issues

JSON mode forces the model to output valid JSON, but it can still hallucinate.

When you set response_format to json_object, the model is constrained to output only JSON. However, this does not guarantee the content is semantically correct. The model might produce valid JSON syntax but with wrong keys or missing fields. You still need a parser to validate the structure before using the data.

Furthermore, JSON mode can sometimes reduce the model's creativity or flexibility. If your prompt is too complex, the model might struggle to fit the narrative into a rigid JSON structure. Use this mode primarily for structured data extraction or function arguments where consistency is more important than nuance.

Rate Limiting and Concurrent Requests

Rate limits are enforced per API key, not per user or IP address.

Most APIs impose a limit on requests per minute and concurrent connections. If you exceed these limits, you will receive a 429 Too Many Requests error. This is critical for applications that spawn multiple threads or processes to handle user requests. A single key can only handle a limited number of parallel requests.

To manage this, implement exponential backoff in your client code. When you receive a 429 error, wait for a short period before retrying. This prevents overwhelming the server and ensures a smoother user experience. Always check the response headers for retry-after values to optimize your wait times.

API Key Replacement Behavior

Regenerating an API key invalidates the previous one immediately.

Some platforms allow only one active key per account. If you generate a new key, the old one stops working instantly. This can cause outages if you have multiple services or scripts using the old key. Always update all instances of your key before rotating it.

This behavior is designed for security, ensuring that a leaked key cannot be used indefinitely. However, it requires careful management in distributed systems. Use environment variables or secret managers to store your key, making updates easier and less error-prone.

Request Body Size Limits

Large payloads can exceed the 8 MB request body limit.

When sending long documents or images encoded as base64, the request body can grow quickly. If you exceed the size limit, the server will reject the request with a 413 Payload Too Large error. This is a hard limit that cannot be increased by upgrading your plan in many cases.

To avoid this, compress your input or split large documents into smaller chunks. You can also use a more efficient encoding method for images. Always monitor the size of your requests, especially when dealing with media-rich content.