Blog

LLM Box: How to Run Local and Cloud Models Together

“Local or cloud?” — in practice, that is not always the right question. A company may have tasks that require the maximum capabilities of a cloud model, alongside scenarios where data must not leave the corporate environment. Add inference costs, security requirements, workload, and the risk of vendor lock-in — and choosing a single model no longer looks like a rational approach.
A more practical solution is to set up model routing: handle sensitive and critical tasks locally, while using external models where it is acceptable.

Why Choosing One Model Isn’t Always the Right Approach

Enterprise AI rarely has just one use case. An employee might need to quickly summarize a public news article, process an internal document, extract a few fields from a PDF, or run an automation that becomes part of a business process.
Each of these use cases comes with different requirements.
For example, summarizing public content doesn’t necessarily require local inference. But if a request contains personal data, financial information, or other data that company policy prohibits from being sent to external services, the requirements change: the data needs to stay within the company’s environment or be anonymized before it leaves.
That’s why routing everything through a single local model isn’t a universal solution either. Running a local model requires infrastructure to be provisioned and maintained, and more demanding workloads can require substantial compute capacity. Sending everything to a cloud model comes with its own trade-offs: data leaves the company’s environment, you depend on the provider’s terms and availability, and you are constrained by the capabilities and limits of the external infrastructure.
There’s another issue: a cloud model may perform well during a pilot but behave differently once the solution goes into production. Traffic increases, access conditions change, or the model itself is updated — and a solution built around the behavior of an external service may need to be reworked.
That may be acceptable for a one-off task. But for a critical internal process, you need consistent and reliable operation.
So instead of asking “local or cloud?”, a better question is: which model can be trusted with a given task, and what rules determine that choice?
This is where a hybrid LLM setup comes in: local and external models run within the same environment, while model selection becomes part of the architecture.

How Model Routing Works

The simplest approach is to let users choose. The interface can provide access to multiple models at once — both local and cloud-based — so employees can select the one that fits the task at hand.
There’s no need to expose every available model to every user. Access can be controlled based on the employee’s role and use case. For example, an employee who only handles document processing might have access to a smaller local model, while someone who regularly works on complex analytical tasks can be given access to more capable models.
In LLM Box, this approach is implemented through a unified interface for working with models. But routing can also be configured at the automation level.
In the agent builder, you can assign a specific model to each step of a workflow. For example, an agent receives a PDF, extracts information from it, analyzes the data, and generates a report. There’s no need to use the same expensive model throughout the entire workflow. A smaller local model may be sufficient for text extraction or pulling out structured fields, while a more capable model can handle the complex analysis.
This has two benefits: compute resources are used more efficiently, and inference costs can be managed at the level of individual operations.
What About Automatic Model Routing?
Yes, but it’s important to distinguish proper model routing from a simple rule like “look at the first request and pick a model.”
In an ongoing conversation, switching models mid-session can affect the quality of the result. The new model may have different capabilities or may not handle the accumulated context as effectively.
For agentic workflows, the approach is different. A more capable model can act as a planner: it takes a complex task, breaks it down into subtasks, and delegates simpler operations to smaller models. For example, one model can determine the sequence of actions, while smaller models extract data, classify documents, or handle other well-defined tasks with predictable outputs.
This approach lets you use a more powerful model where its reasoning capabilities are actually needed, while handing simpler tasks off to more cost-efficient models.
The result is not “one model for everything,” but an architecture built around models with different capabilities.

What Happens to Sensitive Data

Routing should take into account not only the complexity of the task, but also the type of data involved.
If the data must not leave the corporate environment, the request is routed to a local model. Another option is to apply specific rules before sending the request to an external service. For example, LLM Box uses Guardrails before a request is sent to an external service. They can mask personal data, filter out prohibited topics, and enforce the required request format.
This is a key architectural principle: data protection happens before the data leaves the corporate environment, not after an external service has already received the request.
It is also important to know what happened to every request sent to a model. Enterprise deployments need tracing: which model handled the request, what result the system returned, and how the request moved through the workflow. This is useful not only for developers troubleshooting the system, but also for security teams — especially when AI becomes part of business processes.

How to Choose the Right Model for the Task

For enterprise use, model selection shouldn’t come down to “bigger is better.” Larger models have an advantage on complex and ambiguous tasks, but not every use case requires that level of capability.
In practice, there are at least three key factors to consider.
First: model size and response time. The more capable the model, the more compute it typically requires. For an interactive chat where users expect an immediate response, latency can be critical. For background document processing, it may be less important.
Second: performance in the languages you need. This becomes especially important when a company operates in languages other than English. For widely used languages, there are more models to choose from, and local models often have a wider range of fine-tuned variants. For less widely supported languages, the choice is more limited.
For example, the range of local models available for Uzbek is still narrower than for English. Some open models are trained primarily on English and then fine-tuned for other languages. That’s why model selection should consider not only a model’s overall benchmark performance, but also how well it actually performs in the language your company uses and for the specific task at hand.
The landscape is changing, though. New open models that perform well in Uzbek are emerging, while existing models are being fine-tuned for local use cases. For example, in testing models for Uzbek speech transcription, the open-source MegaM model delivered strong results. For companies working with Uzbek, this means the choice is no longer simply “cloud model or nothing” — local options are gradually becoming a practical alternative.
Third: inference costs. Cloud models don’t require a company to purchase its own compute infrastructure, making them convenient for getting started quickly: you pay for usage and get access to a powerful model. But as usage becomes more frequent and workloads grow, cost becomes a significant consideration.
Local inference has a different cost structure: requests aren’t billed per token by an external provider. The main costs are the infrastructure itself, maintenance, and electricity. As a result, for a large and predictable volume of requests, running a local model can make economic sense.

When a Smaller Model Is Enough

A common architectural mistake is using a large general-purpose model for tasks that are well-defined and have a limited range of possible outputs.
Document classification, field extraction, PDF-to-text conversion, document chunking, request filtering, and transcription are just some of the tasks that can often be handled by smaller, specialized models.
Some are compact enough that they don’t require a powerful GPU. They can be deployed locally and used to handle high workloads.
A good example is automating the processing of technical specifications. In one such workflow, large documents pass through a series of stages: a PDF is converted to text, the document is split into chunks, the data is processed, and the results are then assembled and passed to the next stage before a structured report is generated.
This workflow didn’t require the most powerful cloud model. A mid-range local model running on a single GPU was enough. At the same time, the automation increased the department’s throughput several times over.
That’s why choosing an LLM for business should start not with model rankings, but with breaking down the process: which operations actually require complex reasoning, and which can be handled by a small, specialized model?

A Unified API and Freedom from Vendor Lock-In

A hybrid architecture only makes sense if the application doesn’t need to know where each model is running.
Consider an enterprise service such as an internal help desk. The company wants to add AI capabilities to it. If the application is integrated directly with a specific provider, the architecture gradually becomes dependent on that provider’s API, request format, and terms of use.
Instead, you can place a unified access layer between the application and the models.
In LLM Box, the application connects to Gorgona via API, while the platform handles the rest of the request flow. This makes it possible to use a local or cloud model, add Guardrails, trace requests, connect knowledge bases, and use other capabilities without embedding all of that logic directly into the enterprise application.
The result is a single integration point: the application only needs to know one API, while model selection becomes a matter of configuration.
This is important not just for development convenience. It also reduces vendor lock-in.
If a provider raises its prices, changes its terms of use, limits access to a model, or the company simply finds a better option, the application doesn’t need to be completely rewritten. You change the model behind the unified interface, rather than reworking the architecture of the entire system.
For a business, this is a matter of cost and control. The more deeply a specific model is embedded in a critical process, the more expensive it becomes to replace. An abstraction layer makes it possible to avoid turning the choice of an AI provider into an irreversible architectural decision.
Another factor is cost predictability. With cloud models, costs depend on usage and the provider’s pricing. Local infrastructure requires upfront investment, but the compute resources remain under the company’s control. As workloads grow, the company can plan to expand its own infrastructure instead of relying solely on the capacity and limitations of an external service.

What You Need to Make It Work

A hybrid setup doesn’t require choosing one “ideal” model upfront. The first step is to define the tasks and their requirements: what data can be sent outside the corporate environment, which use cases are business-critical, what response time is required, how predictable the workload will be, and what level of quality is needed.
The tasks can then be distributed across different models.
A cloud model may be a good fit for quick experiments and non-critical use cases. A local model can handle data that must stay within the corporate environment. Simple, repetitive operations can be handled by a small specialized model, while more complex reasoning can be assigned to a more capable one. All of them can run within the same infrastructure through a unified management layer.
LLM Box is a Gorgona platform module that combines local models with the ability to connect external ones. It provides a unified interface for managing models, agents, and access controls, making it possible to build this kind of hybrid architecture.
The platform offers three licensing tiers — S, M, and L — designed for different deployment scales. They differ not by the number of “messages per month,” but by the set of available modules. The basic tier provides access to models through a unified interface and supports Guardrails. Higher tiers add capabilities for building agents, working with knowledge bases, evaluating model quality, and connecting external tools via MCP.
A pilot can be deployed within 45 days on Gorgona hardware, with company data connected through a secure channel. This makes it possible to test the architecture on real business tasks rather than in theory: compare models, evaluate quality, workload, and costs, and determine which use cases make more sense to keep in the cloud and which should be moved to local models.
As a result, the question “Which AI model should a business choose?” becomes a more practical one: which model should be used for which task, and according to what rules?
This approach means you don’t have to make a one-time choice between local and cloud AI. Instead, you can manage both as part of a single infrastructure.
Want to see how a hybrid LLM setup would work for your business? Request a Gorgona demo — we’ll show you how to combine local and cloud models within the same environment and design an architecture around your requirements.

Related articles

2026-09-27 12:00 Articles