Launching SwiftInference: Bringing AI Inference Closer to Every User
Kendall Ananyi
“Originally published on the SwiftInference blog” https://www.swiftinference.ai/blog/launching-swiftinference-bringing-ai-inference-closer-to-every-user
The next generation of AI will not be built only on better models. It will also require a better way to deliver those models, closer to users, with predictable latency, resilient infrastructure and far less operational complexity.
By Kendall Ananyi, Founder and CEO of SwiftInference
Today we are launching SwiftInference, a distributed edge AI inference platform built to run AI models closer to the people and machines using them.
We are starting with large language models, speech to text, text to speech, computer vision and image generation. Over time the same infrastructure can support the broader real time computing stack required by agents, robotics, augmented reality, autonomous systems and other applications that cannot afford an unpredictable round trip to a distant cloud region.
The simplest way to understand SwiftInference is as an AI inference network. Customers choose or upload a model, select where they want it deployed, and call it through a familiar API. SwiftInference handles the GPU infrastructure, model deployment, warm up, routing, monitoring, security and usage tracking behind the scenes.
| Training builds the model. Inference is the product experience. Every time a user speaks to a voice agent, asks an assistant a question, analyzes an image or triggers an autonomous action, an inference request is being made. For real time products, the time and the consistency of that request directly shape how the product feels. |
AI has a distance problem and a contention problem
The cloud made it dramatically easier to build software. It gave startups and enterprises access to infrastructure without constructing data centers of their own. AI would not have advanced this quickly without it.
But the centralized model assumes that most requests can travel to a small number of large regions and return without the network becoming part of the user experience. That assumption begins to break when software becomes conversational, continuous and physical.
A voice agent cannot pause awkwardly after every sentence. A safety camera cannot wait while video travels across a continent. A robot cannot send every small decision to a distant region. An always on agent cannot depend on a single provider, a single geography, or an unpredictable queue before it can act.
There are two separate forces at work here and they are usually discussed as though they are one. The first is distance. Photons in fiber and the routers between you and the region you are calling impose a floor that no amount of model optimization removes. The second is contention. When you call a shared endpoint you are queued behind everyone else calling it, and your experience changes depending on what time of day it is and who else showed up. Distance is fixed by where the compute sits. Contention is fixed by who is allowed to use it. Neither is a software problem.
For these applications, average latency is not enough. Tail latency matters. Jitter matters. The slowest requests are the ones users remember, and they are the requests that break natural turn taking, interrupt an automated workflow, or cause an application to miss its operating window.
The insight came from building internet infrastructure
Before SwiftInference I built Tizeti, an internet service provider operating across multiple emerging markets, which became the largest ISP in West Africa by active users. Like many companies we initially placed core software in AWS. When we started, reliable local data center infrastructure was limited, so centralizing the systems was a rational decision. Over the years we built and operated real physical network infrastructure, working with Equinix and MainOne on interconnection and connectivity and with Microsoft, USAID and Meta on connectivity programs across the markets we served.
As local infrastructure matured we moved core systems out of AWS and into an Equinix data center much closer to the people using them. The same software felt more responsive. Nothing in the application had changed. Only the network position had. Distributing the systems across the countries we operated in also reduced the impact of an outage in any one of them.
The second half of the lesson arrived while I was doing that migration. I was leaning on AI models heavily to plan and execute the move, and I noticed two things that had nothing to do with the network I was rebuilding. Responses got noticeably slower at certain times of day, which is contention, not compute. And as the work grew, context kept getting lost when compaction kicked in, so I was re explaining things the model had already been told.
Those are the two ceilings. Distance and contention. You do not get past either of them by writing better software, and the industry keeps trying to. That is why I started this company.
The lesson was never that the cloud is going away. It is that the best architecture is usually a combination. Centralized cloud for control, deep reasoning, training and large elastic workloads. Distributed infrastructure for the real time loop that has to happen close to the user.
What our benchmark showed
We tested that thesis in a controlled study across 1,000 inference trials under mobile realistic Wi-Fi conditions. The SwiftInference edge deployment achieved a P90 latency of approximately 125 milliseconds against 194 milliseconds for the cloud baseline, which is 36 percent faster at the 90th percentile. Latency variance was 3.7 times lower, even though the edge hardware had roughly one third of the raw compute performance of the cloud GPU used in the comparison.
The important result was not simply that one endpoint was faster. It was that proximity and consistency could overcome a large raw compute disadvantage. For real time applications, a slightly slower GPU that is nearby and available can deliver a better experience than a more powerful GPU that is distant, queued, or reached through an unpredictable network path.
| 1,000 controlled trials P90 latency: 125ms at the edge against 194ms in the cloud. Latency variance: 27ms against 100ms. The edge system delivered 36 percent faster P90 latency and 3.7 times lower variance despite using materially slower raw compute hardware. Full methodology and results: Consistent Low-Latency AI Inference at the Network Edge, SSRN Abstract ID 6352298. |
Alongside the paper we run a public comparison you can try yourself. It puts a SwiftInference edge endpoint next to the major frontier providers and measures time to first token on the same prompt. Across 107 demo sessions the SwiftInference endpoint returned a median time to first token of approximately 460 milliseconds, against roughly 2,426 milliseconds and 1,797 milliseconds for two of the large hosted providers. Those are demo session results under demo conditions, not a controlled benchmark, and we present them that way. The controlled work is the paper. The demo exists so that the difference is something you feel rather than something you read.
What SwiftInference is building
Content delivery networks made the web faster by moving frequently used content closer to users. SwiftInference applies the same architectural principle to live model execution. Instead of sending every inference request to a distant centralized region, we place managed GPU capacity at telecom tower sites, carrier neutral interconnection facilities and other edge locations close to demand.
From the customer’s perspective the platform is designed to stay simple:
- Choose a supported model or upload your own.
- Select the cities, regions or GPU slots where it should run.
- Deploy through the control plane without manually configuring each server.
- Call an OpenAI compatible API and receive streamed results.
- Monitor health, GPU utilization, throughput and P50, P95, P99 and time to first token latency.
- Attribute usage and cost by customer, model, location and GPU slot.
- Route to any external provider through the same endpoint using your own stored credentials, and have that traffic metered next to your edge traffic.
- Build voice and text agents on the platform, with tools, MCP servers, a knowledge base and SIP telephony attached.
- Test any modality in a browser playground that reports time to first byte, tokens per second and total time on every request.
Behind that interface SwiftInference manages model distribution, hardware compatibility, container lifecycle, warm up, health checks, failover, routing, observability and billing. A customer should not need to SSH into a tower node, tune every GPU independently, or rebuild an operations team every time a new location is added.
One endpoint for every provider
The fastest way to start using SwiftInference does not require moving a workload at all. The Inference Gateway is a single OpenAI compatible endpoint that reaches every model you already use. You name the model as a provider slug and we route the request with your own stored credential: openai/gpt-4o, anthropic/claude-3-5-sonnet, gemini/gemini-2.0-flash, si/swift-llm-8b for a Swift hosted edge model, or custom/alias/model for an endpoint you run yourself on premises.
Change one base URL and keep everything else. Your keys remain yours, your current provider keeps serving your traffic, and every request through the gateway is metered into the same analytics view, with P50, P95 and P99 broken out by model and by location and exportable as CSV.
That last part is the point, and it is why we built the gateway before we needed it. Almost no team has ever seen their frontier provider and an edge deployment measured on the same axis using their own traffic rather than a vendor benchmark. The gateway makes that comparison a side effect of normal operation. We would much rather a customer moved a workload because their own dashboard told them to than because we did.
It also commits us to the honest outcome. If the gateway shows that the bottleneck is the model rather than the path, the customer sees that before we do, and the right answer is to leave the workload where it is.
The data path matters
Many products call themselves edge platforms while sending the actual inference request through a centralized gateway. That preserves operational control, but it reintroduces the exact network distance the edge was supposed to remove. It is worth checking, because from the outside the two architectures look identical.
SwiftInference separates the control plane from the data plane. The control plane coordinates deployment, model availability, health, usage and billing, and it reaches nodes over an outbound only encrypted tunnel. The data plane is separate. Where carrier or private network connectivity is available, the inference request travels directly through an on net local breakout path to the selected edge node without touching the public internet. A centralized proxy remains available as a fallback, not as the required steady state path.
That distinction is the whole company. We have a signed integration with a tier one United States carrier for local breakout, which is the mechanism that makes on net routing real rather than architectural. Requests from that carrier’s network can reach our compute without leaving it.
This design keeps the management layer secure and consistent while allowing the latency sensitive request and the streamed response to stay local whenever the network permits it.
Where this wins, and where it does not
Edge inference is not a universal upgrade and we would rather say so than have a customer discover it. The honest version:
| Workload | Where it should run | Why |
| Voice agents and telephony | Edge | Turn taking is destroyed by tail latency. Audio is generated locally and consumed locally. |
| Real time vision and physical AI | Edge | Data is generated at the camera or the machine. Moving it is the expensive part. |
| Interactive agents with tool calls | Edge | Every hop compounds. A single request becomes many round trips. |
| Retrieval heavy workloads | Cloud | If the corpus lives centrally, the model should sit next to the corpus. |
| Training and fine tuning | Cloud | Elastic, batch, latency insensitive. This is what hyperscale is for. |
| Deep reasoning on large frontier models | Cloud | Model size, not distance, dominates the response time. |
| Regulated data with locality requirements | Edge | Placement is the requirement, not a performance optimization. |
Why customers will use it
1. Faster and more predictable real time experiences
Voice AI, interactive agents, live vision, gaming, robotics and augmented reality are sensitive to both latency and variance. Running models closer to users reduces the distance in the request path and gives customers the option to reserve capacity instead of competing with unknown workloads on a shared endpoint. That second part addresses contention, which is the half of the problem nobody sells against.
2. Resilience across locations and providers
A distributed network can route around an unavailable node or region. Customers can place models in multiple metros, use cloud GPUs as equivalent fallback nodes, and reduce dependence on a single centralized inference provider.
3. Data locality
Some workloads require data to remain within a country, an enterprise network or a regulated boundary. SwiftInference can deploy models close to the source of the data and supports data locality requirements relevant to sectors such as healthcare and financial services, without requiring the customer to build and operate the complete infrastructure stack themselves.
4. Infrastructure without the operational burden
Buying a workstation, a DGX class system or a GPU server is only the beginning. Teams still need to install runtimes, secure the device, distribute models, patch software, monitor health, replace failed hardware, manage keys and upgrade as models and accelerators change. Every one of those is a recurring cost against engineering time that was supposed to go into the product. SwiftInference turns them into a managed platform.
5. More control over inference economics
Customers can use SwiftInference capacity, reserve dedicated GPU slots, or bring eligible hardware they already own. This creates an alternative to relying exclusively on per token public endpoints and gives AI companies a path to own more of their inference stack without operating every site themselves.
Voice is where we are starting
Every inference provider says they support voice AI. We built the telephony layer. SwiftSIP is a SIP back to back user agent written in Go with no external dependencies, and it runs the entire voice loop on the node: RTP audio in, voice activity detection, speech to text, model inference, text to speech, and RTP audio back out. Nothing in that loop leaves the site.
This matters because voice is the workload where the argument is easiest to verify. A conversation either feels natural or it does not, and every participant in the call can tell within one exchange. It also compounds. A voice agent sends a long fixed system prompt followed by many short turns, so the same context is processed repeatedly, and keeping that context resident next to the GPU rather than rebuilding it every turn is worth more at the edge than anywhere else.
That argument is now something a customer builds rather than reads. The platform ships an agent layer: choose a template, write the system prompt and the first message, pick the voice and the speech to text model and language, bind the agent to one of your model deployments, attach tools, MCP servers and a knowledge base, and talk to it in the browser before a phone number is ever pointed at it. Templates ship for voice agents, legal and financial assistants, a blank custom text agent, and bring your own container for teams that would rather push a Docker image and a manifest through the CLI.
Two of the controls deserve to be called out, because they are the ones nobody else exposes. Interruption sensitivity governs how readily the agent yields when a caller speaks over it. Minimum and maximum reply delay set the window the agent waits before it answers. Those are turn taking parameters, and they are only worth tuning if the rest of the loop is close enough that there is budget left to tune. On a distant shared endpoint the network has already spent it.
Around the agent sit SIP trunks for inbound and outbound calling, and audiences, which are reusable contact lists an agent can dial as a campaign. The point of building the telephony ourselves was never the telephony. It was that owning the whole loop is the only way those parameters mean anything.
Why now
Several shifts have made this the right time to build a distributed inference network:
- AI is moving from occasional prompts to continuous voice, visual, agentic and machine to machine interaction.
- Inference demand is growing across more countries while advanced GPU capacity remains concentrated in a small number of regions. Most of the world currently makes its inference requests to models hosted somewhere else.
- New centralized data center projects increasingly face long power, permitting, cooling and network delivery timelines, and some have stalled outright.
- Compact AI systems and modern inference runtimes can now serve genuinely useful open models from smaller distributed sites. This was not true two years ago.
- Enterprises want stronger control over data location, dedicated capacity, model ownership and vendor dependence.
- Telecom networks and carrier neutral facilities already provide a global footprint of powered, connected locations close to users. The buildings exist. The power is already provisioned. The fiber is already there.
The application layer is ready for real time AI. The missing layer is neutral infrastructure that can place, operate and route inference across many locations as easily as developers use a cloud API today.
Why we are building it this way
SwiftInference is designed as carrier neutral infrastructure. We do not want to build a single carrier walled garden. The platform has to work across telecom operators, tower companies, interconnection facilities, clouds, GPU vendors and model runtimes.
We will expand metro by metro, beginning where customers have a measurable latency, resilience, locality or cost problem. Each edge node is managed through the same software fabric, whether it is a compact pilot system, a production GPU server, or a cloud GPU used for fallback and parity testing.
That common operating model is essential. The network cannot scale if every new node needs a custom integration, a manual SSH session or its own orchestration stack. Our job is to make one GPU and one hundred thousand GPUs look like the same product to the customer.
It is also worth being clear about where the difficulty actually sits. The hardware is available to anyone. What is hard is the network position, the carrier relationships that allow traffic to stay on net, and the software that decides where a given request should run and gets it there. That is the part we are building.
Launching Bring Your Own GPU
We are also launching Bring Your Own GPU, for organizations that have purchased more capacity than they are currently using, or that want to own their inference hardware without building a distributed operations team.
Customers can place selected GPU capacity into the SwiftInference fabric while retaining ownership of the hardware. We deploy and operate it, connect it to the network, monitor it, and make it available through the same control plane and API used across SwiftInference. There is no hardware sale, no lease and no balance sheet transfer. Commercial terms are a capacity agreement, discussed individually.
In the product this is a workflow rather than a conversation: ship, rack, deploy, monitor. Customers browse the partnered colocation and carrier edge facilities on a map, filter by cooling type, place a colocation order and track the shipment to the building. Our technicians rack, cable, power on and burn in the hardware. From that point the servers appear as deployment targets alongside Swift owned capacity, and GPU utilisation, thermals, health and alerts appear in the same dashboard as everything else.
The commercial terms are still a capacity agreement discussed individually, because every situation genuinely is different. The operational path is not. It is a facility, a form, a shipping label and an inventory page.
For AI companies this turns idle capital equipment into production capacity serving their own customers. For data center and infrastructure operators it provides a faster path from installed GPU inventory to a managed inference service. For enterprises it offers greater control over models and capacity without forcing internal teams to become a 24/7 GPU operations group.
Who we are building for first
Our first customers and design partners are companies for whom inference performance is already part of the product rather than a background infrastructure metric:
- Voice AI and contact center platforms that need natural turn taking and predictable response times.
- Real time agents that listen, decide, call tools and act continuously.
- Computer vision and physical AI systems that need decisions near cameras, machines, vehicles or robots.
- AI companies serving users across multiple geographies and looking for a neutral distributed inference layer.
- Enterprises with data locality requirements or underutilized GPU assets.
- Telecom, tower and interconnection partners that want to turn existing sites into useful AI infrastructure.
What is launching today
Today is the beginning of the network. What is launching:
- The SwiftInference platform and its OpenAI compatible inference interface. Deploy a model to an edge node and call it the way you already call everything else.
- The Inference Gateway. One endpoint and sets of customer held credentials, routing to OpenAI, Anthropic, Gemini, SwiftInference hosted models and customer operated on premises endpoints.
- A catalogue of open models across text, vision, speech, image and video, filterable by modality, provider and licence, with upload for your own weights.
- A multimodal playground that reports time to first byte and tokens per second on every request.
- The agent platform, with voice agents, SIP telephony, MCP server support, tool calling and knowledge bases.
- Analytics with P50, P95 and P99 by model and by location, and CSV export.
- The self serve Bring Your Own GPU workflow.
- Team accounts with invitations and roles.
- A free tier that includes ten million tokens per billing period, so that evaluating us does not require a procurement conversation.
We are beginning with customers whose users span the United States and other international markets including Europe and West Africa, and we will add locations based on real customer demand.
How to start
The most useful first conversation is usually not a pitch. It is a measurement.
If you run a latency sensitive workload, you probably do not have per stage visibility into where your response time actually goes. Most teams have an end to end number and a suspicion. Route a slice of your traffic through our gateway with your own provider key and we will sit down with you over what comes back: time to first token and its spread, the latency distribution, and the same numbers by location. That is usually enough to tell distance from queueing from the model actually working. If the answer is that your bottleneck is the model, moving it closer will not help you and we will tell you that.
Measure first, decide later. The best starting point is one workload, one metro and a measurable target for P90 or P99 latency.
There is a shorter version of that. Create a free account, store your existing provider key in the gateway, and change one base URL. Your traffic continues to go exactly where it goes today. Then deploy one model to one edge node, send it the same prompts, and compare the two on time to first token. One workload, one metro, one number.
The larger vision
Inference is the starting point because it is the most urgent and visible real time workload. But an agent does more than call a model. It retrieves data, stores state, uses tools, communicates with other agents and triggers actions. If every one of those steps still travels to a distant region, accelerating only the model call will not be enough.
Over time we expect the SwiftInference edge platform to support more of that loop: caches, storage, databases, retrieval, tool execution and agent to agent communication. The long term goal is a distributed computing layer for applications that need to operate close to users and machines, with the cloud remaining an essential partner rather than the only place computation can happen.
Every tower. Every carrier. Every millisecond.
Join us
We are looking for design partners with a real latency, resilience, data locality or GPU utilization problem.
Learn more and request a pilot: www.swiftinference.ai
Run the comparison: www.swiftinference.ai/demo
Read the benchmark: Consistent Low-Latency AI Inference at the Network Edge, SSRN Abstract ID 6352298