Async Handlers Calling Their Own Port: Event-Loop Deadlock Diagnosis

A pattern that shows up in a surprising number of Python services: an HTTP handler, already running on the event loop, makes an outbound HTTP request to an endpoint served by the same process. Sometimes it is a deliberate internal API call. Sometimes it is a health check, a webhook replay, or a callback registered against the service's own public URL. Under low concurrency it works. Under load it hangs, and the hang looks like a network problem, a database problem, or a mysterious timeout in a dependency that is actually fine.

The underlying cause is almost always the same: the event loop is being asked to do two things at once, and one of them is blocking the other. This article walks through why that happens, how to confirm it, and what to change.

The shape of the problem

Consider a service built on an async framework. A request arrives, the handler runs as a coroutine on the event loop, and somewhere inside it the handler issues an HTTP request to http://localhost:8000/internal/something — the same port the service is listening on.

import httpx
from fastapi import FastAPI

app = FastAPI()

@app.get("/orders/{order_id}")
async def get_order(order_id: str):
    async with httpx.AsyncClient() as client:
        # This hits the same process, same event loop.
        resp = await client.get(f"http://localhost:8000/internal/enrich/{order_id}")
    return resp.json()

@app.get("/internal/enrich/{order_id}")
async def enrich(order_id: str):
    return {"order_id": order_id, "enriched": True}

On a single request this is fine. The event loop schedules the outbound client coroutine, the inbound server coroutine for /internal/enrich runs, the response comes back, and everything resolves. The loop interleaves the two coroutines cooperatively.

Now send concurrent requests. The loop has a fixed set of resources it can use to accept and dispatch inbound connections. If those resources are consumed by handlers that are themselves waiting on inbound connections, the loop cannot make progress on either side. The outbound request waits for the server to accept it; the server cannot accept it because the accept path is starved. That is the deadlock, and it is a resource-exhaustion deadlock rather than a lock-ordering one.

Why it is not always obvious

The failure mode is load-dependent and timing-dependent, which makes it hard to reproduce in a debugger. A few things mask it:

• Connection pooling. An HTTP client with a persistent pool may reuse a connection that is already established, so the accept path is not exercised on every call. The deadlock appears only when the pool is exhausted and new connections are needed.
• Multiple worker processes. If the service runs several worker processes behind a load balancer, the self-call may land on a different worker than the one handling the original request. That worker has its own event loop and its own accept capacity, so the deadlock is intermittent and looks like a routing quirk.
• Timeouts that fire first. Client timeouts, proxy timeouts, and load-balancer idle timeouts may all fire before the true deadlock is visible, so the symptom is a cascade of timeouts rather than a clean hang.
• Async frameworks that offload blocking work. Some frameworks push synchronous handlers to a thread pool. If the self-call is handled by a thread rather than the loop, the deadlock may not occur at all — until someone converts the handler to async def and the behavior changes.

A common pattern in incident write-ups: the service is described as "hanging under load," the on-call engineer checks CPU (low), memory (normal), database latency (normal), and the dependency that the self-call targets (healthy, because it is the same process). The actual problem is invisible to every dashboard that monitors external dependencies.

Confirming the diagnosis

The fastest confirmation is to inspect the event loop's state while the service is hung. There are two practical approaches.

Dump the stack of every task

Python's asyncio exposes the set of live tasks. A small debug endpoint or a signal handler can print them:

import asyncio
import traceback
import signal

def dump_tasks():
    loop = asyncio.get_event_loop()
    tasks = asyncio.all_tasks(loop)
    for task in tasks:
        print("---- task ----")
        print(task)
        stack = task.get_stack()
        for frame in stack:
            traceback.print_stack(frame)

signal.signal(signal.SIGUSR1, lambda *_: dump_tasks())

If the service is deadlocked on self-calls, the dump typically shows many tasks parked on await inside the HTTP client's connection acquisition — waiting for a connection from the pool or waiting for the transport to connect — while the server's accept path has no corresponding task making progress. The asymmetry is the tell: outbound waiters exist, inbound progress does not.

Count connections against the accept backlog

On the host, ss -tn state established '( sport = :8000 )' (or the platform equivalent) shows how many connections are established to the listening port. If the count is pinned at the configured backlog or at the client pool's maximum, and the application is not making progress, the self-call hypothesis is strong.

A useful cross-check: temporarily point the self-call at a different process (a stub server on another port). If the hang disappears, the problem is the shared event loop, not the network or the handler logic.

The mechanisms that actually cause the stall

There are several distinct mechanisms, and they have different fixes. It helps to name them.

1. Accept-path starvation

The server's listening socket has a backlog. The event loop must run the accept callback to pull connections off that backlog and hand them to the protocol layer. If every task on the loop is blocked waiting on an outbound connection that can only be accepted by that same loop, the accept callback never runs. The backlog fills, the kernel stops accepting, and the outbound connect eventually times out.

2. Connection-pool exhaustion

An async HTTP client with a bounded pool (for example, a limit on total connections or per-host connections) will queue requests once the limit is reached. If the requests holding the pool slots are the same ones that need the inbound side to respond, the pool never drains. The client's own semaphore becomes the deadlock point.

3. Single-threaded loop with a blocking call

If any part of the self-call path performs a synchronous operation — DNS resolution via a blocking resolver, a synchronous TLS handshake, a blocking read — the loop stalls entirely. This is not a deadlock in the cooperative sense; it is a stall that looks identical from the outside. The distinction matters because the fix is different: offload the blocking call rather than restructure the call graph.

4. Framework-level concurrency limits

Some frameworks cap the number of concurrent requests handled by a single worker. If the self-call consumes a slot and the handler that would serve it also needs a slot, the cap is reached and no further requests are admitted. This is a soft deadlock: it resolves only when a slot frees, and slots free only when requests complete, which requires slots.

Reproducing it deterministically

A minimal reproduction is worth building before changing anything, because it makes the fix verifiable.

import asyncio
import httpx
from aiohttp import web

async def internal(request):
    return web.json_response({"ok": True})

async def handler(request):
    async with httpx.AsyncClient(limits=httpx.Limits(max_connections=2)) as client:
        # Self-call into the same loop.
        resp = await client.get("http://127.0.0.1:8080/internal")
    return web.json_response(resp.json())

app = web.Application()
app.router.add_get("/internal", internal)
app.router.add_get("/outer", handler)

web.run_app(app, port=8080)

Fire more concurrent requests to /outer than the client's connection limit. The first batch occupies the pool; subsequent requests queue; the inbound /internal requests cannot be served because the loop is saturated with waiters. The service stops responding. Reduce concurrency below the limit and it works again. That threshold behavior is the signature.

Fixes, ordered by how much they change

Short term: break the self-call

The most direct fix is to stop making the HTTP call. If the handler needs data that another endpoint computes, call the underlying function directly rather than going through the network stack:

from myapp.services import enrich_order

@app.get("/orders/{order_id}")
async def get_order(order_id: str):
    return await enrich_order(order_id)

This removes the loop, the pool, the accept path, and the serialization overhead in one move. It is almost always the right answer for intra-process calls. The HTTP boundary was providing nothing except a false sense of decoupling.

Medium term: separate the execution contexts

If the self-call genuinely needs to go over HTTP — for example, because the target is a separate deployable that happens to share a host during development — run the two sides on different event loops or different processes. A dedicated internal worker process, reached over a Unix domain socket or a separate port, has its own accept capacity and cannot be starved by the caller's loop.

Medium term: bound and isolate

If the self-call must stay, give it a dedicated client with a pool sized to the expected concurrency, and give the server a separate listener or worker pool for internal traffic. The goal is to ensure that the resources needed to serve the internal request are not the same resources the caller is holding. This is a resource-partitioning fix, not a correctness fix, and it needs to be revisited whenever concurrency assumptions change.

Long term: make the dependency explicit

The deeper issue is architectural: a service that calls itself over the network has an implicit dependency on its own availability that is not represented anywhere. It cannot be scaled independently, it cannot be deployed independently, and it cannot be reasoned about with the usual tools. The durable fix is to make the dependency explicit — either as an in-process function call, or as a real separate service with its own lifecycle.

What to watch for in code review

A few patterns are worth flagging before they reach production:

• Any httpx, aiohttp, or requests call whose URL host resolves to the service's own listening address.
• Configuration values like SELF_URL, INTERNAL_API_BASE, or CALLBACK_URL that default to localhost or 127.0.0.1.
• Health checks or readiness probes that call the service's own public endpoint.
• Webhook registration code that registers the service's own URL as a callback target.
• Retry logic that reissues a self-call on timeout, compounding the pool pressure.

None of these are always wrong. A self-call to a genuinely separate process on the same host is fine. The problem is the case where "separate" is an assumption rather than a fact.

Observability that would have caught it

The reason this class of bug is expensive is that standard dashboards do not show it. A few additions help:

• Event-loop lag. Measure the time between scheduled callbacks and their actual execution. A rising lag under load is an early signal that the loop is saturated, before requests start timing out.
• In-flight task count by await point. Instrument the HTTP client to record what it is waiting on — pool acquisition, connect, TLS, response — and expose it as a metric. A growing count of tasks waiting on pool acquisition, with no corresponding growth in served requests, is the fingerprint.
• Accept backlog depth. The kernel exposes the listen queue; scraping it catches accept-path starvation directly.
• Per-endpoint concurrency. Track how many requests are in flight per route. A route that calls itself will show correlated in-flight counts on both the caller and the callee.

These are cheap to add and they turn a mysterious hang into a graph that points at the cause.

The general lesson

Async Python gives the appearance of concurrency while running on a single thread with cooperative scheduling. Any operation that requires the loop to make progress in order to complete — including accepting a connection that the loop itself is waiting on — is a potential self-deadlock. The rule of thumb: a coroutine should never await something whose completion depends on the same event loop making progress elsewhere, unless the dependency is explicitly bounded and the bound is enforced.

Self-calls over HTTP violate that rule by default. They are easy to write, they look like clean service boundaries, and they fail in a way that no external dependency monitor will catch. The fix is usually to delete the network hop. When it is not, the fix is to make sure the two sides are not sharing the resource that the deadlock depends on.