Python · Serving

Serving

import ucallfrom ucall import Interface, Metrics, BatchEndpoint, … 1.0.0-rc.1 · 60202332

ucall.Server takes the interfaces to answer on, and backend and cpu_class keywords naming a ucall.Backend and a ucall.CPUClass, by member or by name such as “linux-epollaio”. Its log= callable receives each line the server writes as a ucall.LogLevel and a str, on whichever thread wrote it and holding the interpreter lock, filtered by log_level= and shaped by log_format=, and metrics=ucall.Metrics(address, sample_rate, prefix, tags, interval_microseconds) sends StatsD datagrams from every event loop. An Interface also takes max_request_bytes=, past which a request is answered 413, and reply_directories=, the directories a ucall.Path part names by position. descriptor= takes a listening socket, or its descriptor, that systemd or a supervisor opened, TCP or AF_UNIX, in place of an address; the server accepts on duplicates and never closes it. Without interfaces it answers on ucall.DEFAULT_ADDRESS, “127.0.0.1:8545”. An Interface is an address, optionally pinned to a Linux device, with its own security and admission limits, so one server answers plaintext on loopback and Noise on the wire at once. It admits 1024 connections by default, reads one request per connection at a time unless pipelined_requests allows more, and replies still leave in request order. At its connection budget a new peer evicts the idle keep-alive connection that progressed least recently. Port zero binds a free port, which server.addresses reports.

import ucall

public = ucall.Interface("0.0.0.0:0", event_loops=4, connections=16_384, pipelined_requests=16)
health = ucall.Interface("127.0.0.1:0", connections=4)
server = ucall.Server(public, health, backend="automatic", cpu_class=ucall.CPUClass.SERIAL)
print(f"{server.event_loops} event loops on {server.backend.name} with {server.cpu_class.name} kernels")


@server.get("/users/{user}/files/{*path}")
def listing(user: str, path: str, limit: int = 10, hidden: bool = False):
    return {"user": user, "path": path, "limit": limit, "hidden": hidden}


@server.post("/scale")
def scale(value: float, factor: float = 2.0):
    return value * factor


@server.route("/anything", ucall.Verb.ANY)
def anything(request: ucall.Request):
    request.reply_error(501, f"{request.verb.name} is not served\n")

Endpoints register before any loop is polled, with route taking any Verb and get, post, put, delete and patch naming one. Each decorator registers the function through add_endpoint and returns it unchanged, so it stays callable from Python. Templates capture {name} segments and a greedy {*name} tail, literal segments match before captures, and registering one template and verb twice raises ucall.Error with Status.ALREADY_EXISTS. Routing is exact: an empty segment is a segment, so /users and /users/ are different paths. A path no endpoint matches answers 404, one whose endpoints take other verbs answers 405 with an allow header, a GET endpoint also serves HEAD, and OPTIONS answers 204 with the same allow list. A request that cannot be parsed answers 400, 413 or 501 before reaching any handler.

Registration reads a handler’s signature once. A handler taking one parameter annotated ucall.Request receives the request itself. Any other handler takes parameters annotated str, int, float, bool or bytes, or left bare, and each binds by name from the first place that has it:

  • a placeholder of the template, as sent;
  • the query, percent-decoded with ”+” as a space, a ”%” without two hexadecimal digits answering 400;
  • a JSON object body, parsed once in C and only when an earlier place missed a parameter, with integers of any width kept exact;
  • the parameter’s default.

Text converts as int(text) and float(text) do, bool reads “1”, “true”, “yes” and “on” or “0”, “false”, “no” and “off” in any case, and bytes takes the raw bytes. A JSON value must already have the annotated type, except that an integer widens to float and a string encodes to bytes. A missing or malformed value answers 400 with the reason as plain text, such as “missing parameter ‘limit’”, without calling the handler. Registration raises TypeError for *args, **kwargs, more than 16 parameters, or an annotation outside that list. It refuses an async def handler with TypeError too, since handlers run on the polling thread; a coroutine application registers a plain function, calls request.detach(), and replies from its own event loop.

What a handler returns becomes the reply: bytes, bytearray and memoryview as application/octet-stream, str as UTF-8 text/plain, and anything else as application/json, written in C, or through json.dumps for a value C does not write, such as a dict keyed by numbers. None from a typed handler answers 204, and None from a handler taking the request means it replies itself. A handler’s exception reaches sys.unraisablehook and answers 500, and the loop keeps serving.

server.event_loop_statistics(loop) copies one loop’s counters as a dict, from any thread while the loop runs: polls, system calls, submissions and completions, accepted connections, requests, and the bytes received, sent and copied. Divided by requests, they price one call.

API

Types

BackendAn I/O implementation for Server and Client; AUTOMATIC picks the best one available.
__init__.py:123 ↗
class Backend(enum.IntEnum)

Values

AUTOMATIC0
POSIX1
LINUX_EPOLLAIO2
LINUX_URING6123
LINUX_URING6184
BSD_KQUEUE5
WINDOWS_IOCP6

Examples

>>> ucall.Server(ucall.Interface("127.0.0.1:0"), backend=ucall.Backend.POSIX).backend
<Backend.POSIX: 1>
VerbAn HTTP method; ANY registers an endpoint for every method and never arrives itself.
__init__.py:170 ↗
class Verb(enum.IntEnum)

Values

UNKNOWN0
GET1
POST2
PUT3
DELETE4
PATCH5
OPTIONS7
ANY8

Examples

>>> ucall.Verb.POST == 2
True
LogLevelThe severity of a line the Server log callback receives, from the most verbose up.
__init__.py:201 ↗
class LogLevel(enum.IntEnum)

Values

DEBUG0
INFO1
WARNING2
ERROR3
FATAL4

Examples

>>> ucall.LogLevel.WARNING > ucall.LogLevel.INFO
True
LogFormatHow the library spells a log line: as human-readable fields, or as one JSON object.
__init__.py:215 ↗
class LogFormat(enum.IntEnum)

Values

HUMAN0
JSON1

Examples

>>> ucall.LogFormat.JSON
<LogFormat.JSON: 1>
ClassInterfaceOne address a server answers on, with the security and admission of what arrives there.12
lib.pyi:300 ↗
class Interface

Raises

  • TypeError: An argument of the wrong type.
  • ValueError: A key other than 32 bytes long, or an unknown security.

Examples

>>> interface = ucall.Interface("127.0.0.1:0", connections=64, pipelined_requests=16)
>>> interface.connections, interface.security
(64, <Security.NONE: 0>)
Interface
lib.pyi:336 ↗
Interface(
    address: str = '',
    /,
    *,
    descriptor: int | socket.socket | None = None,
    device_name: str | None = None,
    security: Security | str = Security.NONE,
    noise_private_key: bytes | None = None,
    peer_authorizer: Callable[[bytes], bool] | None = None,
    noise_peer_key: bytes | None = None,
    event_loops: int = 0,
    connections: int = 1024,
    pipelined_requests: int = 0,
    max_request_bytes: int = 0,
    reply_directories: Sequence[str | bytes | os.PathLike[str] | os.PathLike[bytes]] = (),
)
addressThe "host:port" to listen on.
lib.pyi:354 ↗
address: str
device_nameThe pinned device, or None.
lib.pyi:357 ↗
device_name: str | None
securityEncryption of accepted connections.
lib.pyi:360 ↗
security: Security
noise_private_keyThe 32-byte Noise identity, or None.
lib.pyi:363 ↗
noise_private_key: bytes | None
peer_authorizerThe Noise peer vetting, or None.
lib.pyi:366 ↗
peer_authorizer: Callable[[bytes], bool] | None
event_loopsRequested loops; 0 chooses.
lib.pyi:369 ↗
event_loops: int
connectionsConnections open at once.
lib.pyi:372 ↗
connections: int
pipelined_requestsRequests one connection carries at once; 0 means one.
lib.pyi:375 ↗
pipelined_requests: int
noise_peer_keyThe 32-byte static key KK expects, or None.
lib.pyi:378 ↗
noise_peer_key: bytes | None
max_request_bytesThe largest request head and buffered body; 0 takes the kernel's receive buffer.
lib.pyi:381 ↗
max_request_bytes: int
reply_directoriesThe directories Path parts name by position.
lib.pyi:384 ↗
reply_directories: tuple[str | bytes, ...]
ClassMetricsWhere a Server sends StatsD datagrams, each event loop on its own schedule.6
lib.pyi:387 ↗
class Metrics

Examples

>>> metrics = ucall.Metrics("127.0.0.1:8125", tags=("service:inference",))
>>> server = ucall.Server(ucall.Interface("127.0.0.1:0"), metrics=metrics)
Metrics
lib.pyi:402 ↗
Metrics(
    address: str | bytes,
    sample_rate: float = 0.0,
    prefix: str | bytes | None = None,
    tags: Sequence[str | bytes] = (),
    interval_microseconds: int = 0,
)
addressThe StatsD "host:port".
lib.pyi:411 ↗
address: str | bytes
sample_rateLatencies reported; 0 takes 1.
lib.pyi:414 ↗
sample_rate: float
prefixThe metric name prefix, or None for "ucall".
lib.pyi:417 ↗
prefix: str | bytes | None
tagsGlobal "key:value" tags.
lib.pyi:420 ↗
tags: tuple[str | bytes, ...]
interval_microsecondsMicroseconds between datagrams.
lib.pyi:423 ↗
interval_microseconds: int
ClassBatchEndpointOne batching route of a server, polled by one application thread.4
lib.pyi:943 ↗
class BatchEndpoint

Requests matching it from every event loop share one queue, which take_batch and take_batches hand to the handler as a list once enough arrived or the oldest waited long enough. The handler runs on its polling owner's thread and replies to each request, in any order.

Examples

>>> @server.batch("/infer", min_size=8, max_size=64, max_latency_microseconds=500)
... def infer(requests: list[ucall.Request]):
...     for request in requests:
...         request.reply(bytes(request.body))
>>> infer in server.batch_endpoints
True
take_batchRuns the handler on this thread if a batch is due, waiting for one at most timeout_microseconds.
lib.pyi:961 ↗
def take_batch(timeout_microseconds: int = 0) -> None

Each batch endpoint has one polling owner, and the wait polls no event loop.

Parameters

timeout_microsecondsintThe longest wait in microseconds; 0 checks once.

Raises

  • Error: The server failed, or was released.

Examples

>>> endpoint.take_batch(timeout_microseconds=1_000)
take_batchesRuns every due batch on this thread, sleeping in between, until Server.stop().
lib.pyi:975 ↗
def take_batches() -> None

Requests still queued at the stop are released without a call to the handler.

Raises

  • Error: The server failed, or was released.

Examples

>>> poller = threading.Thread(target=endpoint.take_batches)
>>> poller.start()
statisticsCounts the batches the endpoint handed out since registration, and why each fired.
lib.pyi:987 ↗
def statistics() -> dict[str, int]

Each counter is read on its own, so a snapshot of an endpoint being polled is not one instant, and batches a stop released count nowhere.

Returns

dict[str, int]: batches handed to the handler and the requests in them, then the filled_batches that reached max_size, the ready_batches that reached min_size, and the expired_batches that waited out max_latency_microseconds.

Examples

>>> counters = endpoint.statistics()
>>> counters["requests"] / max(counters["batches"], 1)
0.0
handlerThe function handed each batch.
lib.pyi:959 ↗
handler: Callable[[list[Request]], object]
ClassServerAn HTTP server over one or more interfaces, polled by threads the application starts.20
lib.pyi:1004 ↗
class Server

It starts no threads: each event loop needs one owner calling take_calls, and each batch endpoint one calling BatchEndpoint.take_batches, while run serves the simplest server on the calling thread. Register every endpoint before the first poll.

Raises

  • Error: An interface that cannot be bound, such as an address in use, or a reply directory that cannot be opened.

Examples

>>> server = ucall.Server(ucall.Interface("127.0.0.1:0"), backend="posix")
>>> server.event_loops, server.backend
(1, <Backend.POSIX: 1>)
Server
lib.pyi:1030 ↗
Server(
    *interfaces: Interface = (),
    backend: Backend | str = Backend.AUTOMATIC,
    cpu_class: CPUClass | str = CPUClass.UNKNOWN,
    log: Callable[[LogLevel, str], object] | None = None,
    log_level: LogLevel | str = LogLevel.DEBUG,
    log_format: LogFormat | str = LogFormat.HUMAN,
    metrics: Metrics | None = None,
)
add_endpointRoutes verb requests matching the URI template path to handler, on the polling thread.
lib.pyi:1055 ↗
def add_endpoint(
    path: str | bytes,
    verb: Verb | str,
    handler: F,
    /,
    body: BodyDelivery | str = BodyDelivery.BUFFERED,
) -> F

A handler taking one parameter annotated Request receives the request, which it may answer later from any thread, and what it returns other than None is the reply. Any other handler's parameters, annotated str, int, float, bool or bytes or left bare, bind by name from a path placeholder, then the query, then a JSON object body, then their defaults; a missing or malformed value answers 400, and None returned answers 204. Returned bytes-like objects reply as binary, str as UTF-8 text, and anything else as JSON. An exception answers 500.

Parameters

pathstr | bytesA URI template such as /users/{name} or /files/{*path}.
verbVerb | strThe method to answer; Verb.ANY answers every one.
handlerFThe function to route to.
bodyBodyDelivery | strBodyDelivery.STREAMED runs the handler once the headers arrived, its body pulled through DetachedRequest.body_chunks. Defaults to buffered.

Returns

handler itself.

Raises

  • TypeError: A parameter that cannot bind, such as *args or one annotated list.
  • Error: A duplicate route or an invalid template.

Examples

>>> def scale(value: float, factor: float = 2.0):
...     return value * factor
>>> server.add_endpoint("/scale/{value}", ucall.Verb.GET, scale) is scale
True
add_batch_endpointQueues matching requests from every event loop for handler(requests), called in batches.
lib.pyi:1092 ↗
def add_batch_endpoint(
    path: str | bytes,
    verb: Verb | str,
    handler: Callable[[list[Request]], object],
    /,
    min_size: int = 0,
    max_size: int = 0,
    max_latency_microseconds: int = 0,
) -> BatchEndpoint

A batch is due once min_size requests arrived, max_size are queued, or the oldest waited max_latency_microseconds. The handler runs on the thread polling the returned endpoint and replies to each request.

Parameters

pathstr | bytesA URI template such as "/infer".
verbVerb | strThe method to answer; Verb.ANY answers every one.
handlerCallable[[list[Request]], object]The function handed each batch.
min_sizeintRequests that make a batch due at once; 0 means 1.
max_sizeintRequests one batch holds at most; 0 means 32.
max_latency_microsecondsintThe longest the oldest request waits for a batch to fill, in microseconds; 0 means 10 ms.

Returns

The endpoint to drain.

Raises

  • Error: A duplicate route or an invalid template.

Examples

>>> endpoint = server.add_batch_endpoint("/sizes", "POST", lambda requests: None, 4)
>>> endpoint in server.batch_endpoints
True
routeRegisters the decorated function for verb requests matching the URI template path.
lib.pyi:1128 ↗
def route(
    path: str | bytes,
    verb: Verb | str = Verb.GET,
    *,
    body: BodyDelivery | str = BodyDelivery.BUFFERED,
) -> Callable[[F], F]

Parameters

pathstr | bytesA URI template such as "/users/{name}".
verbVerb | strThe method to answer. Defaults to Verb.GET.
bodyBodyDelivery | strWhen the handler runs, as add_endpoint takes it.

Returns

Callable[[F], F]: A decorator calling add_endpoint and returning the function as is.

Examples

>>> @server.route("/anything", ucall.Verb.ANY)
... def anything(request: ucall.Request):
...     return request.verb.name
getRegisters the decorated function for GET requests matching the URI template path.
lib.pyi:1150 ↗
def get(
    path: str | bytes,
    /,
    *,
    body: BodyDelivery | str = BodyDelivery.BUFFERED,
) -> Callable[[F], F]

Parameters

pathstr | bytesA URI template such as "/users/{name}".
bodyBodyDelivery | strWhen the handler runs, as add_endpoint takes it.

Returns

Callable[[F], F]: A decorator calling add_endpoint and returning the function as is.

Examples

>>> @server.get("/users/{name}")
... def user(name: str, verbose: bool = False):
...     return {"name": name, "verbose": verbose}
postRegisters the decorated function for POST requests matching the URI template path.
lib.pyi:1165 ↗
def post(
    path: str | bytes,
    /,
    *,
    body: BodyDelivery | str = BodyDelivery.BUFFERED,
) -> Callable[[F], F]

Parameters

pathstr | bytesA URI template such as "/sum".
bodyBodyDelivery | strWhen the handler runs, as add_endpoint takes it.

Returns

Callable[[F], F]: A decorator calling add_endpoint and returning the function as is.

Examples

>>> @server.post("/sum")
... def sum(a: int, b: float = 0.5):
...     return a + b
putRegisters the decorated function for PUT requests matching the URI template path.
lib.pyi:1180 ↗
def put(
    path: str | bytes,
    /,
    *,
    body: BodyDelivery | str = BodyDelivery.BUFFERED,
) -> Callable[[F], F]

Parameters

pathstr | bytesA URI template such as /files/{*path}.
bodyBodyDelivery | strWhen the handler runs, as add_endpoint takes it.

Returns

Callable[[F], F]: A decorator calling add_endpoint and returning the function as is.

Examples

>>> @server.put("/files/{*path}")
... def store(request: ucall.Request):
...     return len(request.body)
deleteRegisters the decorated function for DELETE requests matching the URI template path.
lib.pyi:1195 ↗
def delete(
    path: str | bytes,
    /,
    *,
    body: BodyDelivery | str = BodyDelivery.BUFFERED,
) -> Callable[[F], F]

Parameters

pathstr | bytesA URI template such as /files/{*path}.
bodyBodyDelivery | strWhen the handler runs, as add_endpoint takes it.

Returns

Callable[[F], F]: A decorator calling add_endpoint and returning the function as is.

Examples

>>> @server.delete("/files/{*path}")
... def remove(path: str):
...     return None
patchRegisters the decorated function for PATCH requests matching the URI template path.
lib.pyi:1210 ↗
def patch(
    path: str | bytes,
    /,
    *,
    body: BodyDelivery | str = BodyDelivery.BUFFERED,
) -> Callable[[F], F]

Parameters

pathstr | bytesA URI template such as "/users/{name}".
bodyBodyDelivery | strWhen the handler runs, as add_endpoint takes it.

Returns

Callable[[F], F]: A decorator calling add_endpoint and returning the function as is.

Examples

>>> @server.patch("/users/{name}")
... def rename(name: str, alias: str):
...     return f"{name} is {alias}"
batchRegisters the decorated handler(requests) as a batch endpoint, replacing it with that.
lib.pyi:1225 ↗
def batch(
    path: str | bytes,
    verb: Verb | str = Verb.POST,
    *,
    min_size: int = 0,
    max_size: int = 0,
    max_latency_microseconds: int = 0,
) -> Callable[[Callable[[list[Request]], object]], BatchEndpoint]

Parameters

pathstr | bytesA URI template such as "/infer".
verbVerb | strThe method to answer. Defaults to Verb.POST.
min_sizeintRequests that make a batch due at once; 0 means 1.
max_sizeintRequests one batch holds at most; 0 means 32.
max_latency_microsecondsintThe longest the oldest request waits, in microseconds; 0 means 10 ms.

Returns

A decorator calling add_batch_endpoint and returning its BatchEndpoint.

Examples

>>> @server.batch("/reverse", min_size=4, max_size=4)
... def reverse(requests: list[ucall.Request]):
...     for request in reversed(requests):
...         request.reply(request.body)
>>> type(reverse).__name__
'BatchEndpoint'
take_callAdvances one event loop and runs its ready handlers on this thread.
lib.pyi:1255 ↗
def take_call(
    event_loop: int = 0,
    timeout_microseconds: int = 0,
) -> bool

Each loop has one polling owner, and handlers run inside the call, so a slow one delays its return past timeout_microseconds.

Parameters

event_loopintThe loop to advance, below event_loops. Defaults to 0.
timeout_microsecondsintThe longest wait for I/O in microseconds; 0 polls once.

Returns

False once the loop retired after stop(), with nothing left to take.

Raises

  • Error: An event_loop past the last one, or an I/O failure.

Examples

>>> while server.take_call(0, timeout_microseconds=1_000): pass
take_callsDrives one event loop on this thread until stop() or an I/O failure.
lib.pyi:1274 ↗
def take_calls(event_loop: int = 0) -> None

The interpreter lock is released meanwhile and taken back only to run handlers, so one thread per loop serves in parallel with the others. Signals reach Python only once it returns, so call it from threads other than the main one.

Parameters

event_loopintThe loop to drive, below event_loops. Defaults to 0.

Raises

  • Error: An event_loop past the last one, or an I/O failure.

Examples

>>> pollers = [threading.Thread(target=server.take_calls, args=(loop,))
...            for loop in range(server.event_loops)]
runServes a server of one event loop and no batch endpoints on the calling thread.
lib.pyi:1291 ↗
def run() -> None

It returns after stop() or KeyboardInterrupt, polling in slices of 100 ms and checking for signals between them, and once stopped it lets the loop finish the requests in flight; a second KeyboardInterrupt cuts them.

Raises

  • RuntimeError: The server has more than one event loop or any batch endpoint.
  • KeyboardInterrupt: Ctrl+C arrived while serving.
  • Error: An I/O failure.

Examples

>>> runner = threading.Thread(target=server.run)
>>> runner.start()
>>> server.stop()
>>> runner.join()
stopAsks the server to stop, waking every polling caller, of loops and batch endpoints, to return.
lib.pyi:1309 ↗
def stop() -> None

The first call drains: no new connections, idle ones close, and the requests already held are answered before their connections close. A second call cuts the replies still in flight.

Safe from any thread and from a handler. Join the polling threads before releasing the server.

Examples

>>> server.stop()
event_loop_statisticsCounts what one event loop did since initialization.
lib.pyi:1322 ↗
def event_loop_statistics(event_loop: int, /) -> dict[str, int]

Divided by requests, the counters price one call.

Parameters

event_loopintThe loop to read, below event_loops.

Returns

dict[str, int]: polls, empty_polls, kernel_enters, submissions, completions, cancellations, accepts, requests, receive_rearms, bytes_received, bytes_sent, bytes_copied, zero_copy_sends, zero_copy_fallbacks, copying_sends, file_reads_asynchronous, file_reads_blocking, bytes_read, file_window_overlaps, linger_resets, ready_passes, replied_dequeues, refused_requests, handshake_failures and authorizer_refusals, each read on its own.

Raises

  • Error: An event_loop past the last one.

Examples

>>> server.event_loop_statistics(0)["requests"]
0
event_loopsResolved loops, each needing one polling thread.
lib.pyi:1041 ↗
event_loops: int
backendThe I/O implementation initialization chose.
lib.pyi:1044 ↗
backend: Backend
cpu_classThe CPU kernels initialization chose.
lib.pyi:1047 ↗
cpu_class: CPUClass
addressesThe resolved "host:port" of each interface.
lib.pyi:1050 ↗
addresses: tuple[str, ...]
batch_endpointsBatch endpoints, each needing a polling owner.
lib.pyi:1053 ↗
batch_endpoints: tuple[BatchEndpoint, ...]

Constants

F
__init__.py:36 ↗
F = TypeVar('F', )
DEFAULT_ADDRESSThe address a Server without interfaces answers on, "127.0.0.1:8545".
__init__.py:39 ↗
DEFAULT_ADDRESS: Final[str]