# Batches

> UCall for Rust

A batch endpoint gathers matching requests from every event loop into one queue and hands them to its handler in batches, which suits work that is cheaper per request in bulk, like GPU inference.

```rust
use std::time::Duration;
use ucall::*;

let mut server = Server::new(&[Interface::new("127.0.0.1:0").pipelined_requests(8)])?;
let batching = Batching { min_size: 16, max_size: 64, max_latency: Duration::from_millis(2) };
let infer = server.batch("/infer", Verb::Post, batching, |batch: Batch| {
    let inputs: Vec<&[u8]> = batch.iter().map(Request::body).collect();
    let scores: Vec<String> = inputs.iter().map(|input| input.len().to_string()).collect();
    for (request, score) in batch.into_iter().zip(scores) {
        let _ = request.reply(HTTPStatus::OK, &[], score.as_bytes());
    }
})?;

std::thread::scope(|scope| {
    scope.spawn(|| server.take_calls(0));
    scope.spawn(|| infer.take_batches());
    scope.spawn(|| {
        std::thread::sleep(Duration::from_millis(100));
        println!("{:?}", infer.statistics());
        server.stop();
    });
});
```

`Batching` dispatches as soon as `min_size` requests wait, never more than `max_size` at once, and a smaller batch once its oldest request waited `max_latency`; zero fields mean 1, 32 and 10 milliseconds.
The handler runs on the thread polling the endpoint through `take_batch` or `take_batches`, never on an event loop, so a slow batch never stalls the network.
A `Batch` lends its requests in arrival order: `get` and `iter` borrow them, `take` moves one out, and iterating it by value takes them all, forwards or reversed.
Requests the handler neither replies to nor keeps answer 500 once it returns, and all of them do when it panics.
Taken requests may be replied to later, from any thread, while their responses still leave each connection in request order.

The queue holds as many requests as the interfaces admit at once, connections times pipelined requests, and refuses more with 503.
`Server::stop` releases the requests still queued without a reply, and a dropped `BatchEndpoint` leaves its queue unanswered until the server is freed.
`BatchEndpoint::statistics` counts batches and requests, and whether each batch was dispatched on reaching `max_size`, on reaching `min_size`, or once `max_latency` passed, from any thread while the endpoint is polled.

`use ucall::Batch;`

`use ucall::Batching;`

`use ucall::BodyDelivery;`

## Batch

The requests of one due batch, from every event loop, lent to the handler in arrival order.

```rust
pub struct Batch<'batch>
```

Each request is taken out at most once, by index or by iterating; whatever the handler leaves
answers 500 once it returns.

Implements: `IntoIterator`

## len

Requests in the batch, taken or not.

```rust
pub fn len(&self) -> usize
```

## is_empty

Whether the batch has no requests, which a dispatched batch never is.

```rust
pub fn is_empty(&self) -> bool
```

## get

Borrows request `index`, unless it was taken.

```rust
pub fn get(&self, index: usize) -> Option<&Request>
```

## take

Takes request `index` out of the batch, to reply to it here or on another thread.

```rust
pub fn take(&mut self, index: usize) -> Option<Request>
```

## iter

Borrows every request not yet taken, in order.

```rust
pub fn iter(&self) -> impl DoubleEndedIterator<Item = &Request>
```

## Batching

When a batch endpoint hands its queue to the handler; zero fields take the defaults of 1 request, 32 requests, and 10 milliseconds.

```rust
pub struct Batching
```

- `min_size` `usize`: Dispatch as soon as this many requests wait.
- `max_size` `usize`: Never hand the handler more than this many requests at once.
- `max_latency` `Duration`: Dispatch a smaller batch once its oldest request waited this long.

Implements: `Clone`, `Copy`, `StructuralPartialEq`, `PartialEq`, `Eq`, [`Debug`](https://www.unum.cloud/docs/ucall/rust/serving.md#Debug), `Default`

## BodyDelivery

When an endpoint's handler runs: once the body is buffered, or at headers time, with the body pulled through [`DetachedRequest::body_chunk`](https://www.unum.cloud/docs/ucall/rust/requests.md#body_chunk).

```rust
pub enum BodyDelivery
```

- `Buffered`: The handler runs once the whole body arrived, read through [`body`](https://www.unum.cloud/docs/ucall/rust/calling.md#body).
- `Streamed`: The handler runs once the headers arrived, and the body is pulled chunk by chunk.

Implements: `Clone`, `Copy`, `StructuralPartialEq`, `PartialEq`, `Eq`, [`Debug`](https://www.unum.cloud/docs/ucall/rust/serving.md#Debug), `Hash`, `Default`