LlmProvider Trait

The LlmProvider trait defines the interface that all LLM provider implementations must satisfy. It abstracts the differences between provider APIs behind a common interface.

Trait Definition

The trait is defined in src/client/traits.rs:

use super::error::LlmError;
use super::http::HttpClient;
use super::models::{Message, MessageOptions, StreamEvent};
use futures::Stream;
use std::future::Future;
use std::pin::Pin;

pub trait LlmProvider {
    /// Send a message to the LLM.
    /// Returns the assistant's response message or an error.
    fn send_msg(
        &self,
        client: &HttpClient,
        messages: &[Message],
        options: &MessageOptions,
    ) -> Pin<Box<dyn Future<Output = Result<Message, LlmError>> + Send>>;

    /// Send a streaming message to the LLM.
    /// Returns a stream of events as they arrive from the API.
    fn send_msg_stream(
        &self,
        _client: &HttpClient,
        _messages: &[Message],
        _options: &MessageOptions,
    ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, LlmError>> + Send>>, LlmError>> + Send>> {
        Box::pin(async {
            Err(LlmError::new(
                "NOT_IMPLEMENTED",
                "Streaming not supported for this provider",
            ))
        })
    }
}

Required Method: send_msg

The send_msg method handles synchronous (non-streaming) message completion:

fn send_msg(
    &self,
    client: &HttpClient,
    messages: &[Message],
    options: &MessageOptions,
) -> Pin<Box<dyn Future<Output = Result<Message, LlmError>> + Send>>;

Parameters:

client: &HttpClient - HTTP client for making API requests
messages: &[Message] - Conversation history
options: &MessageOptions - Request options (max_tokens, temperature, tools, etc.)

Returns a pinned, boxed future that resolves to either a Message or an LlmError.

Optional Method: send_msg_stream

The send_msg_stream method handles streaming message completion:

fn send_msg_stream(
    &self,
    client: &HttpClient,
    messages: &[Message],
    options: &MessageOptions,
) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, LlmError>> + Send>>, LlmError>> + Send>>;

The default implementation returns a NOT_IMPLEMENTED error. Providers that support streaming override this method.

The return type is a future that resolves to a stream of StreamEvent values.

Message Type

The Message type represents conversation messages:

pub struct Message {
    pub role: MessageRole,
    pub content: Vec<ContentBlock>,
}

pub enum MessageRole {
    User,
    Assistant,
    System,
}

pub enum ContentBlock {
    Text(String),
    ToolUse { id: String, name: String, input: Value },
    ToolResult { tool_use_id: String, content: String },
}

MessageOptions Type

Request options passed to the provider:

pub struct MessageOptions {
    pub max_tokens: Option<u32>,
    pub temperature: Option<f32>,
    pub system_prompt: Option<String>,
    pub tools: Vec<LLMTool>,
    pub tool_choice: Option<ToolChoice>,
}

StreamEvent Type

Events emitted during streaming:

pub enum StreamEvent {
    /// Text content delta
    TextDelta(String),
    /// Tool use request
    ToolUse { id: String, name: String, input: Value },
    /// Message complete
    MessageComplete,
    /// Error during streaming
    Error(String),
}

LlmError Type

Error type returned by provider methods:

#[derive(Debug, Clone)]
pub struct LlmError {
    pub error_code: String,
    pub error_message: String,
}

impl LlmError {
    pub fn new(code: &str, message: &str) -> Self {
        Self {
            error_code: code.to_string(),
            error_message: message.to_string(),
        }
    }
}

HttpClient

The HttpClient handles HTTP communication with TLS and connection pooling:

impl HttpClient {
    pub fn new() -> Result<Self, LlmError>;

    pub async fn post(
        &self,
        url: &str,
        headers: &[(&str, &str)],
        body: &str,
    ) -> Result<String, LlmError>;

    pub async fn post_stream(
        &self,
        url: &str,
        headers: &[(&str, &str)],
        body: &str,
    ) -> Result<impl Stream<Item = Result<Bytes, LlmError>>, LlmError>;
}

Providers use post() for synchronous requests and post_stream() for streaming.

Pinned Boxed Future Pattern

The trait uses Pin<Box<dyn Future<...> + Send>> for async methods because:

Trait object safety - Async functions return opaque types; boxing makes them concrete
Send bound - Allows the future to be sent across threads
Pin - Required for self-referential futures

Implementation pattern:

fn send_msg(
    &self,
    client: &HttpClient,
    messages: &[Message],
    options: &MessageOptions,
) -> Pin<Box<dyn Future<Output = Result<Message, LlmError>> + Send>> {
    // Clone data needed in the async block
    let client = client.clone();
    let api_key = self.api_key.clone();
    let messages = messages.to_vec();
    let options = options.clone();

    Box::pin(async move {
        // Async implementation here
        // Uses cloned values, not references
    })
}

LLMClient Wrapper

The LLMClient provides a cleaner interface over the trait:

pub struct LLMClient {
    http_client: HttpClient,
    provider: Box<dyn LlmProvider + Send + Sync>,
}

impl LLMClient {
    pub fn new(provider: Box<dyn LlmProvider + Send + Sync>) -> Result<Self, LlmError>;

    pub async fn send_message(
        &self,
        messages: &[Message],
        options: &MessageOptions,
    ) -> Result<Message, LlmError>;

    pub async fn send_message_stream(
        &self,
        messages: &[Message],
        options: &MessageOptions,
    ) -> Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, LlmError>> + Send>>, LlmError>;
}

Provider Instantiation

Providers are created and wrapped in LLMClient:

use agent_air::client::{LLMClient, AnthropicProvider, OpenAIProvider};

// Anthropic
let provider = AnthropicProvider::new(api_key, model);
let client = LLMClient::new(Box::new(provider))?;

// OpenAI
let provider = OpenAIProvider::new(api_key, model);
let client = LLMClient::new(Box::new(provider))?;

Implementation Requirements

When implementing LlmProvider:

send_msg is required - Must handle non-streaming requests
send_msg_stream is optional - Override only if streaming is supported
Clone data for async blocks - Cannot hold references across await points
Convert API responses - Parse provider-specific responses to generic Message
Handle errors - Convert provider errors to LlmError

Minimal Implementation

A minimal provider implementation:

use agent_air::client::{
    LlmProvider, HttpClient, Message, MessageOptions, LlmError
};
use std::future::Future;
use std::pin::Pin;

pub struct MinimalProvider {
    pub api_key: String,
    pub model: String,
}

impl LlmProvider for MinimalProvider {
    fn send_msg(
        &self,
        client: &HttpClient,
        messages: &[Message],
        options: &MessageOptions,
    ) -> Pin<Box<dyn Future<Output = Result<Message, LlmError>> + Send>> {
        let client = client.clone();
        let api_key = self.api_key.clone();
        let model = self.model.clone();
        let messages = messages.to_vec();
        let options = options.clone();

        Box::pin(async move {
            // Build request body
            let body = build_request(&messages, &options, &model)?;

            // Make API call
            let headers = [
                ("Authorization", format!("Bearer {}", api_key).as_str()),
                ("Content-Type", "application/json"),
            ];
            let response = client.post("https://api.example.com/v1/chat", &headers, &body).await?;

            // Parse response
            parse_response(&response)
        })
    }

    // send_msg_stream uses default NOT_IMPLEMENTED
}

See Custom Providers for a complete implementation guide.

Building Agents

Architecture & Internals