Provider SDK

3. Provider Development Guide¶

This guide is designed to walk you through the process of extending the SLOP Server by adding your own custom providers (pkg/providers/<unique_provider_name>). It details the key steps, from initial setup to testing your new provider.

Setting Up¶

Before you begin developing a new provider, ensure your Go development environment is correctly configured and you have access to the SLOP Server project source code. It is recommended to familiarize yourself with the overall provider architecture described in the Core Concepts section of this document.

The initial steps for setting up your provider are as follows:

1. Choose a Provider Name: Select a unique and descriptive name for your provider, adhering to Go naming conventions (lowercase, single word, no hyphens or spaces). For example, myprovider.
2. Create Directory Structure: In the pkg/providers/ directory of your SLOP Server project, create a new directory named after your provider (e.g., pkg/providers/myprovider/).
3. Go Module Initialization (if necessary): If your provider has specific dependencies not managed by the main project module, you might need to manage its dependencies. However, in most cases, dependencies will be handled at the main project level.
4. Prepare Internal Dependencies: Your provider will likely interact with several core components of the SLOP Server. Ensure you understand how to import and use the interfaces and structs provided by internal packages such as internal/providers, internal/core, internal/jobmanager, internal/pubsub, and internal/resources.

The slop-server-cli provider create <provider_name> CLI tool can be used to automatically generate a basic structure for your new provider, including initial files and the models and services directories. This will provide a solid starting point for your development.

Basic Provider Structure¶

A typical provider within the SLOP Server is organized modularly to encapsulate its logic and dependencies. While the structure might vary slightly based on the provider's complexity, here are the common components found in the pkg/providers/<provider_name>/ directory:

• adapter.go: This is the heart of the provider. This file contains the implementation of the main adapter struct (e.g., MyProviderAdapter) that satisfies the providers.Provider interface (defined in internal/providers/interfaces.go). It handles initialization, configuration, registration of capabilities (tools, webhooks, etc.), and interaction with the SLOP Server core. It usually includes the init() function to register the provider's factory with the central registry.

• client.go (Optional): If your provider interacts with an external API or service, this file will contain the HTTP client logic, methods for making requests, handling authentication with the third-party service, and parsing responses. The unipile example shows a UnipileClient with its own services.

• models/ (Directory, Optional): This directory contains the Go struct definitions specific to your provider's domain. These models can represent data exchanged with an external service, specific configurations, or entities manipulated by the provider. For example, in unipile, account.go, email.go, etc., are found in pkg/providers/unipile/models/.

• services/ (Directory, Optional): For more complex providers, this directory can organize business logic into different services. Each service can then expose a set of capabilities (which will become tools). The unipile example uses this pattern with account_service.go, email_service.go, etc., in pkg/providers/unipile/services/. Each service file typically defines a service struct and a GetCapabilities() method.

• *_webhook_handler.go (Optional): If your provider needs to receive webhooks from external services, the HTTP handlers for these webhooks are implemented here. These handlers are then registered via the adapter's GetRoutes() method. For example, email_webhook_handler.go in unipile.

• *_job.go (Optional): For asynchronous tasks or background processing, this type of file defines the job logic. These jobs are then registered and managed by the central JobManager. sync_job.go in unipile is an example.

• store.go or pg_store.go (Optional): If your provider requires specific data persistence (beyond what the CoreStore or ResourceManager offer globally), you can define a store interface (e.g., MyProviderStoreInterface) and its implementation (often for PostgreSQL, hence pg_store.go). The unipile example has a pg_store.go for its specific storage needs.

• auth.go (Optional): May contain logic specific to the provider's own authentication or managing authentication mechanisms for the external services it uses. unipile has an auth.go for handling authentication links.

• helper.go (Optional): Utility functions specific to the provider that don't fit into other categories.

• readme.md: A Markdown file describing the provider, its purpose, configuration, capabilities, and any relevant development notes. This file is crucial for the provider's maintainability and understanding.

The unipile_pkg_files.txt example clearly illustrates this structure with files like adapter.go, client.go, and the models/ and services/ directories.

Implementing the Adapter¶

The adapter is the central component of your provider. It's a Go struct that implements the providers.Provider interface defined in internal/providers/interfaces.go. This interface defines the contract all providers must adhere to for proper integration with the SLOP Server.

Key methods of the providers.Provider interface your adapter must implement:

• GetID() string: Returns the unique identifier for your provider (e.g., "myprovider"). This ID is used for registration and configuration.
• GetName() string: Returns a human-readable name for your provider (e.g., "My Provider Integration").
• GetDescription() string: Returns a detailed description of the provider's role and functionalities.
• GetCompactDescription() string: Returns a short, concise description of the provider.
• Initialize(config map[string]interface{}) error: This method is called by the SLOP Server at startup to configure the provider. It receives a configuration map (config) extracted from the server's global configuration file (e.g., config.yaml) or environment variables. Here, you should:
  • Validate dependencies injected during adapter creation (via the factory).
  • Read and validate provider-specific configuration parameters (API keys, base URLs, etc.).
  • Initialize external service clients, provider-specific database connections, etc.
  • Mark the provider as initialized (usually via an internal boolean initialized). The unipile/adapter.go example shows how to load base_url, api_key, and client_secret from configuration or environment variables.
• GetProviderTools() []providers.ProviderTool: Returns a list of all tools (capabilities) exposed by this provider. Each tool is defined by a providers.ProviderTool struct. This method is typically called after Initialize.
• ExecuteTool(ctx context.Context, toolID string, params map[string]interface{}) (interface{}, error): Executes a specific tool identified by toolID with the provided params. The toolID usually follows a format like providerID_serviceName_capabilityName or providerID_toolName.
• GetRoutes() []providers.Route (Optional): If your provider exposes HTTP endpoints (e.g., for webhooks), this method returns a list of providers.Route.
• Shutdown() error (Optional): Allows the provider to cleanly release its resources (close connections, stop goroutines, etc.) when the server shuts down.

In addition to providers.Provider, your adapter can also implement other optional interfaces from the internal/providers package to extend its functionality:

• SchemaInitializer: If your provider needs to create or migrate a specific database schema during initialization. You would then implement InitializeSchema(ctx context.Context, pool *pgxpool.Pool) error.
• EventInfoProvider: If your provider publishes specific events, it can implement GetEventDefinitions() []providers.EventDefinition to declare them.
• Factory and init() Registration : Each provider must also supply a "factory" function (e.g., newMyProviderAdapterFactory) that takes providers.ProviderDependencies as an argument and returns an instance of providers.Provider (your adapter) and an error. These dependencies (ProviderDependencies) include core services like CoreStore, PgxPool, EncryptionKey, ResourceManager, Publisher (for Pub/Sub), and JobManager.

This factory is then registered in the init() function of your adapter.go file by calling providers.RegisterFactory(ProviderID, newMyProviderAdapterFactory). This allows the SLOP Server to discover and instantiate your provider at startup.

The unipile/adapter.go example shows a UnipileAdapter struct with its dependencies, its newUnipileAdapterFactory factory, and the implementation of the providers.Provider interface methods.

Registering the Provider¶

For your provider to be recognized and usable by the SLOP Server, it must be explicitly registered during the server's initialization phase. This process involves two main steps:

1. Creating a Provider Factory: In your adapter.go file, you must define a "factory" function. This function is responsible for creating an instance of your provider adapter. It takes a providers.ProviderDependencies struct as an argument, which contains all the core dependencies your provider might need (like CoreStore, PgxPool, EncryptionKey, ResourceManager, Publisher, JobManager). The factory must return an instance of providers.Provider (i.e., your adapter) and an error.

   Example factory signature (from unipile/adapter.go):

   // newUnipileAdapterFactory is the factory function called by the registry.
   func newUnipileAdapterFactory(deps providers.ProviderDependencies) (providers.Provider, error) {
       // ... dependency checks ...
       adapter, err := NewUnipileAdapter(deps.CoreStore, deps.EncryptionKey, deps.PgxPool, deps.ResourceManager, deps.Publisher, deps.JobManager)
       // ... error handling ...
       return adapter, nil
   }
   

2. Registration via init() function: Still in your adapter.go, you will use Go's special init() function to register your factory with the SLOP Server's central provider registry. This is done by calling providers.RegisterFactory(ProviderID, yourFactoryFunction). The ProviderID must be a unique string constant identifying your provider (e.g., const ProviderID = "myprovider").

   Example registration (from unipile/adapter.go):

   const (
       ProviderID = "unipile"
   )
   
   func init() {
       log.Println("DEBUG: unipile init() registering factory")
       if err := providers.RegisterFactory(ProviderID, newUnipileAdapterFactory); err != nil {
           log.Fatalf("CRITICAL: Failed to register provider factory for '%s': %v", ProviderID, err)
       }
       log.Printf("DEBUG: Provider factory for '%s' registration submitted.", ProviderID)
   }
   

Once these steps are completed, the SLOP Server will be able to instantiate and initialize your provider at startup. The server's initialization process (GetAllProviders(), then calling Initialize() on each provider) will ensure your provider is ready to operate.

Defining and Registering Tools¶

Tools are discrete functionalities exposed by your provider. They can be called by users via the API, by LLMs, or by other system components. Defining and registering tools primarily happens via your adapter's GetProviderTools() method.

Tool Structure (providers.ProviderTool)¶

Each tool is represented by the providers.ProviderTool struct, which includes the following fields:

• ID: A unique identifier for the tool, typically in the format providerID_serviceName_capabilityName or providerID_toolName (e.g., "myprovider_email_send").
• Name: A human-readable name for the tool (e.g., "Send Email").
• Scope: Usually an empty string "" for local providers.
• Description: A clear and concise explanation of what the tool does, its effects, and when to use it. This description is crucial for LLMs.
• Parameters: A map[string]providers.Parameter describing the parameters the tool accepts. Each providers.Parameter defines the type (string, integer, boolean, object, array), description, requirement status (Required), category (body, query, path), etc.
• Examples: A list of providers.ToolExample illustrating how to use the tool with sample parameters and expected results.

Implementing GetProviderTools()¶

In your adapter's GetProviderTools() method, you will construct and return a slice ([]providers.ProviderTool) of all tools offered by your provider.

If you use a service-based architecture (as in unipile with its services/ directory), each service might have a GetCapabilities() []providers.ProviderCapability method. The adapter's GetProviderTools() method would then iterate over these services, retrieve their capabilities, and transform them into providers.ProviderTool.

Example (conceptual, based on unipile/adapter.go):

func (a *MyProviderAdapter) GetProviderTools() []providers.ProviderTool {
    if !a.initialized {
        slog.Warn("Attempted to get tools from uninitialized provider", "provider", a.GetID())
        return nil
    }

    var tools []providers.ProviderTool
    providerID := a.GetID()

    // If you have an internal service registry, like in UnipileClient
    // internalServices := a.client.Registry.GetAll()
    // for serviceName, service := range internalServices {
    //    for _, capability := range service.GetCapabilities() { // GetCapabilities() is a method of your service
    //        tool := providers.ProviderTool{
    //            ID:          fmt.Sprintf("%s_%s_%s", providerID, serviceName, capability.Name),
    //            Name:        capability.Name,
    //            Description: capability.Description,
    //            Parameters:  capability.Parameters, // map[string]providers.Parameter
    //            Examples:    capability.Examples,
    //        }
    //        tools = append(tools, tool)
    //    }
    // }

    // Or define tools directly
    tools = append(tools, providers.ProviderTool{
        ID:          fmt.Sprintf("%s_my_feature", providerID),
        Name:        "My Specific Feature",
        Description: "Performs a specific action for my provider.",
        Parameters: map[string]providers.Parameter{
            "param1": {
                Type:        "string",
                Description: "Description of parameter 1.",
                Required:    true,
                Category:    "body",
            },
        },
    })

    slog.Debug("Returning tools for provider", "provider", providerID, "count", len(tools))
    return tools
}

Tool Parameters (providers.Parameter)¶

Precise parameter definition is essential. The providers.Parameter struct includes:

• Type: Data type (string, integer, boolean, number, array, object).
• Description: Explanation of the parameter.
• Required: Boolean indicating if the parameter is mandatory.
• Category: Where the parameter is expected in an HTTP request (body, query, path, header). For tools called by LLMs or internally, body is often used by convention for a set of parameters.
• Enum: List of possible values if the parameter is an enumeration.
• Default: Default value if the parameter is not provided.
• Items: If Type is array, Items (of type *providers.Parameter) describes the type of array elements.
• Properties: If Type is object, Properties (of type map[string]providers.Parameter) describes the object's fields.

A clear definition of tools and their parameters enables seamless integration with the rest of the SLOP Server and facilitates their use by LLMs.

Handling Tool Execution¶

Once tools are defined and registered, their execution logic must be implemented in your adapter's ExecuteTool(ctx context.Context, toolID string, params map[string]interface{}) (interface{}, error) method.

This method is responsible for:

1. Checking Initialization: Ensure the provider is initialized before attempting to execute a tool.
2. Identifying the Tool: Parse toolID to determine which specific tool needs to be executed. The toolID usually follows a format like providerID_serviceName_capabilityName or providerID_toolName. You'll need to extract the service name (if applicable) and the capability/tool name.
3. Validating and Extracting Parameters: The params are provided as a map[string]interface{}. You must extract the expected values, convert them to the correct types, and validate their presence if mandatory, as well as their format.
4. Executing Business Logic: Call the internal function or method that actually implements the tool's functionality. Pass the context.Context and validated parameters to this function.
5. Returning the Result: The method must return two values: interface{} for the execution result (which will typically be serialized to JSON) and an error if the execution failed.

Example structure for ExecuteTool (based on unipile/adapter.go):

func (a *MyProviderAdapter) ExecuteTool(ctx context.Context, toolID string, params map[string]interface{}) (interface{}, error) {
    if !a.initialized {
        return nil, fmt.Errorf("provider %s not initialized", a.GetID())
    }
    slog.InfoContext(ctx, "Executing tool", "provider", a.GetID(), "toolID", toolID)

    // Parse toolID. Example for "providerID_serviceName_capabilityName"
    parts := strings.SplitN(toolID, "_", 3)
    if len(parts) < 2 || parts[0] != a.GetID() { // At least providerID_toolName
        return nil, fmt.Errorf("invalid or mismatched tool ID format for provider %s: %s", a.GetID(), toolID)
    }

    var serviceName, capabilityName string
    if len(parts) == 3 {
        serviceName = parts[1]
        capabilityName = parts[2]
    } else { // len(parts) == 2, format providerID_toolName
        capabilityName = parts[1]
        // serviceName remains empty, or you have a convention for direct tools
    }

    // Route to specific tool logic
    // If using an internal services architecture:
    // internalService := a.client.Registry.GetService(serviceName) // Hypothetical
    // if internalService == nil {
    //    return nil, fmt.Errorf("unknown service '%s' for provider %s", serviceName, a.GetID())
    // }
    // return internalService.ExecuteCapability(ctx, capabilityName, params)

    // Or a switch on capabilityName (or a combination of serviceName/capabilityName)
    switch capabilityName { // or toolID if the format is simpler
    case "my_feature":
        // Extract and validate parameters from 'params'
        param1, ok := params["param1"].(string)
        if !ok || param1 == "" {
            return nil, errors.New("missing or invalid 'param1' parameter")
        }
        return a.doMyFeature(ctx, param1) // Call the implementation function
    // case "another_tool":
        // return a.doAnotherTool(ctx, params)
    default:
        return nil, fmt.Errorf("unknown tool ID or capability '%s' (service: '%s') for provider %s", capabilityName, serviceName, a.GetID())
    }
}

// Implementation function for the "my_feature" tool
func (a *MyProviderAdapter) doMyFeature(ctx context.Context, param1 string) (interface{}, error) {
    slog.InfoContext(ctx, "Executing my_feature", "param1", param1)
    // ... tool's business logic ...
    result := map[string]interface{}{
        "status":  "success",
        "message": "Feature executed with " + param1,
    }
    return result, nil
}

Proper error handling and returning clear, informative error messages are crucial. Parameter validation can be facilitated by libraries like github.com/mitchellh/mapstructure to decode the parameter map into a tool-specific Go struct, allowing the use of validation tags.

Implementing Webhook Handlers¶

If your provider needs to receive asynchronous notifications from external services (e.g., a status update, a new event), you will need to implement webhook handlers. These handlers are HTTP endpoints that the external service will call.

Key Steps:

1. Define Routes: In your adapter's GetRoutes() []providers.Route method, you declare the routes for your webhooks. Each providers.Route specifies:

   • Path: The URL path for the webhook (e.g., "/webhooks/myprovider/notification").
   • Method: The expected HTTP method (usually "POST").
   • Handler: An instance of http.Handler (often http.HandlerFunc(yourHandlerFunction)).
   • Description: A brief description of what the webhook does.
   • Auth: A boolean indicating if this route requires SLOP Server authentication. For external webhooks, this is typically false.

   Example GetRoutes (inspired by unipile/adapter.go):

   func (a *MyProviderAdapter) GetRoutes() []providers.Route {
       return []providers.Route{
           {
               Path:        fmt.Sprintf("/webhooks/%s/notify", a.GetID()),
               Method:      "POST",
               Handler:     http.HandlerFunc(a.handleWebhookNotification), // Your handler function
               Description: "Handles incoming notifications from MyProvider.",
               Auth:        false,
           },
       }
   }
   

2. Implement the Handler Function: Create a method on your adapter (or a separate function) that matches the signature of an http.HandlerFunc (i.e., func(w http.ResponseWriter, r *http.Request)). In this function:

   • Validate the Request (Optional but recommended): Check the source, signature (if the external service provides one), or other headers to ensure the request is legitimate.
   • Read and Parse the Request Body: The webhook payload will be in r.Body. Decode it (often JSON) into an appropriate Go struct.
   • Process the Notification: Perform the necessary actions based on the webhook's content. This might involve updating internal data, storing information, or publishing an internal event (see next section).
   • Respond to the External Service: Send an appropriate HTTP response (usually http.StatusOK (200) if everything went well, or an error code otherwise). It's important to respond quickly to avoid timeouts on the external service's side.

   Example handler function:

   func (a *MyProviderAdapter) handleWebhookNotification(w http.ResponseWriter, r *http.Request) {
       ctx := r.Context() // Use the request's context
       slog.InfoContext(ctx, "Webhook notification received", "provider", a.GetID(), "path", r.URL.Path)
   
       // Optional: Validate the request (e.g., check a secret signature)
   
       var payload models.MyWebhookNotification // Your struct for the payload
       if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
           slog.ErrorContext(ctx, "Failed to decode webhook payload", "error", err)
           http.Error(w, "Invalid payload", http.StatusBadRequest)
           return
       }
       defer r.Body.Close()
   
       // ... Process the payload ...
       // For example, publish an internal event:
       // eventData := map[string]interface{}{"id": payload.ID, "status": payload.Status}
       // if err := a.notifier.Publish(ctx, fmt.Sprintf("%s.notification.received", a.GetID()), eventData); err != nil {
       //    slog.ErrorContext(ctx, "Failed to publish webhook event", "error", err)
       //    // Decide if this should result in an HTTP error for the webhook
       // }
   
       slog.InfoContext(ctx, "Webhook processed successfully", "payload_id", payload.ID)
       w.WriteHeader(http.StatusOK)
       fmt.Fprintln(w, "Notification received")
   }
   

The SLOP Server will automatically route incoming requests matching the declared paths to your handlers.

Publishing Events¶

Providers can publish events to the SLOP Server's internal Pub/Sub system. This allows for decoupling components: a provider can signal that an event has occurred without knowing who (if anyone) is listening or reacting to that event.

Key Steps:

1. Dependency on Publisher: Ensure your adapter receives an instance of pubsub.Publisher via its dependencies (injected by the factory and stored in a field of your adapter, e.g., a.notifier).

2. Define Events (Optional but recommended): If your provider implements the providers.EventInfoProvider interface, you can declare the types of events it publishes via the GetEventDefinitions() []providers.EventDefinition method. Each providers.EventDefinition includes:

   • ID: A unique identifier for the event type (e.g., "myprovider.entity.created").
   • Description: An explanation of what the event signifies.
   • PayloadSchema: A schema (often JSON Schema as a map[string]interface{}) describing the expected structure of the event's payload.

   Example GetEventDefinitions:

   // func (a *MyProviderAdapter) GetEventDefinitions() []providers.EventDefinition {
   //    return []providers.EventDefinition{
   //        {
   //            ID:          fmt.Sprintf("%s.entity.created", a.GetID()),
   //            Description: "Triggered when a new entity is created via MyProvider.",
   //            PayloadSchema: map[string]interface{}{
   //                "type": "object",
   //                "properties": map[string]interface{}{
   //                    "entity_id": map[string]interface{}{"type": "string"},
   //                    "details":   map[string]interface{}{"type": "string"},
   //                },
   //                "required": []string{"entity_id"},
   //            },
   //        },
   //    }
   // }
   

3. Publish an Event: To publish an event, use the Publish(ctx context.Context, eventID string, payload interface{}) error method of the pubsub.Publisher instance.

   • eventID: The unique identifier for the event (e.g., "myprovider.entity.created").
   • payload: The data associated with the event. This can be any Go struct that can be serialized (usually to JSON), often a map[string]interface{} or a specific struct.

   Example of publishing an event:

   // In a method of your adapter or service
   // func (a *MyProviderAdapter) createEntity(ctx context.Context, data MyEntityData) error {
   //    // ... entity creation logic ...
   //    entityID := "123"
   //    details := "Some details"
   // 
   //    eventPayload := map[string]interface{}{
   //        "entity_id": entityID,
   //        "details":   details,
   //    }
   //    eventID := fmt.Sprintf("%s.entity.created", a.GetID())
   // 
   //    if err := a.notifier.Publish(ctx, eventID, eventPayload); err != nil {
   //        slog.ErrorContext(ctx, "Failed to publish entity creation event", "eventID", eventID, "error", err)
   //        return err // Or handle the error differently
   //    }
   //    slog.InfoContext(ctx, "Entity creation event published", "eventID", eventID)
   //    return nil
   // }
   

Published events can then be consumed by other parts of the system, such as configured event triggers that can launch jobs, call other tools, or notify users.

Registering and Implementing Jobs¶

Jobs are background tasks managed by the SLOP Server's central JobManager. Your provider can define specific job types and provide the logic to execute them.

Key Steps:

1. Dependency on JobManager: Your adapter must have access to an instance of jobmanager.Manager, typically injected via ProviderDependencies during its creation by the factory.

2. Define Job Type: Choose a unique identifier for each job type your provider handles (e.g., "myprovider.sync_data").

3. Implement Job Logic: The execution logic for a job is usually encapsulated in a function or method that takes a context.Context