Execution & Activity

Understanding how tools execute and monitoring their performance is critical for developing reliable tool-using agents. Prisme.ai provides powerful Activity tracking capabilities that give you visibility into every aspect of tool execution, from request processing to result delivery.

The Value of Activity Tracking

Activity tracking in Prisme.ai provides:

Visibility

Complete transparency into how tools execute

Diagnostics

Powerful debugging for error resolution

Analytics

Performance metrics and usage patterns

Logging

Comprehensive audit trails of all operations

Monitoring

Real-time observation of system behavior

Optimization

Data-driven insights for performance improvements

Understanding the Event System

At the core of Prisme.ai’s Activity tracking is its event-driven architecture:

Key Components

Events

Structured messages representing actions, status changes, or notificationsKey attributes:

Event type
Timestamp
Source identifier
Payload data
Request context

Publishers

Components that emit events when actions occurExamples:

Tool execution requests
Automation processes
User interactions
System status changes

Subscribers

Components that listen for and process specific eventsExamples:

Automation workflows
Activity monitors
Logging systems
Analytics processors

Event Bus

The central message broker that routes eventsResponsibilities:

Reliable message delivery
Event persistence
Publisher/subscriber coordination
Load management

The Execution Lifecycle

Tool execution follows a well-defined lifecycle that generates events at each stage:

Request Initiation

The process begins when an agent decides to use a tool.Events generated:

tool.request.initiated
tool.parameter.validation.started

Key information:

Tool name and parameters
Request source (conversation ID, user ID)
Timestamp and request ID
Context information

Parameter Validation

The system validates that all required parameters are present and properly formatted.Events generated:

tool.parameter.validation.completed
tool.parameter.validation.failed (if issues occur)

Key information:

Validation status
Error details (if any)
Parameter transformations
Schema conformance

Execution

The actual tool code runs, performing its designated function.Events generated:

tool.execution.started
tool.execution.step.completed (for multi-step tools)
tool.execution.completed

Key information:

Execution duration
Resource utilization
Intermediate state
External system interactions

Result Processing

The system processes the tool’s output for agent consumption.Events generated:

tool.result.processing.started
tool.result.processing.completed

Key information:

Raw output data
Processed results
Formatting operations
Output size and structure

Response Delivery

The formatted results are returned to the agent.Events generated:

tool.response.delivery.started
tool.response.delivery.completed

Key information:

Final response format
Delivery time
End-to-end latency
Response size

Accessing Activity in Prisme.ai

You can monitor and analyze tool execution through several interfaces:

The primary interface for monitoring and debugging tool execution.Key features:

Chronological event listing
Filtering by event type, status, or date range
Detailed event inspection
Real-time monitoring
Search capabilities

Access:

Navigate to the “Activity” tab in your AI Builder workspace
Use filtering to focus on relevant events
Click on events to see detailed information
Toggle real-time updates for live monitoring

Common Event Types

Prisme.ai’s event system uses a hierarchical naming convention for clarity and filtering:

Tool Request Events

Events related to the initial tool request and setup.Key event types:

tool.request.initiated: Initial request for tool execution
tool.request.validated: Request passed preliminary validation
tool.request.rejected: Request failed preliminary checks

Example event data:

{
  "type": "tool.request.initiated",
  "timestamp": "2025-03-15T14:32:56.789Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "parameters": {
      "symbol": "AAPL",
      "dateRange": "5d"
    },
    "requester": {
      "conversationId": "conv-789012",
      "userId": "user-345678"
    }
  }
}

Execution Events

Events related to the actual tool execution process.Key event types:

tool.execution.started: Tool execution has begun
tool.execution.step.completed: A step in multi-step execution completed
tool.execution.completed: Tool execution finished successfully
tool.execution.failed: Tool execution encountered an error

Example event data:

{
  "type": "tool.execution.completed",
  "timestamp": "2025-03-15T14:33:01.456Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "duration": 4667,
    "resourceUsage": {
      "cpuTime": 1250,
      "memoryPeak": 256384
    },
    "externalCalls": [
      {
        "service": "finance-api",
        "endpoint": "/stock/price",
        "duration": 3890
      }
    ]
  }
}

Result Events

Events related to the processing and delivery of tool results.Key event types:

tool.result.generated: Raw results from tool execution
tool.result.processed: Results after formatting and preparation
tool.result.delivered: Final delivery to the requester

Example event data:

{
  "type": "tool.result.processed",
  "timestamp": "2025-03-15T14:33:02.123Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "rawResultSize": 8712,
    "processedResultSize": 2340,
    "resultSummary": {
      "symbol": "AAPL",
      "currentPrice": 187.43,
      "priceChange": 2.31,
      "percentChange": 1.25,
      "dataPoints": 5
    }
  }
}

Error Events

Events related to errors during the tool execution lifecycle.Key event types:

tool.error.validation: Error during parameter validation
tool.error.execution: Error during tool execution
tool.error.timeout: Execution exceeded allocated time
tool.error.resource: Resource limit exceeded

Example event data:

{
  "type": "tool.error.execution",
  "timestamp": "2025-03-15T14:33:00.789Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "errorCode": "API_UNAVAILABLE",
    "errorMessage": "External finance API returned status 503",
    "errorDetails": {
      "httpStatus": 503,
      "responseBody": "Service temporarily unavailable",
      "endpoint": "https://finance-api.example.com/stock/price"
    },
    "executionStage": "external_api_call"
  }
}

Debugging with Activity

Activity tracking provides powerful capabilities for diagnosing and resolving issues:

Identify the Issue

Use filtering to locate relevant events.Effective filtering strategies:

Search for specific request IDs
Filter by error event types
Focus on a specific time range
Filter by tool name or source
Search for specific error codes

Trace Execution Flow

Follow the sequence of events to understand the execution path.Key analysis techniques:

Map the complete event sequence
Identify missing or unexpected events
Analyze timing between events
Examine state changes across events
Look for execution branches

Examine Event Details

Inspect detailed event data for diagnostic information.Focus areas:

Parameter values and transformations
Error details and context
Performance metrics
External system interactions
Resource utilization

Compare with Successful Executions

Analyze differences between failed and successful runs.Comparison strategies:

Identify parameter differences
Compare timing patterns
Analyze resource usage
Examine external factors
Look for environmental changes

Implement and Verify Fixes

Make changes and confirm resolution through activity monitoring.Verification approach:

Test the same parameters
Monitor in real-time
Verify complete event sequence
Check performance metrics
Test edge cases

Monitoring Performance with Activity

Activity data is invaluable for performance optimization:

Latency Analysis

Identify and address performance bottlenecks.Key metrics to monitor:

End-to-end latency: Total time from request to response
Execution time: Time spent in actual tool execution
External call latency: Time spent waiting for external services
Processing overhead: Time spent in non-execution activities

Implementation approach:

// Calculate average latencies by tool
function analyzeToolLatencies(events, timeRange) {
  const executions = {};
  
  // Group events by request ID
  events.forEach(event => {
    if (!executions[event.data.requestId]) {
      executions[event.data.requestId] = {
        tool: event.data.toolName,
        events: []
      };
    }
    executions[event.data.requestId].events.push(event);
  });
  
  // Calculate metrics for each execution
  const toolMetrics = {};
  Object.values(executions).forEach(execution => {
    const startEvent = execution.events.find(e => e.type === 'tool.request.initiated');
    const endEvent = execution.events.find(e => e.type === 'tool.result.delivered');
    const execStartEvent = execution.events.find(e => e.type === 'tool.execution.started');
    const execEndEvent = execution.events.find(e => e.type === 'tool.execution.completed');
    
    if (startEvent && endEvent && execStartEvent && execEndEvent) {
      const tool = execution.tool;
      if (!toolMetrics[tool]) {
        toolMetrics[tool] = {
          count: 0,
          totalLatency: 0,
          totalExecTime: 0,
          totalOverhead: 0
        };
      }
      
      const latency = endEvent.timestamp - startEvent.timestamp;
      const execTime = execEndEvent.timestamp - execStartEvent.timestamp;
      const overhead = latency - execTime;
      
      toolMetrics[tool].count++;
      toolMetrics[tool].totalLatency += latency;
      toolMetrics[tool].totalExecTime += execTime;
      toolMetrics[tool].totalOverhead += overhead;
    }
  });
  
  // Calculate averages
  Object.keys(toolMetrics).forEach(tool => {
    const metrics = toolMetrics[tool];
    metrics.avgLatency = metrics.totalLatency / metrics.count;
    metrics.avgExecTime = metrics.totalExecTime / metrics.count;
    metrics.avgOverhead = metrics.totalOverhead / metrics.count;
    metrics.overheadPercentage = (metrics.avgOverhead / metrics.avgLatency) * 100;
  });
  
  return toolMetrics;
}

Error Rate Tracking

Monitor and minimize tool failures.Key metrics to track:

Overall error rate: Percentage of tool executions that fail
Error breakdown by type: Distribution of different error categories
Error trends: Changes in error rates over time
Correlation analysis: Factors associated with higher error rates

Implementation approach:

// Calculate error rates by tool and type
function analyzeErrorRates(events, timeRange) {
  const toolStats = {};
  
  // Count total executions and errors
  events.forEach(event => {
    const tool = event.data.toolName;
    
    if (!toolStats[tool]) {
      toolStats[tool] = {
        totalExecutions: 0,
        errors: {
          total: 0,
          byType: {}
        }
      };
    }
    
    if (event.type === 'tool.execution.started') {
      toolStats[tool].totalExecutions++;
    }
    
    if (event.type.startsWith('tool.error.')) {
      toolStats[tool].errors.total++;
      
      const errorType = event.type.replace('tool.error.', '');
      if (!toolStats[tool].errors.byType[errorType]) {
        toolStats[tool].errors.byType[errorType] = 0;
      }
      toolStats[tool].errors.byType[errorType]++;
    }
  });
  
  // Calculate rates
  Object.keys(toolStats).forEach(tool => {
    const stats = toolStats[tool];
    stats.errorRate = stats.errors.total / stats.totalExecutions;
    
    stats.errors.typeBreakdown = {};
    Object.keys(stats.errors.byType).forEach(errorType => {
      stats.errors.typeBreakdown[errorType] = 
        stats.errors.byType[errorType] / stats.errors.total;
    });
  });
  
  return toolStats;
}

Usage Pattern Analysis

Understand how tools are being used.Key metrics to analyze:

Usage frequency: How often each tool is called
Parameter distributions: Common values and patterns
Usage contexts: When and why tools are invoked
User segmentation: Differences in tool usage across users

Implementation approach:

// Analyze tool usage patterns
function analyzeUsagePatterns(events, timeRange) {
  const usageStats = {
    byTool: {},
    byHour: Array(24).fill(0),
    byUser: {},
    parameterDistributions: {}
  };
  
  events.forEach(event => {
    if (event.type === 'tool.request.initiated') {
      const tool = event.data.toolName;
      const timestamp = new Date(event.timestamp);
      const hour = timestamp.getHours();
      const userId = event.data.requester?.userId || 'unknown';
      const parameters = event.data.parameters || {};
      
      // Count by tool
      if (!usageStats.byTool[tool]) {
        usageStats.byTool[tool] = 0;
      }
      usageStats.byTool[tool]++;
      
      // Count by hour
      usageStats.byHour[hour]++;
      
      // Count by user
      if (!usageStats.byUser[userId]) {
        usageStats.byUser[userId] = {
          total: 0,
          byTool: {}
        };
      }
      usageStats.byUser[userId].total++;
      if (!usageStats.byUser[userId].byTool[tool]) {
        usageStats.byUser[userId].byTool[tool] = 0;
      }
      usageStats.byUser[userId].byTool[tool]++;
      
      // Analyze parameters
      if (!usageStats.parameterDistributions[tool]) {
        usageStats.parameterDistributions[tool] = {};
      }
      Object.keys(parameters).forEach(param => {
        if (!usageStats.parameterDistributions[tool][param]) {
          usageStats.parameterDistributions[tool][param] = {};
        }
        
        const value = String(parameters[param]);
        if (!usageStats.parameterDistributions[tool][param][value]) {
          usageStats.parameterDistributions[tool][param][value] = 0;
        }
        usageStats.parameterDistributions[tool][param][value]++;
      });
    }
  });
  
  return usageStats;
}

Optimizing Tool Execution

Use Activity data to drive execution optimizations:

Parameter Optimization

Refine parameter handling based on usage patternsStrategies:

Add intelligent defaults for common values
Implement parameter validation improvements
Optimize parameter extraction from user inputs
Add preprocessing for frequently used parameter formats

Execution Efficiency

Improve tool execution performanceStrategies:

Identify and optimize slow execution steps
Implement caching for frequent operations
Reduce external API latency
Optimize resource utilization

Error Reduction

Minimize tool execution failuresStrategies:

Add robust error handling for common failures
Implement automatic retries for transient issues
Improve validation to catch problems early
Create fallback mechanisms for critical functions

Result Processing

Enhance result formatting and deliveryStrategies:

Optimize result formatting for LLM consumption
Reduce result size for better context utilization
Implement progressive result delivery for long operations
Add enrichment for more valuable outputs

Custom Event Emission

You can emit custom events to enhance visibility into your tools:

Custom Event Types

Define your own event types for specialized monitoring.Naming conventions:

custom.[workspace].[category].[action]

Examples:

custom.finance-assistant.calculation.started
custom.support-agent.ticket.created
custom.inventory-tool.stock.checked

This allows for clear organization and filtering of your custom events.

Emitting Custom Events

Add custom event emission to your tools.Example implementation:

slug: product-recommendation-tool
do:
  # Emit custom event to track recommendation request
  - emit:
      event: custom.product-tool.recommendation.requested
      data:
        userId: '{{event.data.userId}}'
        productCategory: '{{event.data.parameters.category}}'
        priceRange: '{{event.data.parameters.priceRange}}'
        timestamp: '{{now()}}'
  
  # Execute recommendation logic
  - ProductCatalog.search:
      category: '{{event.data.parameters.category}}'
      priceMin: '{{event.data.parameters.priceRange.min}}'
      priceMax: '{{event.data.parameters.priceRange.max}}'
      output: searchResults
  
  # Emit event for search completion
  - emit:
      event: custom.product-tool.search.completed
      data:
        resultsCount: '{{length(searchResults.products)}}'
        executionTime: '{{searchResults.executionTime}}'
  
  # Apply recommendation algorithm
  - RecommendationEngine.process:
      products: '{{searchResults.products}}'
      userPreferences: '{{event.data.parameters.preferences}}'
      output: recommendations
  
  # Emit final recommendation event
  - emit:
      event: custom.product-tool.recommendation.generated
      data:
        recommendationCount: '{{length(recommendations)}}'
        topRecommendationId: '{{recommendations[0].id}}'
        algorithmsApplied: '{{recommendations.algorithmsApplied}}'
  
  # Return results
  - set:
      name: output
      value:
        recommendations: '{{recommendations}}'

This approach:

Provides detailed visibility into tool execution stages
Captures business-specific metrics
Enables more nuanced performance analysis
Creates richer debugging context

Event Payload Best Practices

Design effective event payloads for maximum utility.Recommendations:

Include identifiers: Always add request IDs, user IDs, and other correlation identifiers
Add timestamps: Include timing information for performance analysis
Provide context: Add relevant business context to make events meaningful
Size appropriately: Include useful data without making events too large
Structure consistently: Use consistent naming and structure across related events

Example event structure:

{
  "type": "custom.order-tool.validation.completed",
  "timestamp": "2025-03-15T15:45:12.678Z",
  "source": "workspace:commerce-assistant",
  "data": {
    "requestId": "req-abc123",
    "orderId": "ORD-98765",
    "validationResults": {
      "inventory": {
        "status": "passed",
        "duration": 124
      },
      "pricing": {
        "status": "passed",
        "duration": 86
      },
      "shipping": {
        "status": "warning",
        "message": "Delivery may be delayed",
        "duration": 231
      }
    },
    "totalDuration": 441,
    "validationPassed": true
  }
}

Real-Time Monitoring

For critical tools, implement real-time monitoring using Activity:

Set Up Real-Time Dashboards

Create specialized dashboards for key metrics.Include dashboards for:

Error rates and types
Execution latency
Usage volume
Resource utilization
Business-specific metrics

Configure Alerts

Define thresholds and conditions for notifications.Alert types to consider:

Error rate spikes
Latency threshold violations
Abnormal usage patterns
Resource exhaustion warnings
Custom business metric anomalies

Implement Event Subscribers

Create subscribers for real-time processing.Example implementation:

slug: error-rate-monitor
when:
  event: tool.error.*
do:
  # Increment error counter
  - cache.increment:
      key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
      value: 1
      expire: 86400  # 24 hours
  
  # Get current hour's execution count
  - cache.get:
      key: 'executions:{{date(now(), "YYYY-MM-DD:HH")}}'
      output: executionCount
  
  # Get current hour's error count
  - cache.get:
      key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
      output: errorCount
  
  # Calculate error rate
  - set:
      name: errorRate
      value: '{{errorCount / executionCount}}'
  
  # Check if threshold exceeded
  - conditions:
      '{{errorRate > 0.05 && executionCount > 10}}':  # 5% error rate with at least 10 executions
        - notification.send:
            channel: 'slack'
            message: 'Alert: Tool error rate at {{errorRate * 100}}% ({{errorCount}} errors out of {{executionCount}} executions) in the past hour'

Monitor Service Health

Track overall system performance and health.Focus areas:

API response times
Service availability
Resource utilization
Queue lengths
Error distributions

Activity for Compliance and Audit

Beyond operational uses, Activity provides essential compliance capabilities:

Audit Trails

Maintain comprehensive records of all tool interactionsKey aspects:

Complete chronological record of events
User and system actions
Parameter and result tracking
Timing information
Access evidence

Compliance Reporting

Generate reports for regulatory requirementsCapabilities:

Data access audit reports
Processing evidence
Security monitoring
Policy enforcement verification
Chain of custody documentation

Forensic Analysis

Detailed investigation of specific incidentsUse cases:

Security incident investigation
Error root cause analysis
User behavior auditing
System interaction reconstruction
Performance issue diagnostics

Records Retention

Maintain activity data according to policiesConsiderations:

Retention period configuration
Data archiving strategies
Privacy compliance
Data minimization
Access controls

Best Practices for Activity Utilization

Develop an Event Strategy

Plan your event ecosystem for maximum value.Recommendations:

Define an event taxonomy: Create a consistent naming and categorization system
Identify key lifecycle events: Ensure all important state transitions emit events
Balance detail and volume: Emit enough events for visibility without overwhelming storage
Consider event consumers: Design with reporting, monitoring, and analysis needs in mind
Document event types: Maintain clear documentation of all events and their meanings

Implementation approach:

Event Naming Structure:

[domain].[entity].[action]

Examples:
- tool.request.initiated
- order.payment.processed
- user.profile.updated
- system.resource.allocated

Implement Effective Filtering

Create filtering strategies for efficient data access.Recommendations:

Use consistent metadata: Add standard fields like request IDs to enable correlation
Include categorization data: Add tags, types, and categories for efficient filtering
Consider time-based access: Optimize for common time-range queries
Balance filter granularity: Too granular filters can be as problematic as too broad ones
Test filter performance: Ensure filtering is efficient at production scale

Example filtering approach:

// Effective event filtering
const events = await activity.queryEvents({
  // Time range
  timeRange: {
    start: new Date(Date.now() - (7 * 24 * 60 * 60 * 1000)), // 7 days ago
    end: new Date()
  },
  // Event types (using hierarchical pattern)
  eventTypes: ['tool.execution.*', 'tool.error.*'],
  // Source filter
  sources: ['workspace:customer-service'],
  // Metadata filters
  filters: [
    { field: 'data.requestId', operator: 'eq', value: 'req-123456' },
    { field: 'data.toolName', operator: 'in', value: ['CustomerLookup', 'OrderStatus'] }
  ],
  // Sorting
  sort: { field: 'timestamp', direction: 'desc' },
  // Pagination
  limit: 100,
  offset: 0
});

Design for Scalability

Ensure your activity management scales with usage.Recommendations:

Implement data lifecycle management: Archive older events to control storage growth
Consider sampling for high-volume events: For very frequent events, consider sampling
Optimize query patterns: Design queries to use available indexes
Leverage aggregation: Use pre-aggregated metrics for frequent analyses
Plan for growth: Design with future scale in mind

Example lifecycle implementation:

slug: activity-lifecycle-management
when:
  schedule: '0 0 * * *'  # Daily at midnight
do:
  # Archive events older than 30 days
  - activity.archiveEvents:
      olderThan: '{{date.add(now(), -30, "days")}}'
      destination: 'cold-storage'
  
  # Delete archived events older than 365 days
  - activity.deleteArchivedEvents:
      olderThan: '{{date.add(now(), -365, "days")}}'
  
  # Aggregate metrics for retention
  - activity.aggregateMetrics:
      timeRange:
        start: '{{date.add(now(), -2, "days")}}'
        end: '{{date.add(now(), -1, "days")}}'
      dimensions: ['toolName', 'hour', 'errorType']
      metrics: ['count', 'avgDuration', 'errorRate']
  
  # Emit completion event
  - emit:
      event: system.activity.lifecycle.completed
      data:
        timestamp: '{{now()}}'

Activity in AI Knowledge

For AI Knowledge users, Activity information is accessible through built-in interfaces:

Access the Activity Log

Find the Activity section in AI Knowledge.Key features:

Chronological event listing
Filtering by event type and date
Search capabilities
Drill-down for details

Monitor Tool Usage

Track how agents are using tools.Available information:

Tool usage frequency
Success and error rates
Performance metrics
Parameter patterns

Analyze Conversation Flows

Examine how tools integrate into conversations.Analysis capabilities:

Tool usage in conversation context
User reactions to tool results
Multi-turn tool interactions
Conversation paths and patterns

Export Activity Data

Extract data for external analysis.Export options:

CSV for spreadsheet analysis
JSON for programmatic processing
Filtered or complete data sets
Custom date ranges

Next Steps

Ready to implement effective execution monitoring and activity tracking? Continue with these resources:

Error Handling

Learn how to implement robust error management

AI Builder Event System

Explore the event system in greater depth

Getting Started

Prompting & RAG Agent

Tool-Using Agents

Execution & Activity

The Value of Activity Tracking

Visibility

Diagnostics

Analytics

Logging

Monitoring

Optimization

Understanding the Event System

Key Components

Events

Publishers

Subscribers

Event Bus

The Execution Lifecycle

Accessing Activity in Prisme.ai

Common Event Types

Debugging with Activity

Monitoring Performance with Activity

Optimizing Tool Execution

Parameter Optimization

Execution Efficiency

Error Reduction

Result Processing

Custom Event Emission

Real-Time Monitoring

Activity for Compliance and Audit

Audit Trails

Compliance Reporting

Forensic Analysis

Records Retention

Best Practices for Activity Utilization

Activity in AI Knowledge

Next Steps

Error Handling

AI Builder Event System

Getting Started

Prompting & RAG Agent

Tool-Using Agents

​The Value of Activity Tracking

Visibility

Diagnostics

Analytics

Logging

Monitoring

Optimization

​Understanding the Event System

​Key Components

Events

Publishers

Subscribers

Event Bus

​The Execution Lifecycle

​Accessing Activity in Prisme.ai

​Common Event Types

​Debugging with Activity

​Monitoring Performance with Activity

​Optimizing Tool Execution

Parameter Optimization

Execution Efficiency

Error Reduction

Result Processing

​Custom Event Emission

​Real-Time Monitoring

​Activity for Compliance and Audit

Audit Trails

Compliance Reporting

Forensic Analysis

Records Retention

​Best Practices for Activity Utilization

​Activity in AI Knowledge

​Next Steps

Error Handling

AI Builder Event System

The Value of Activity Tracking

Understanding the Event System

Key Components

The Execution Lifecycle

Accessing Activity in Prisme.ai

Common Event Types

Debugging with Activity

Monitoring Performance with Activity

Optimizing Tool Execution

Custom Event Emission

Real-Time Monitoring

Activity for Compliance and Audit

Best Practices for Activity Utilization

Activity in AI Knowledge

Next Steps