Skip to main content
Understanding how tools execute and monitoring their performance is critical for developing reliable tool-using agents. Prisme.ai provides powerful Activity tracking capabilities that give you visibility into every aspect of tool execution, from request processing to result delivery.

​
The Value of Activity Tracking

Activity tracking in Prisme.ai provides:

Visibility

Complete transparency into how tools execute

Diagnostics

Powerful debugging for error resolution

Analytics

Performance metrics and usage patterns

Logging

Comprehensive audit trails of all operations

Monitoring

Real-time observation of system behavior

Optimization

Data-driven insights for performance improvements

​
Understanding the Event System

At the core of Prisme.ai’s Activity tracking is its event-driven architecture:

​
Key Components

Events

Structured messages representing actions, status changes, or notificationsKey attributes:
  • Event type
  • Timestamp
  • Source identifier
  • Payload data
  • Request context

Publishers

Components that emit events when actions occurExamples:
  • Tool execution requests
  • Automation processes
  • User interactions
  • System status changes

Subscribers

Components that listen for and process specific eventsExamples:
  • Automation workflows
  • Activity monitors
  • Logging systems
  • Analytics processors

Event Bus

The central message broker that routes eventsResponsibilities:
  • Reliable message delivery
  • Event persistence
  • Publisher/subscriber coordination
  • Load management

​
The Execution Lifecycle

Tool execution follows a well-defined lifecycle that generates events at each stage:
1

Request Initiation

The process begins when an agent decides to use a tool.Events generated:
  • tool.request.initiated
  • tool.parameter.validation.started
Key information:
  • Tool name and parameters
  • Request source (conversation ID, user ID)
  • Timestamp and request ID
  • Context information
2

Parameter Validation

The system validates that all required parameters are present and properly formatted.Events generated:
  • tool.parameter.validation.completed
  • tool.parameter.validation.failed (if issues occur)
Key information:
  • Validation status
  • Error details (if any)
  • Parameter transformations
  • Schema conformance
3

Execution

The actual tool code runs, performing its designated function.Events generated:
  • tool.execution.started
  • tool.execution.step.completed (for multi-step tools)
  • tool.execution.completed
Key information:
  • Execution duration
  • Resource utilization
  • Intermediate state
  • External system interactions
4

Result Processing

The system processes the tool’s output for agent consumption.Events generated:
  • tool.result.processing.started
  • tool.result.processing.completed
Key information:
  • Raw output data
  • Processed results
  • Formatting operations
  • Output size and structure
5

Response Delivery

The formatted results are returned to the agent.Events generated:
  • tool.response.delivery.started
  • tool.response.delivery.completed
Key information:
  • Final response format
  • Delivery time
  • End-to-end latency
  • Response size

​
Accessing Activity in Prisme.ai

You can monitor and analyze tool execution through several interfaces:
  • Activity View
  • Event API
  • Dashboard Visualizations
The primary interface for monitoring and debugging tool execution.Key features:
  • Chronological event listing
  • Filtering by event type, status, or date range
  • Detailed event inspection
  • Real-time monitoring
  • Search capabilities
Access:
  • Navigate to the “Activity” tab in your AI Builder workspace
  • Use filtering to focus on relevant events
  • Click on events to see detailed information
  • Toggle real-time updates for live monitoring
Screenshot2025 04 12at23 46 14 Pn

​
Common Event Types

Prisme.ai’s event system uses a hierarchical naming convention for clarity and filtering:
Events related to the initial tool request and setup.Key event types:
  • tool.request.initiated: Initial request for tool execution
  • tool.request.validated: Request passed preliminary validation
  • tool.request.rejected: Request failed preliminary checks
Example event data:
{
  "type": "tool.request.initiated",
  "timestamp": "2025-03-15T14:32:56.789Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "parameters": {
      "symbol": "AAPL",
      "dateRange": "5d"
    },
    "requester": {
      "conversationId": "conv-789012",
      "userId": "user-345678"
    }
  }
}
Events related to the actual tool execution process.Key event types:
  • tool.execution.started: Tool execution has begun
  • tool.execution.step.completed: A step in multi-step execution completed
  • tool.execution.completed: Tool execution finished successfully
  • tool.execution.failed: Tool execution encountered an error
Example event data:
{
  "type": "tool.execution.completed",
  "timestamp": "2025-03-15T14:33:01.456Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "duration": 4667,
    "resourceUsage": {
      "cpuTime": 1250,
      "memoryPeak": 256384
    },
    "externalCalls": [
      {
        "service": "finance-api",
        "endpoint": "/stock/price",
        "duration": 3890
      }
    ]
  }
}
Events related to the processing and delivery of tool results.Key event types:
  • tool.result.generated: Raw results from tool execution
  • tool.result.processed: Results after formatting and preparation
  • tool.result.delivered: Final delivery to the requester
Example event data:
{
  "type": "tool.result.processed",
  "timestamp": "2025-03-15T14:33:02.123Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "rawResultSize": 8712,
    "processedResultSize": 2340,
    "resultSummary": {
      "symbol": "AAPL",
      "currentPrice": 187.43,
      "priceChange": 2.31,
      "percentChange": 1.25,
      "dataPoints": 5
    }
  }
}
Events related to errors during the tool execution lifecycle.Key event types:
  • tool.error.validation: Error during parameter validation
  • tool.error.execution: Error during tool execution
  • tool.error.timeout: Execution exceeded allocated time
  • tool.error.resource: Resource limit exceeded
Example event data:
{
  "type": "tool.error.execution",
  "timestamp": "2025-03-15T14:33:00.789Z",
  "source": "workspace:finance-assistant",
  "data": {
    "toolName": "StockPriceChecker",
    "requestId": "req-123456",
    "errorCode": "API_UNAVAILABLE",
    "errorMessage": "External finance API returned status 503",
    "errorDetails": {
      "httpStatus": 503,
      "responseBody": "Service temporarily unavailable",
      "endpoint": "https://finance-api.example.com/stock/price"
    },
    "executionStage": "external_api_call"
  }
}

​
Debugging with Activity

Activity tracking provides powerful capabilities for diagnosing and resolving issues:
1

Identify the Issue

Use filtering to locate relevant events.Effective filtering strategies:
  • Search for specific request IDs
  • Filter by error event types
  • Focus on a specific time range
  • Filter by tool name or source
  • Search for specific error codes
2

Trace Execution Flow

Follow the sequence of events to understand the execution path.Key analysis techniques:
  • Map the complete event sequence
  • Identify missing or unexpected events
  • Analyze timing between events
  • Examine state changes across events
  • Look for execution branches
3

Examine Event Details

Inspect detailed event data for diagnostic information.Focus areas:
  • Parameter values and transformations
  • Error details and context
  • Performance metrics
  • External system interactions
  • Resource utilization
4

Compare with Successful Executions

Analyze differences between failed and successful runs.Comparison strategies:
  • Identify parameter differences
  • Compare timing patterns
  • Analyze resource usage
  • Examine external factors
  • Look for environmental changes
5

Implement and Verify Fixes

Make changes and confirm resolution through activity monitoring.Verification approach:
  • Test the same parameters
  • Monitor in real-time
  • Verify complete event sequence
  • Check performance metrics
  • Test edge cases

​
Monitoring Performance with Activity

Activity data is invaluable for performance optimization:
Identify and address performance bottlenecks.Key metrics to monitor:
  • End-to-end latency: Total time from request to response
  • Execution time: Time spent in actual tool execution
  • External call latency: Time spent waiting for external services
  • Processing overhead: Time spent in non-execution activities
Implementation approach:
// Calculate average latencies by tool
function analyzeToolLatencies(events, timeRange) {
  const executions = {};
  
  // Group events by request ID
  events.forEach(event => {
    if (!executions[event.data.requestId]) {
      executions[event.data.requestId] = {
        tool: event.data.toolName,
        events: []
      };
    }
    executions[event.data.requestId].events.push(event);
  });
  
  // Calculate metrics for each execution
  const toolMetrics = {};
  Object.values(executions).forEach(execution => {
    const startEvent = execution.events.find(e => e.type === 'tool.request.initiated');
    const endEvent = execution.events.find(e => e.type === 'tool.result.delivered');
    const execStartEvent = execution.events.find(e => e.type === 'tool.execution.started');
    const execEndEvent = execution.events.find(e => e.type === 'tool.execution.completed');
    
    if (startEvent && endEvent && execStartEvent && execEndEvent) {
      const tool = execution.tool;
      if (!toolMetrics[tool]) {
        toolMetrics[tool] = {
          count: 0,
          totalLatency: 0,
          totalExecTime: 0,
          totalOverhead: 0
        };
      }
      
      const latency = endEvent.timestamp - startEvent.timestamp;
      const execTime = execEndEvent.timestamp - execStartEvent.timestamp;
      const overhead = latency - execTime;
      
      toolMetrics[tool].count++;
      toolMetrics[tool].totalLatency += latency;
      toolMetrics[tool].totalExecTime += execTime;
      toolMetrics[tool].totalOverhead += overhead;
    }
  });
  
  // Calculate averages
  Object.keys(toolMetrics).forEach(tool => {
    const metrics = toolMetrics[tool];
    metrics.avgLatency = metrics.totalLatency / metrics.count;
    metrics.avgExecTime = metrics.totalExecTime / metrics.count;
    metrics.avgOverhead = metrics.totalOverhead / metrics.count;
    metrics.overheadPercentage = (metrics.avgOverhead / metrics.avgLatency) * 100;
  });
  
  return toolMetrics;
}
Monitor and minimize tool failures.Key metrics to track:
  • Overall error rate: Percentage of tool executions that fail
  • Error breakdown by type: Distribution of different error categories
  • Error trends: Changes in error rates over time
  • Correlation analysis: Factors associated with higher error rates
Implementation approach:
// Calculate error rates by tool and type
function analyzeErrorRates(events, timeRange) {
  const toolStats = {};
  
  // Count total executions and errors
  events.forEach(event => {
    const tool = event.data.toolName;
    
    if (!toolStats[tool]) {
      toolStats[tool] = {
        totalExecutions: 0,
        errors: {
          total: 0,
          byType: {}
        }
      };
    }
    
    if (event.type === 'tool.execution.started') {
      toolStats[tool].totalExecutions++;
    }
    
    if (event.type.startsWith('tool.error.')) {
      toolStats[tool].errors.total++;
      
      const errorType = event.type.replace('tool.error.', '');
      if (!toolStats[tool].errors.byType[errorType]) {
        toolStats[tool].errors.byType[errorType] = 0;
      }
      toolStats[tool].errors.byType[errorType]++;
    }
  });
  
  // Calculate rates
  Object.keys(toolStats).forEach(tool => {
    const stats = toolStats[tool];
    stats.errorRate = stats.errors.total / stats.totalExecutions;
    
    stats.errors.typeBreakdown = {};
    Object.keys(stats.errors.byType).forEach(errorType => {
      stats.errors.typeBreakdown[errorType] = 
        stats.errors.byType[errorType] / stats.errors.total;
    });
  });
  
  return toolStats;
}
Understand how tools are being used.Key metrics to analyze:
  • Usage frequency: How often each tool is called
  • Parameter distributions: Common values and patterns
  • Usage contexts: When and why tools are invoked
  • User segmentation: Differences in tool usage across users
Implementation approach:
// Analyze tool usage patterns
function analyzeUsagePatterns(events, timeRange) {
  const usageStats = {
    byTool: {},
    byHour: Array(24).fill(0),
    byUser: {},
    parameterDistributions: {}
  };
  
  events.forEach(event => {
    if (event.type === 'tool.request.initiated') {
      const tool = event.data.toolName;
      const timestamp = new Date(event.timestamp);
      const hour = timestamp.getHours();
      const userId = event.data.requester?.userId || 'unknown';
      const parameters = event.data.parameters || {};
      
      // Count by tool
      if (!usageStats.byTool[tool]) {
        usageStats.byTool[tool] = 0;
      }
      usageStats.byTool[tool]++;
      
      // Count by hour
      usageStats.byHour[hour]++;
      
      // Count by user
      if (!usageStats.byUser[userId]) {
        usageStats.byUser[userId] = {
          total: 0,
          byTool: {}
        };
      }
      usageStats.byUser[userId].total++;
      if (!usageStats.byUser[userId].byTool[tool]) {
        usageStats.byUser[userId].byTool[tool] = 0;
      }
      usageStats.byUser[userId].byTool[tool]++;
      
      // Analyze parameters
      if (!usageStats.parameterDistributions[tool]) {
        usageStats.parameterDistributions[tool] = {};
      }
      Object.keys(parameters).forEach(param => {
        if (!usageStats.parameterDistributions[tool][param]) {
          usageStats.parameterDistributions[tool][param] = {};
        }
        
        const value = String(parameters[param]);
        if (!usageStats.parameterDistributions[tool][param][value]) {
          usageStats.parameterDistributions[tool][param][value] = 0;
        }
        usageStats.parameterDistributions[tool][param][value]++;
      });
    }
  });
  
  return usageStats;
}

​
Optimizing Tool Execution

Use Activity data to drive execution optimizations:

Parameter Optimization

Refine parameter handling based on usage patternsStrategies:
  • Add intelligent defaults for common values
  • Implement parameter validation improvements
  • Optimize parameter extraction from user inputs
  • Add preprocessing for frequently used parameter formats

Execution Efficiency

Improve tool execution performanceStrategies:
  • Identify and optimize slow execution steps
  • Implement caching for frequent operations
  • Reduce external API latency
  • Optimize resource utilization

Error Reduction

Minimize tool execution failuresStrategies:
  • Add robust error handling for common failures
  • Implement automatic retries for transient issues
  • Improve validation to catch problems early
  • Create fallback mechanisms for critical functions

Result Processing

Enhance result formatting and deliveryStrategies:
  • Optimize result formatting for LLM consumption
  • Reduce result size for better context utilization
  • Implement progressive result delivery for long operations
  • Add enrichment for more valuable outputs

​
Custom Event Emission

You can emit custom events to enhance visibility into your tools:
Define your own event types for specialized monitoring.Naming conventions:
custom.[workspace].[category].[action]
Examples:
  • custom.finance-assistant.calculation.started
  • custom.support-agent.ticket.created
  • custom.inventory-tool.stock.checked
This allows for clear organization and filtering of your custom events.
Add custom event emission to your tools.Example implementation:
slug: product-recommendation-tool
do:
  # Emit custom event to track recommendation request
  - emit:
      event: custom.product-tool.recommendation.requested
      data:
        userId: '{{event.data.userId}}'
        productCategory: '{{event.data.parameters.category}}'
        priceRange: '{{event.data.parameters.priceRange}}'
        timestamp: '{{now()}}'
  
  # Execute recommendation logic
  - ProductCatalog.search:
      category: '{{event.data.parameters.category}}'
      priceMin: '{{event.data.parameters.priceRange.min}}'
      priceMax: '{{event.data.parameters.priceRange.max}}'
      output: searchResults
  
  # Emit event for search completion
  - emit:
      event: custom.product-tool.search.completed
      data:
        resultsCount: '{{length(searchResults.products)}}'
        executionTime: '{{searchResults.executionTime}}'
  
  # Apply recommendation algorithm
  - RecommendationEngine.process:
      products: '{{searchResults.products}}'
      userPreferences: '{{event.data.parameters.preferences}}'
      output: recommendations
  
  # Emit final recommendation event
  - emit:
      event: custom.product-tool.recommendation.generated
      data:
        recommendationCount: '{{length(recommendations)}}'
        topRecommendationId: '{{recommendations[0].id}}'
        algorithmsApplied: '{{recommendations.algorithmsApplied}}'
  
  # Return results
  - set:
      name: output
      value:
        recommendations: '{{recommendations}}'
This approach:
  • Provides detailed visibility into tool execution stages
  • Captures business-specific metrics
  • Enables more nuanced performance analysis
  • Creates richer debugging context
Design effective event payloads for maximum utility.Recommendations:
  • Include identifiers: Always add request IDs, user IDs, and other correlation identifiers
  • Add timestamps: Include timing information for performance analysis
  • Provide context: Add relevant business context to make events meaningful
  • Size appropriately: Include useful data without making events too large
  • Structure consistently: Use consistent naming and structure across related events
Example event structure:
{
  "type": "custom.order-tool.validation.completed",
  "timestamp": "2025-03-15T15:45:12.678Z",
  "source": "workspace:commerce-assistant",
  "data": {
    "requestId": "req-abc123",
    "orderId": "ORD-98765",
    "validationResults": {
      "inventory": {
        "status": "passed",
        "duration": 124
      },
      "pricing": {
        "status": "passed",
        "duration": 86
      },
      "shipping": {
        "status": "warning",
        "message": "Delivery may be delayed",
        "duration": 231
      }
    },
    "totalDuration": 441,
    "validationPassed": true
  }
}

​
Real-Time Monitoring

For critical tools, implement real-time monitoring using Activity:
1

Set Up Real-Time Dashboards

Create specialized dashboards for key metrics.Include dashboards for:
  • Error rates and types
  • Execution latency
  • Usage volume
  • Resource utilization
  • Business-specific metrics
2

Configure Alerts

Define thresholds and conditions for notifications.Alert types to consider:
  • Error rate spikes
  • Latency threshold violations
  • Abnormal usage patterns
  • Resource exhaustion warnings
  • Custom business metric anomalies
3

Implement Event Subscribers

Create subscribers for real-time processing.Example implementation:
slug: error-rate-monitor
when:
  event: tool.error.*
do:
  # Increment error counter
  - cache.increment:
      key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
      value: 1
      expire: 86400  # 24 hours
  
  # Get current hour's execution count
  - cache.get:
      key: 'executions:{{date(now(), "YYYY-MM-DD:HH")}}'
      output: executionCount
  
  # Get current hour's error count
  - cache.get:
      key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
      output: errorCount
  
  # Calculate error rate
  - set:
      name: errorRate
      value: '{{errorCount / executionCount}}'
  
  # Check if threshold exceeded
  - conditions:
      '{{errorRate > 0.05 && executionCount > 10}}':  # 5% error rate with at least 10 executions
        - notification.send:
            channel: 'slack'
            message: 'Alert: Tool error rate at {{errorRate * 100}}% ({{errorCount}} errors out of {{executionCount}} executions) in the past hour'
4

Monitor Service Health

Track overall system performance and health.Focus areas:
  • API response times
  • Service availability
  • Resource utilization
  • Queue lengths
  • Error distributions

​
Activity for Compliance and Audit

Beyond operational uses, Activity provides essential compliance capabilities:

Audit Trails

Maintain comprehensive records of all tool interactionsKey aspects:
  • Complete chronological record of events
  • User and system actions
  • Parameter and result tracking
  • Timing information
  • Access evidence

Compliance Reporting

Generate reports for regulatory requirementsCapabilities:
  • Data access audit reports
  • Processing evidence
  • Security monitoring
  • Policy enforcement verification
  • Chain of custody documentation

Forensic Analysis

Detailed investigation of specific incidentsUse cases:
  • Security incident investigation
  • Error root cause analysis
  • User behavior auditing
  • System interaction reconstruction
  • Performance issue diagnostics

Records Retention

Maintain activity data according to policiesConsiderations:
  • Retention period configuration
  • Data archiving strategies
  • Privacy compliance
  • Data minimization
  • Access controls

​
Best Practices for Activity Utilization

Plan your event ecosystem for maximum value.Recommendations:
  • Define an event taxonomy: Create a consistent naming and categorization system
  • Identify key lifecycle events: Ensure all important state transitions emit events
  • Balance detail and volume: Emit enough events for visibility without overwhelming storage
  • Consider event consumers: Design with reporting, monitoring, and analysis needs in mind
  • Document event types: Maintain clear documentation of all events and their meanings
Implementation approach:
Event Naming Structure:

[domain].[entity].[action]

Examples:
- tool.request.initiated
- order.payment.processed
- user.profile.updated
- system.resource.allocated
Create filtering strategies for efficient data access.Recommendations:
  • Use consistent metadata: Add standard fields like request IDs to enable correlation
  • Include categorization data: Add tags, types, and categories for efficient filtering
  • Consider time-based access: Optimize for common time-range queries
  • Balance filter granularity: Too granular filters can be as problematic as too broad ones
  • Test filter performance: Ensure filtering is efficient at production scale
Example filtering approach:
// Effective event filtering
const events = await activity.queryEvents({
  // Time range
  timeRange: {
    start: new Date(Date.now() - (7 * 24 * 60 * 60 * 1000)), // 7 days ago
    end: new Date()
  },
  // Event types (using hierarchical pattern)
  eventTypes: ['tool.execution.*', 'tool.error.*'],
  // Source filter
  sources: ['workspace:customer-service'],
  // Metadata filters
  filters: [
    { field: 'data.requestId', operator: 'eq', value: 'req-123456' },
    { field: 'data.toolName', operator: 'in', value: ['CustomerLookup', 'OrderStatus'] }
  ],
  // Sorting
  sort: { field: 'timestamp', direction: 'desc' },
  // Pagination
  limit: 100,
  offset: 0
});
Ensure your activity management scales with usage.Recommendations:
  • Implement data lifecycle management: Archive older events to control storage growth
  • Consider sampling for high-volume events: For very frequent events, consider sampling
  • Optimize query patterns: Design queries to use available indexes
  • Leverage aggregation: Use pre-aggregated metrics for frequent analyses
  • Plan for growth: Design with future scale in mind
Example lifecycle implementation:
slug: activity-lifecycle-management
when:
  schedule: '0 0 * * *'  # Daily at midnight
do:
  # Archive events older than 30 days
  - activity.archiveEvents:
      olderThan: '{{date.add(now(), -30, "days")}}'
      destination: 'cold-storage'
  
  # Delete archived events older than 365 days
  - activity.deleteArchivedEvents:
      olderThan: '{{date.add(now(), -365, "days")}}'
  
  # Aggregate metrics for retention
  - activity.aggregateMetrics:
      timeRange:
        start: '{{date.add(now(), -2, "days")}}'
        end: '{{date.add(now(), -1, "days")}}'
      dimensions: ['toolName', 'hour', 'errorType']
      metrics: ['count', 'avgDuration', 'errorRate']
  
  # Emit completion event
  - emit:
      event: system.activity.lifecycle.completed
      data:
        timestamp: '{{now()}}'

​
Activity in AI Knowledge

For AI Knowledge users, Activity information is accessible through built-in interfaces:
1

Access the Activity Log

Find the Activity section in AI Knowledge.Key features:
  • Chronological event listing
  • Filtering by event type and date
  • Search capabilities
  • Drill-down for details
2

Monitor Tool Usage

Track how agents are using tools.Available information:
  • Tool usage frequency
  • Success and error rates
  • Performance metrics
  • Parameter patterns
3

Analyze Conversation Flows

Examine how tools integrate into conversations.Analysis capabilities:
  • Tool usage in conversation context
  • User reactions to tool results
  • Multi-turn tool interactions
  • Conversation paths and patterns
4

Export Activity Data

Extract data for external analysis.Export options:
  • CSV for spreadsheet analysis
  • JSON for programmatic processing
  • Filtered or complete data sets
  • Custom date ranges

​
Next Steps

Ready to implement effective execution monitoring and activity tracking? Continue with these resources:

Error Handling

Learn how to implement robust error management

AI Builder Event System

Explore the event system in greater depth
⌘I