Understanding how tools execute and monitoring their performance is critical for developing reliable tool-using agents. Prisme.ai provides powerful Activity tracking capabilities that give you visibility into every aspect of tool execution, from request processing to result delivery.

The Value of Activity Tracking

Activity tracking in Prisme.ai provides:

Visibility

Complete transparency into how tools execute

Diagnostics

Powerful debugging for error resolution

Analytics

Performance metrics and usage patterns

Logging

Comprehensive audit trails of all operations

Monitoring

Real-time observation of system behavior

Optimization

Data-driven insights for performance improvements

Understanding the Event System

At the core of Prisme.ai’s Activity tracking is its event-driven architecture:

Key Components

Events

Structured messages representing actions, status changes, or notifications

Key attributes:

  • Event type
  • Timestamp
  • Source identifier
  • Payload data
  • Request context

Publishers

Components that emit events when actions occur

Examples:

  • Tool execution requests
  • Automation processes
  • User interactions
  • System status changes

Subscribers

Components that listen for and process specific events

Examples:

  • Automation workflows
  • Activity monitors
  • Logging systems
  • Analytics processors

Event Bus

The central message broker that routes events

Responsibilities:

  • Reliable message delivery
  • Event persistence
  • Publisher/subscriber coordination
  • Load management

The Execution Lifecycle

Tool execution follows a well-defined lifecycle that generates events at each stage:

1

Request Initiation

The process begins when an agent decides to use a tool.

Events generated:

  • tool.request.initiated
  • tool.parameter.validation.started

Key information:

  • Tool name and parameters
  • Request source (conversation ID, user ID)
  • Timestamp and request ID
  • Context information
2

Parameter Validation

The system validates that all required parameters are present and properly formatted.

Events generated:

  • tool.parameter.validation.completed
  • tool.parameter.validation.failed (if issues occur)

Key information:

  • Validation status
  • Error details (if any)
  • Parameter transformations
  • Schema conformance
3

Execution

The actual tool code runs, performing its designated function.

Events generated:

  • tool.execution.started
  • tool.execution.step.completed (for multi-step tools)
  • tool.execution.completed

Key information:

  • Execution duration
  • Resource utilization
  • Intermediate state
  • External system interactions
4

Result Processing

The system processes the tool’s output for agent consumption.

Events generated:

  • tool.result.processing.started
  • tool.result.processing.completed

Key information:

  • Raw output data
  • Processed results
  • Formatting operations
  • Output size and structure
5

Response Delivery

The formatted results are returned to the agent.

Events generated:

  • tool.response.delivery.started
  • tool.response.delivery.completed

Key information:

  • Final response format
  • Delivery time
  • End-to-end latency
  • Response size

Accessing Activity in Prisme.ai

You can monitor and analyze tool execution through several interfaces:

The primary interface for monitoring and debugging tool execution.

Key features:

  • Chronological event listing
  • Filtering by event type, status, or date range
  • Detailed event inspection
  • Real-time monitoring
  • Search capabilities

Access:

  • Navigate to the “Activity” tab in your AI Builder workspace
  • Use filtering to focus on relevant events
  • Click on events to see detailed information
  • Toggle real-time updates for live monitoring

Common Event Types

Prisme.ai’s event system uses a hierarchical naming convention for clarity and filtering:

Debugging with Activity

Activity tracking provides powerful capabilities for diagnosing and resolving issues:

1

Identify the Issue

Use filtering to locate relevant events.

Effective filtering strategies:

  • Search for specific request IDs
  • Filter by error event types
  • Focus on a specific time range
  • Filter by tool name or source
  • Search for specific error codes
2

Trace Execution Flow

Follow the sequence of events to understand the execution path.

Key analysis techniques:

  • Map the complete event sequence
  • Identify missing or unexpected events
  • Analyze timing between events
  • Examine state changes across events
  • Look for execution branches
3

Examine Event Details

Inspect detailed event data for diagnostic information.

Focus areas:

  • Parameter values and transformations
  • Error details and context
  • Performance metrics
  • External system interactions
  • Resource utilization
4

Compare with Successful Executions

Analyze differences between failed and successful runs.

Comparison strategies:

  • Identify parameter differences
  • Compare timing patterns
  • Analyze resource usage
  • Examine external factors
  • Look for environmental changes
5

Implement and Verify Fixes

Make changes and confirm resolution through activity monitoring.

Verification approach:

  • Test the same parameters
  • Monitor in real-time
  • Verify complete event sequence
  • Check performance metrics
  • Test edge cases

Monitoring Performance with Activity

Activity data is invaluable for performance optimization:

Optimizing Tool Execution

Use Activity data to drive execution optimizations:

Parameter Optimization

Refine parameter handling based on usage patterns

Strategies:

  • Add intelligent defaults for common values
  • Implement parameter validation improvements
  • Optimize parameter extraction from user inputs
  • Add preprocessing for frequently used parameter formats

Execution Efficiency

Improve tool execution performance

Strategies:

  • Identify and optimize slow execution steps
  • Implement caching for frequent operations
  • Reduce external API latency
  • Optimize resource utilization

Error Reduction

Minimize tool execution failures

Strategies:

  • Add robust error handling for common failures
  • Implement automatic retries for transient issues
  • Improve validation to catch problems early
  • Create fallback mechanisms for critical functions

Result Processing

Enhance result formatting and delivery

Strategies:

  • Optimize result formatting for LLM consumption
  • Reduce result size for better context utilization
  • Implement progressive result delivery for long operations
  • Add enrichment for more valuable outputs

Custom Event Emission

You can emit custom events to enhance visibility into your tools:

Real-Time Monitoring

For critical tools, implement real-time monitoring using Activity:

1

Set Up Real-Time Dashboards

Create specialized dashboards for key metrics.

Include dashboards for:

  • Error rates and types
  • Execution latency
  • Usage volume
  • Resource utilization
  • Business-specific metrics
2

Configure Alerts

Define thresholds and conditions for notifications.

Alert types to consider:

  • Error rate spikes
  • Latency threshold violations
  • Abnormal usage patterns
  • Resource exhaustion warnings
  • Custom business metric anomalies
3

Implement Event Subscribers

Create subscribers for real-time processing.

Example implementation:

slug: error-rate-monitor
when:
  event: tool.error.*
do:
  # Increment error counter
  - cache.increment:
      key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
      value: 1
      expire: 86400  # 24 hours
  
  # Get current hour's execution count
  - cache.get:
      key: 'executions:{{date(now(), "YYYY-MM-DD:HH")}}'
      output: executionCount
  
  # Get current hour's error count
  - cache.get:
      key: 'error:{{date(now(), "YYYY-MM-DD:HH")}}'
      output: errorCount
  
  # Calculate error rate
  - set:
      name: errorRate
      value: '{{errorCount / executionCount}}'
  
  # Check if threshold exceeded
  - conditions:
      '{{errorRate > 0.05 && executionCount > 10}}':  # 5% error rate with at least 10 executions
        - notification.send:
            channel: 'slack'
            message: 'Alert: Tool error rate at {{errorRate * 100}}% ({{errorCount}} errors out of {{executionCount}} executions) in the past hour'
4

Monitor Service Health

Track overall system performance and health.

Focus areas:

  • API response times
  • Service availability
  • Resource utilization
  • Queue lengths
  • Error distributions

Activity for Compliance and Audit

Beyond operational uses, Activity provides essential compliance capabilities:

Audit Trails

Maintain comprehensive records of all tool interactions

Key aspects:

  • Complete chronological record of events
  • User and system actions
  • Parameter and result tracking
  • Timing information
  • Access evidence

Compliance Reporting

Generate reports for regulatory requirements

Capabilities:

  • Data access audit reports
  • Processing evidence
  • Security monitoring
  • Policy enforcement verification
  • Chain of custody documentation

Forensic Analysis

Detailed investigation of specific incidents

Use cases:

  • Security incident investigation
  • Error root cause analysis
  • User behavior auditing
  • System interaction reconstruction
  • Performance issue diagnostics

Records Retention

Maintain activity data according to policies

Considerations:

  • Retention period configuration
  • Data archiving strategies
  • Privacy compliance
  • Data minimization
  • Access controls

Best Practices for Activity Utilization

Activity in AI Knowledge

For AI Knowledge users, Activity information is accessible through built-in interfaces:

1

Access the Activity Log

Find the Activity section in AI Knowledge.

Key features:

  • Chronological event listing
  • Filtering by event type and date
  • Search capabilities
  • Drill-down for details
2

Monitor Tool Usage

Track how agents are using tools.

Available information:

  • Tool usage frequency
  • Success and error rates
  • Performance metrics
  • Parameter patterns
3

Analyze Conversation Flows

Examine how tools integrate into conversations.

Analysis capabilities:

  • Tool usage in conversation context
  • User reactions to tool results
  • Multi-turn tool interactions
  • Conversation paths and patterns
4

Export Activity Data

Extract data for external analysis.

Export options:

  • CSV for spreadsheet analysis
  • JSON for programmatic processing
  • Filtered or complete data sets
  • Custom date ranges

Next Steps

Ready to implement effective execution monitoring and activity tracking? Continue with these resources: