Testing & Debugging

Thorough testing and effective debugging are essential to creating reliable, high-quality AI applications. This guide provides comprehensive information on best practices, tools, and methodologies to validate functionality, identify issues, and ensure optimal performance.

​

Testing Approaches

​

Manual Testing

Interactive testing during development:

• Preview Mode: Test pages in real-time as you build
• Manual Execution: Run automations with custom inputs
• Event Simulation: Trigger events to test responses
• Interactive Debugging: Step through processes
• Visual Validation: Verify UI appearance and behavior

Manual testing is ideal during active development and for exploratory testing of new features.

​

Automated Testing

Systematic validation with predefined test cases:

• Test Sets: Collections of predefined scenarios
• Expected Results: Defined success criteria
• Batch Execution: Run multiple tests at once
• Regression Testing: Ensure changes don’t break existing functionality
• Scheduled Validation: Regular automated checks

Automated testing ensures consistent validation and is especially valuable for regression testing.

​

Integration Testing

Verify connections between components:

• End-to-End Testing: Full user journey validation
• API Testing: Validate external system connections
• Event Chain Testing: Verify event propagation
• Data Flow Validation: Check information passing between components
• System Integration: Test interaction with external systems

Integration testing ensures components work together correctly within the application ecosystem.

​

The Activity Log

The Activity log is your central dashboard for monitoring and debugging:

​

Working with the Activity Log

1. Accessing the Activity Log: Open the Activity section in your workspace.

   The Activity log is available through:

   • The main workspace navigation
   • Context links from errors
   • Links in automation execution views

2. Filtering Events: Focus on relevant information:

   Filtering options include:

   • Event Types: System events, custom events, errors
   • Time Range: Custom time periods or presets
   • Source: Specific components or external systems
   • Search: Text search across event details
   • Advanced Filters: Combine multiple criteria

3. Analyzing Event Sequences: Understand the flow of events:

   Sequence analysis features:

   • Correlation IDs: Connect related events
   • Timeline View: Visualize event order
   • Context Linking: Jump to related events
   • Parent-Child Relationships: See event hierarchies
   • Duration Indicators: Identify lengthy operations

4. Inspecting Event Details: Examine the specifics of individual events:

   Available details include:

   • Event Type and Name: What happened
   • Timestamp: When it occurred
   • Source: Which component triggered it
   • Payload: Complete data associated with the event
   • Metadata: Additional contextual information

​

Debugging Tools

AI Builder provides specialized tools for identifying and resolving issues:

​

Logging

Track execution and data state with logging:

• Log Instruction: Add explicit logging in automations using EDA
• Event Methods: Use log, warn, error
• Structured Logging: Log objects and complex data
• Log Levels: Use appropriate severity levels
• Contextual Information: Include relevant state information

   - emit:
      event: error
      payload: 
          level: "info",
          message: "Process completed",
          duration: processingTime,
          items: itemCount

Logs appear in the Activity view and can be filtered for easy access.

​

Error Diagnostics

Identify and resolve error conditions:

• Error Captures: Detailed information on failures
• Stack Traces: See where errors occurred
• Context Preservation: Examine state at error time
• Error Categorization: Distinguish different error types
• Suggested Solutions: Get guidance on fixing common errors

Error diagnostics help pinpoint the root cause of failures quickly.

​

Network Monitoring

Track API calls and network operations:

• Request Logging: See outgoing HTTP requests
• Response Tracking: View complete responses
• Timing Information: Measure request duration
• Headers and Authentication: Verify request configuration
• Error Detection: Identify failed requests

Network monitoring is essential for debugging integration issues.

​

Testing UI Components

Validate Block and Page functionality:

​

Component Preview

Test UI components in isolation:

The preview environment allows you to:

• See the component as users will
• Try different property configurations
• Test interactions
• View responsive behavior
• Check accessibility features

​

Interactive Testing

Verify component behavior with user interactions:

Interactive testing includes:

• Clicking buttons and controls
• Entering form data
• Triggering events
• Checking state changes
• Verifying visual feedback

​

Event Verification

Ensure events are emitted and handled correctly:

Event testing approaches:

• Trigger events manually
• Verify event payloads
• Check event receivers respond appropriately
• Test event chains
• Validate error handling

​

Responsive Testing

Verify components work across device sizes:

Responsive testing tools:

• Device size presets
• Custom dimension settings
• Orientation switching
• Behavior validation at breakpoints
• Touch interaction simulation

​

Testing Automations

Validate backend logic and processes:

​

Manual Execution

Run automations on demand with test inputs:

• Input Configuration: Provide test data
• Execution Controls: Start, and resume
• Output Inspection: Check results

Manual execution is ideal for development testing and detailed analysis.

​

Test Cases

Create and run predefined test scenarios using postman or similar:

• Test Definition: Specify inputs and expected outputs
• Validation Rules: Define success criteria
• Batch Running: Execute multiple tests
• Results Comparison: Actual vs. expected
• History Tracking: Monitor test stability over time

Test cases provide structured validation for complex logic.

​

Performance Testing

For performance testing of automations, the following approach can be used:

​

Load Testing with Artillery

The k6/Artillery tool can be used for performance testing:

import http from 'k6/http';
import { check, group, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '5m', target: 100 }, // simulate ramp-up of traffic from 1 to 100 users over 5 minutes.
    { duration: '10m', target: 100 }, // stay at 100 users for 10 minutes
    { duration: '5m', target: 0 }, // ramp-down to 0 users
  ],
  thresholds: {
    'http_req_duration': ['p(99)<1500'], // 99% of requests must complete below 1.5s
    'webhook run successfully': ['p(99)<1500'], // 99% of requests must complete below 1.5s
  },
};

// Define your test scenario here
export default () => {
  const webhookRes = http.post(`${BASE_URL}/v2/workspaces/${WORKSPACE_ID}/webhooks/Automation`, {
    foo: "too"
  });

  check(webhookRes, {
    'webhook run successfully': (resp) => resp.status === 200,
    'is body present': (resp) => resp.json().hasOwnProperty('body'),
  });

  sleep(1);
};

​

Test Scenarios

When conducting performance tests, consider these scenarios:

1. Single Instance Test: 100 concurrent users during 10 minutes, with 1 runtime/events/api-gateway instance
2. Multiple Instance Test: 100 concurrent users during 10 minutes, with 2 runtime/events/api-gateway instances

These tests can provide insights into the scalability of your application.

​

Testing with WebSockets

When testing applications that use WebSockets, consider these factors:

• When using HTTP fallback with multiple prismeai-events instances, sticky sessions are required
• For testing with WebSockets, anonymous sessions are automatically generated for each Virtual User
• The following user variables are defined by artillery-prismeai-plugins:
  • userId: Current anonymous userId
  • sessionId: Current anonymous sessionId
  • token: Current anonymous token, used for HTTP requests

Please refer to https://gitlab.com/prisme.ai/prismeai-dsul-loadtest.

​

Interpreting Results

Performance test results provide various metrics:

• engine.socketio.emit: Total number of events emitted during the test
• engine.socketio.emit_rate: Average rate of emitted events
• engine.socketio.rcv: Total number of events received during the test
• engine.socketio.rcv_rate: Average rate of received events
• prismeai.automations_execution: Automations execution rate
• prismeai.automations_execution.AUTOMATION_SLUG: Execution time histogram for each automation
• prismeai.emits_ack: Time taken to receive acknowledgments for emitted events
• prismeai.waits.EVENT_TYPE: Time taken to receive each event type
• vusers.completed: Number of successful Virtual User completions
• vusers.failed: Number of failed Virtual Users
• vusers.session_length: Time taken to complete entire scenarios

​

Debugging Common Issues

Solutions for frequently encountered problems:

​

Event Communication Issues

Troubleshoot event-related problems:

• Missing Events: Events not being emitted
• Event Payload Issues: Incorrect or missing data
• Listener Problems: Events not being received
• Timing Issues: Events occurring in unexpected order
• Scope Limitations: Events not visible across contexts

Debugging approach:

1. Verify events are being emitted using Activity log
2. Check event name spelling and case sensitivity
3. Examine event payload for expected data
4. Confirm listeners are configured correctly
5. Test with simplified event flows to isolate issues

​

Integration Failures

Address problems with external system connections:

• Authentication Failures: Invalid credentials or tokens
• Endpoint Issues: Incorrect URLs or unavailable services
• Request Format Problems: Improperly structured requests
• Response Handling Errors: Unexpected response formats
• Rate Limiting: Exceeded usage limits

Debugging approach:

1. Check connection parameters and credentials
2. Verify endpoint URLs and availability
3. Test API calls with external tools (e.g., Postman)
4. Examine request and response logs
5. Implement proper error handling for integration points

​

Performance Bottlenecks

Identify and resolve performance issues:

• Long-Running Operations: Processes taking too long
• Resource Consumption: Excessive memory or CPU usage
• Concurrency Problems: Issues with parallel execution
• Network Latency: Slow external connections
• Data Volume Challenges: Performance with large datasets

Debugging approach:

1. Add timing logs to measure operation duration
2. Identify patterns in slow performance
3. Optimize resource-intensive operations
4. Implement caching for repeated operations
5. Consider asynchronous processing for long-running tasks

​

UI Rendering Issues

Fix problems with user interface display:

• Layout Problems: Components not positioned correctly
• Styling Inconsistencies: Visual appearance issues
• Responsiveness Failures: Issues on different device sizes
• Update Timing: UI not reflecting data changes
• Animation Glitches: Visual transitions not working properly

Debugging approach:

1. Use browser developer tools to inspect elements
2. Check CSS styles and conflicts
3. Test across multiple device sizes and browsers
4. Verify data binding is working correctly
5. Simplify complex layouts to isolate issues

​

Automated Monitoring

Set up continuous monitoring for production applications:

​

Health Checks

Monitor system health automatically:

• Availability monitoring
• Response time tracking
• Resource utilization checks
• Dependency status verification
• Scheduled validation tests

​

Alerts & Notifications

Get notified of important issues:

• Error rate thresholds
• Performance degradation alerts
• Availability incidents
• Security event notifications
• Custom business metric alerts

​

Usage Analytics

Track how your application is being used:

• User activity patterns
• Feature utilization
• Performance trends
• Error patterns
• Resource consumption

​

Logging Pipelines

Centralize and analyze logs:

• Log aggregation
• Structured logging
• Log retention policies
• Search and analysis
• Log-based alerting

​

Testing Best Practices

​

Test Early and Often

Integrate testing throughout development:

• Test during development, not just at the end
• Create tests as you build features
• Automate tests for continuous validation
• Run tests before committing changes
• Make testing part of the development workflow

​

Create Representative Test Cases

Ensure tests reflect real-world usage:

• Cover common user scenarios
• Include edge cases and error conditions
• Test with realistic data volumes
• Validate against actual user requirements
• Consider different user roles and permissions

​

Isolate Components

Test units independently:

• Test blocks in isolation before page integration
• Validate individual automations separately
• Use mocks for external dependencies
• Focus on specific functionality in each test
• Create clear test boundaries

​

Maintain Test Documentation

Document your testing approach:

• Describe test purpose and scope
• Document test prerequisites
• Explain expected outcomes
• Maintain test data sets
• Record testing decisions and rationale

​

Use Version Control

Track test changes alongside code:

• Version control test scripts and configurations
• Link tests to specific application versions
• Document test evolution
• Manage test environment configurations
• Enable collaborative test development

​

Automate Regression Testing

Ensure changes don’t break existing functionality:

• Create automated test suites
• Run regression tests after changes
• Maintain a core set of smoke tests
• Prioritize tests for critical functionality
• Analyze test results systematically

​

Debugging Best Practices

​

Systematic Approach

Use methodical debugging techniques:

• Define the problem clearly
• Isolate the issue with minimal test cases
• Form and test hypotheses
• Make one change at a time
• Document your findings

​

Effective Logging

Implement informative logging:

• Log meaningful context, not just values
• Use appropriate log levels
• Include timestamps and correlation IDs
• Avoid logging sensitive information
• Structure logs for easy analysis

​

Use Available Tools

Leverage debugging tools effectively:

• Master the Activity log filtering
• Monitor network requests for integration issues
• Inspect variables at key points
• Utilize specialized tools for specific problems

​

Root Cause Analysis

Find underlying issues, not just symptoms:

• Ask “why” multiple times
• Look for patterns across failures
• Consider system-wide factors
• Test boundary conditions
• Validate assumptions

​

Collaborative Debugging

Work together to solve complex problems:

• Share debugging information clearly
• Document reproduction steps
• Discuss alternative hypotheses
• Use pair debugging for complex issues
• Learn from each other’s approaches

​

Preventive Debugging

Avoid future problems:

• Add assertions for expected conditions
• Implement defensive programming
• Create monitors for early warning
• Document known issues and solutions
• Share debugging knowledge

​

Testing Different Components

​

Testing Automations

For automations, you can use Postman or similar tools to test your API endpoints. Since blocks are already tested in the codebase at Prisme.ai, you only need to configure them with the appropriate data.

​

Testing Pages

For pages, use the same UI end-to-end testing tools you would use for web applications. This ensures that the user interface works correctly across different browsers and devices.

​

Scaling Considerations

When scaling applications that use WebSockets:

• To enable WebSockets with HTTP fallback and multiple instances, HTTP traffic must be forwarded with sticky sessions
• Without sticky sessions, users may experience HTTP 400 errors with “Session ID unknown” messages
• You can either disable HTTP fallback to force WebSocket establishment or configure sticky sessions

Example Istio configuration for sticky sessions:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: events-sticky-session
spec:
  host: prismeai-events-svc
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpHeaderName: 'user-agent' # Identify same users with user agent header

​

Rate Limiting Considerations

For performance testing, you may need to disable or increase runtime rate limits to avoid throttling:

RATE_LIMIT_DISABLED=true

Refer to the architecture documentation for more details on rate limits.