Performance

To test your application's performance, you can use the performance object available in the context.

const slowPlugin: Promise<MessageResponse> = {
  name: 'slow-plugin',
  middleware: async (context, next): Promise<MessageResponse> => {

    context.performance.markStart("SlowPluginPerformance");
    await slowFunction();
    context.performance.markEnd("SlowPluginPerformance");

    // Continue middleware chain
    return next();
  },
};

After collecting your performance data, you can access it through the same performance object. Performance tracking requires all processes to complete, so it uses effect instead of middleware, as it runs after the response is finalized.

const analyticsPlugin: Promise<MessageResponse> = {
  name: 'analytics',
  effects: [analyticsEffect]
};

async function analyticsEffect(context: RequestContext, response: MessageResponse): Promise<void> {
  console.log(context.performance.getMeasureTotal("SlowPluginPerformance"))
}

Measures vs Marks

This concept is inspired by the Web Performance API. Marks are essentially named sequences that the performance tool uses to measure execution time. For instance, if you have a tool for your AI and want to evaluate its performance, you might find it triggered multiple times by the AI. Therefore, a single mark can be part of multiple measures. A measure is constructed using two marks: start and end.

INFO

You can also access all marks and measures using getMarks and getMeasures

Default measures

Byorg automatically gathers performance data. Middleware measures are collected in two separate phases: before handling the response and after it.

export const PerformanceMarks = {
  processMessages: 'processMessages',
  middlewareBeforeHandler: 'middleware:beforeHandler',
  middlewareAfterHandler: 'middleware:afterHandler',
  chatModel: 'chatModel',
  toolExecution: 'toolExecution',
  errorHandler: 'errorHandler',
} as const;