Prompt Optimizer for AI Agents

• Generate realistic test scenarios - automatically builds customer scenarios based on your Agent's prompt, language, knowledge base, and configured actions

• Run real conversations - tests actual live chats or calls against your Agent, not static simulations

• Track configured actions - confirms whether bookings, transfers, workflows, and other actions fire at the right point in the conversation

• Score and explain results - gives a pass/fail outcome for each test, with reasons for why it succeeded or failed

• Compare prompt changes - shows exactly what changed between your original prompt and an optimised variation, before you apply anything

• Protect your live Agent - tests run on a cloned version of your Agent, so nothing changes until you approve it

• Test in multiple languages - keeps scenarios, conversations, and evaluations aligned to your Agent's configured language

• Reuse past scenarios - reload scenarios from a previous run to compare performance after a change

Currently, Prompt Optimizer can only be used for Conversation AI and Voice AI Agents. Reviews AI, Content AI and Managed Agents created in the Agent Studio do not currently support Prompt Optimizer.

1. From the main Ivorey™ menu, go to AI Agents > Conversation AI / Voice AI > Agents List > open the Agent you want to test > and click [Prompt Optimizer] above the prompt window

2. Click [Generate Scenarios] - Prompt Optimizer will automatically create contextual test scenarios based on your Agent's prompt, language, Knowledge Base, and configured actions:

   1. Review and select the scenarios you want to test

   2. (optional) Click [+ Add Custom Scenario] to create custom ones for specific workflows or edge cases

3. Click the [1x] drop-down at the top left, to select a number of test chats/calls per scenario (up to 5x per scenario)
   
   Note: Running multiple chats/calls helps test consistency since AI behaviour can vary slightly between calls. You can run up to 10 test chats/calls at a time - e.g. if you select 5x chats/calls per scenario, you can only select and run 2 scenarios.

4. Click the [Run Chats/Calls] button, in the top right corner > and review the pre-run information:

   1. Check how many free test messages/minutes are remaining for the day (each account receives 100 free messages or 20 free minutes, per day)

   2. (optional - Voice AI only) Enter a test phone number

   3. Click [Continue to run chats/calls] and Prompt Optimizer will start real conversations based on your selected scenarios
      
      Note: Prompt Optimizer creates a dedicated test contact to avoid unintended activity. For Voice AI, these test calls are real voice calls.

5. Wait for the tests to be completed > and review the Test Results:

   1. Pass/fail summary and score - overall accuracy percentage across all tested scenarios

   2. Cost - total cost and breakdown

   3. AI scoring and reasoning - why each test passed or failed

   4. Individual transcripts (and recordings for calls) - read exactly what was said (and listen back for calls)

   5. Tools - which actions were triggered and whether they fired correctly (shown directly in the transcript)
      
      Note: The accuracy score is not the only measure of success. Always review transcripts, recordings, and AI reasoning before deciding whether to update your prompt.

6. If tests failed, click [Improvise] in the top right corner:

   1. Select [Improvise] again, to generate one optimised prompt variation based on the failures

   2. Select [Auto] to run up to multiple variations automatically and find the best performer

   3. Wait for the evaluation processing to complete

7. Click the [Prompt Diff] button to see exactly what changed between your original prompt and the optimised version

   1. Switch between Inline and Side by side to compare

   2. Click [View Full] to read through the entire optimised prompt with no comparison

8. Click the [AI Reasoning] button to get a deeper look at why changes were made

9. If you want to use an optimised prompt, click [Prompt Diff] > [Use Prompt] to apply the optimised prompt to your live Agent

Note: Your live Agent is not updated until you click Use Prompt. All testing runs on a cloned version of your Agent, so nothing changes until you confirm.

Yes, running multiple executions on the same scenario helps reveal inconsistent behaviour, since AI responses can vary between runs. You can also reuse scenarios from previous test runs. This is useful for comparing performance after you change the prompt, Knowledge Base, actions, or configuration.

Yes, testing with the Prompt Optimizer can fire real actions such as booking appointments or sending emails. Prompt Optimizer creates a temporary test contact to keep this separate from your real customer records.

If you’re concerned about test bookings blocking real slots on your calendar, for example, you can use a dedicated test calendar, workflow, etc. to avoid unintended activity.

Prompt Optimizer runs real, scenario-based tests and generates tested prompt variations based on the results. It's different from simpler tools that only review or rewrite selected text on request without running live tests.