Test versions of tools¶
A tool that sends mail, saves something or books something does so on every call. In an evaluation or a trial you usually do not want that. A test version says what answers instead of the tool: a fixed answer you write yourself, or another tool of your organisation, such as the same connection to a test environment. This page is for whoever sets up tools and evaluations.
Where a test version applies¶
- Always: in every run of an evaluation, whether you start it yourself or it runs every night, and in both runs of a trial of a leaner assistant.
- Never: in a conversation. A chat, a virtual employee and someone who uses a tool themselves in the AI Toolbox always call the real tool.
A tool without a test version is called for real, in a test too. If a test version cannot answer, for example because the chosen tool was deleted, the call fails. The real tool is never called instead.
Choose a test version¶
For each test version you choose What answers in a test?:
- No test version: the real tool is called, in a test too. This is the default.
- Fixed answer: you write the answer yourself, as Text or as JSON (an object or a list), at most 64 KiB. You can paste an answer from a real call.
- Another tool: under Tool that answers, choose a tool your organisation made itself. You cannot choose the tool itself. That other tool gets the same request.
On an API or database tool¶
Open the tool in the engine room. Under what the tool does is the Test version block. Choose a test version and save the tool with Save; the test version is kept with the rest. You need the same right as to change the tool. A tool shared with you without its contents only says that you cannot see whether it has a test version.
On a connector¶
A connector's page has the Test versions card, between the actions and the versions. Each action the connector offers has a test version of its own. Choose Save test versions to keep them together. A connector a subscription installed shows them greyed. See also Connectors.
On a fixed tool¶
A fixed tool, such as searching the web, belongs to every organisation and has no page of its own. In the engine room's overview its row has the Test version button. It opens a window with that tool's test version, for the whole organisation. Anyone who sees the overview can read it; only an administrator of the organisation changes it. Choose No test version and Save to remove it.
What you see in a run¶
Each step of a run lists under Tools used which tools the assistant called, in order. A call the test version answered carries the label Test version: the real tool was not called. That line is there also when you may not read the run's answers.
Test version or leaving out¶
You can also leave a tool out of trials or out of an evaluation's runs. A test version is usually better: a tool left out is not offered, so a scenario that checks it fails. A tool with a test version is still called, but nothing real happens.
If a test version chooses another tool that needs approval, that other tool's approval applies. Nobody can answer in a run, so that tool is not offered there.