Improvements¶

Artific can read your organisation's recent conversations on a schedule and suggest concrete improvements to your assistants and skills: an addition to an assistant's instructions, a skill that often does not help, a separate assistant for something people keep asking the wrong one. Each such proposal is a suggestion; an analysis is one occasion on which Artific reads the conversations and writes suggestions. An analysis changes nothing by itself, unless your organisation chooses that Artific applies some kinds itself: the rest waits for a person who reviews the suggestion and accepts it.

This page is for the administrator who switches the analyses on and sets them up, and for whoever may change an assistant or a skill and handles the suggestions about it.

What switching on means¶

The analyses are off until an administrator switches them on. Switching on is your organisation's consent, and the dialog Switch on improvement analyses? says what to:

• The analysis model reads your organisation's conversations, with personal data replaced by placeholders, such as <PERSON_1> instead of a name.
• Every analysis costs model calls, up to a maximum each time, and they count towards your organisation's monthly cap. Once the cap is reached, an analysis skips the rest.
• When Answer feedback counts is on, your members' thumbs, with their reasons and comments, go along without a name (Answer feedback).

Artific never shows what an analysis costs in euros. You see the number of analyses and model calls only if your organisation lets you see costs (Costs).

What nobody sees¶

A suggestion says how many conversations it rests on, never which ones or whose. No screen names or links a conversation, a person or a memory record, and a suggestion's history does not say who accepted or dismissed it. An administrator does not see it either, and neither does the API. For a shared fact a suggestion does not even say how many people or memory records it rests on.

Who may do what¶

Only someone who may change an assistant or a skill makes a change to it: write on the kind alone is not enough there. Someone who holds write but may not change the assistant can still dismiss the suggestion. You can also give the kind improvement in a custom role (Kinds and actions).

An administrator of the organisation above yours sees the settings and the analyses, but does not change them or start an analysis: switching on is your own organisation's consent, and an analysis costs it money. They see the suggestions only in a support session your organisation allows, and in it they only look.

Where to find it¶

• The list is under AI assistants › Improvements, after Quality control, with the tabs Suggestions and Analyses. The menu entry shows only for whoever holds see.
• The settings are under Organisation management › Settings, in the Improvements section, with the Set up button.
• The Improvement suggestions panel sits in an assistant's settings, above the form, and on a skill's page, above the instructions (The panel).

If your organisation's environment offers no improvements, you see neither the menu entry nor the section in the settings, and a direct link says Improvements are not available here. If you expect them, ask the administrators of the organisation above yours, or Artific if there is none.

Switching the analyses on and setting them up¶

1. Go to Organisation management › Settings and choose Set up in the Improvements section.
2. Turn Analyses on on and confirm in the dialog with Switch on.
3. Choose when and over what, the analysis model, and whether answer feedback counts (see below).
4. Save the page. Everything on the page is saved at once.

The Improvements section in the organisation settings then says in one line how things stand, for example On · Every night 03:00 · 5 checks on · Artific applies nothing itself yet.

If the organisation above yours manages these settings, Read only shows at the top and you cannot save anything. Ask an administrator of that organisation.

When and over what¶

• Clock: Every night 03:00, Every weekday 03:00, Every Monday 03:00, or Advanced for your own Cron pattern of five fields: minute, hour, day, month, weekday.
• Time zone: the time zone the clock runs in.
• Period: how many days back an analysis reads, 1 to 30. An analysis never reads further back than your organisation keeps traces.

Below the fields it says when the next analysis runs.

Analysis model¶

The analysis model reads the conversations and writes the suggestions. Choose Installation default to follow the default, also when it changes, or Own choice for a chat model from your organisation's catalogue. The top of the section says which model is in use now and why.

Artific never falls back on the helper model. Without an analysis model the checks give only figures or nothing; each check says what is then missing.

Answer feedback¶

With Answer feedback counts in suggestions and in memory, your members' thumbs up and down, with the reason and the comment, go to the analysis model without a name. It also switches on the check Answers often rated negative, which you can switch off afterwards.

• It applies to memory too, even with the analyses off: the thumbs help decide whether a session went well.
• It applies to the whole organisation. You cannot switch it on or off per assistant.
• If your organisation collects no answer feedback, the switch is off and cannot be turned on. Switch the feedback on first under Organisation management › Settings › Feedback.
• Switching the analyses on switches this on too, unless you switched it off yourself before: Artific remembers that, also when you switch the analyses off and on again later.
• If your organisation set down itself when an answer counts as good or bad, Artific leaves those rules alone and refuses to save this switch, with the reason in the message.

Checks¶

A check is one thing an analysis looks for. Each check has a switch, says what it needs and what it did last time. What it needs is read as it is now: an orange tag is missing.

Traces means your organisation records model calls as traces (Organisation settings); without traces an analysis reads no conversations and gives figures only. A leaner assistant only tries something once a monthly limit for trials is set (below).

A fact several people know reads the memory each assistant keeps of the people who chat with it. Switch it on and an analysis looks there for something about your organisation, not about a person: something at least three people told the same assistant, or something one person said that is clearly about the organisation. A fact holding an address, a phone number or a link is never suggested. What people told different assistants does not add up. Each reading is a model call that counts towards your organisation's usage. A project's memory and the shared memory itself are not read. A fact found goes into shared memory once someone accepts it (below), or by itself if you choose so under Apply by itself.

Trials of a leaner assistant¶

A leaner assistant tries one change to an assistant at a time: fewer tools first, then a cheaper model, then shorter instructions. It does so with a trial: the assistant's own evaluation runs twice, with the current version and with the change, on the same scenarios and with the same judging model. If the leaner version does no worse and is at least 10% cheaper or faster, you get a suggestion.

Under the checks is the block Trials of a leaner assistant. Only an administrator of the organisation sets it; for anyone else it is greyed out, with the reason beside it.

• Monthly limit for trials, in euro. Empty until you fill it in, and needed to switch the check on. All trials together never cost more than this a month, within your organisation's monthly cap. You can change the limit but not remove it. Switching the check off keeps the limit.
• Assistants that may be tried. At first nothing is ticked: without a tick no assistant is tried. An assistant without an evaluation of at least five scenarios and a judging model is greyed out, with the reason beside it; make such an evaluation first (below).
• Leave out of trials. Under each ticked assistant are its tools, none ticked at first. A tool you tick is offered in no trial: so nothing it would do happens. A tool is left out by its name: if another ticked assistant uses a tool with the same name, it is left out of that assistant's trials too. A tool left out that no ticked assistant uses any more stays listed until you untick it. A tool an evaluation already leaves out itself stays out of the trials as well.
• At the latest analysis says in plain words what the latest analysis did with the trials: how many started, and why others did not, for example because the monthly limit was used up.

A test version is usually better than leaving a tool out: a tool with a test version is still called in a trial, but nothing real happens.

Apply by itself¶

Per kind of suggestion you choose how far Artific goes by itself. It holds for all assistants at once.

You can choose so for Change the assistant, Rewrite the skill and New skill. A new assistant is always only a suggestion: Artific never makes an assistant by itself. For Share a fact you choose Only propose or Apply and publish: then Artific shares a fact it finds by itself, with nobody reading it first. Such a fact comes from what people said in their own conversations and goes to everyone in the organisation; you change or delete it under Shared memory like any note. Only an administrator, who may change the organisation's settings, switches this on or back off. The other kinds, such as a connection or tidying up a skill, always wait for a person.

With Apply and publish for an assistant, mind two things. If Artific adds a tool that Artific would refuse on save, that cannot be undone afterwards, only reported: remove the tool yourself in the assistant's settings. And a newly added tool may not work properly for up to an hour, until Artific has put its description in the instructions.

If Artific could not apply a suggestion by itself, for instance because a placeholder is still in it or a rule of your organisation does not allow it, it stays open and the suggestion's page says why. A draft that routes the conversations differently from the published assistant is published only by an administrator, on the assistant's page in Artific's console.

Who hears when Artific changes something by itself¶

Choose who is notified when Artific publishes something by itself: Whoever may edit it (the editors of the assistant or skill that changed, on by default), Administrators, and Groups you pick, in any combination. Tick nothing and nobody is notified; only the mark in the list stays. Where something has no editors of its own, such as a shared fact, administrators hear it instead of the editors.

Everyone chooses how they hear it, in My notifications. You change this block only if you may change the organisation's settings; others only see it.

Last month¶

Whoever in your organisation may see costs sees how many analyses ran last month and how many model calls they made. Others do not see this section.

The suggestions¶

The Suggestions tab shows the organisation's suggestions, by default those with the status Open. Filter by Status, Check and About (a kind, such as Assistant or Skill, or one assistant or skill by name). Each row shows the kind of suggestion, the title the analysis model wrote, what it is about, the check, the status and when it was last seen. Load more at the bottom loads the next suggestions.

A suggestion marked Figures only, no proposed change reports something the model could not work out, for example because there was no analysis model. You can read it and handle it, but there is nothing to apply.

If the analyses are off, the list says Improvement analyses are off, with Go to settings for whoever may switch them on. While an analysis runs, the top of the list says so, and new suggestions appear by themselves once it is done.

Viewing a suggestion¶

A suggestion shows:

• Finding: the title, the summary and the Explanation, as the analysis model wrote them. This text is not translated; it is in the language the model wrote in.
• Before and after: what accepting changes, with the current version beside the new one. Added text is green, removed text red and struck through, and tools and skills appear as tags. The section a tool puts into the instructions itself is dashed, as Section of tool … · managed by Artific.
• About, Check, Based on (the number of conversations), Last seen, the Figures the check found, the History (each step with a date only) and the Note, if someone wrote one.

Calls and tokens a suggestion would save per month are shown only if your organisation lets you see costs.

Handling a suggestion¶

What Accept does depends on the kind of suggestion; the suggestion itself says so.

• Dismiss opens Dismiss suggestion, with an optional Note of at most 1,000 characters. The suggestion does not come back for a while, unless the problem doubles in that time.
• Reopen opens a dismissed suggestion again. If a later analysis found the same thing again, that is not possible, and There is a newer suggestion tells you to reopen that newer one.
• If you may read the suggestion but not change the assistant or skill, the top of the suggestion says so and you cannot accept. Ask an editor of the assistant or an administrator.

Changing an assistant¶

Above the change it says Accepting saves version … of …, under your name. After accepting, everyone who chats with the assistant chats with the new version at once. The previous version stays under Versions; you can always go back.

Adjust opens the text the suggestion adds, or with instructions (replaced) the whole new text, so you can change it first. What you change goes into the new version; the suggestion itself keeps the original text. Back to the suggestion undoes your change.

Accepting is not possible in these cases, and the suggestion says why:

• Placeholders. The analysis model read the conversations with personal data replaced, so the proposed text may hold a placeholder such as <PERSON_1>. Replace each placeholder with Adjust; Next placeholder takes you to the next one. Then you can accept.
• The assistant has a newer version by now. The suggestion is about an older version. The next analysis refreshes the suggestion; you can also change the assistant yourself.
• Someone else just saved the assistant. Read the before and after again and then accept. Your adjustments are kept.
• The new version is refused, for example because a proposed tool cannot be given to the assistant. The reason shows above the suggestion and your adjustments are kept.

If the proposed text repeats a tool's section, the paragraph is orange with a warning: Artific puts that section back itself on save, so the text would appear twice. Remove it with Adjust, or leave it if that is intended.

Rewriting or creating a skill¶

With Rewrite the skill the suggestion says how many assistants get the new version at once, because they follow the latest version, and how many have pinned a version and keep it. A New skill changes nothing until an assistant loads it.

You can adjust the text first. Keep exactly one line with the skill's name in its header, in lowercase letters, digits and hyphens; otherwise Artific refuses the version. You need the right to change a skill with that name.

If publishing fails, the version has been written, but the suggestion is Applied with Not live yet, and the previous version still runs. Choose Try again or Publish to try again.

A skill that comes from a publication is A shared copy: only the publication changes it. You can only dismiss such a suggestion.

Creating a new assistant¶

Create assistant opens the form for a new assistant, with the proposed name, description and instructions filled in. The assistant the suggestion came from stays as it is. The new assistant starts without tools and without users; you choose who may use it. As soon as you save it, the suggestion is Applied.

If that last step fails, a message says the assistant was created but the suggestion could not be set to applied. The assistant stays.

A quality control from conversations¶

An evaluation from real conversations drafts scenarios from conversations that went well, with the details made up. The suggestion shows each scenario as it will be saved: the title, the steps and their checks. A scenario that cannot be saved here, for example with a check the form does not know, is shown with the reason and left out. If no scenario at all can be saved, Accept is greyed out.

• New evaluation: Create evaluation opens the assistant's Quality control tab with the form for a new quality control, the proposed name and judge model filled in. If the analysis found no suitable judge model, a warning says you need to choose one; none is chosen for you. As soon as you save the quality control, with the scenarios in it, the suggestion is Applied. If that last step fails, the quality control stays and a message says so.
• Add scenarios: the suggestion names the quality control and how many scenarios it holds now. Accept adds the scenarios at the end, in your name. If the quality control got a new version in the meantime, accepting is no longer possible and you dismiss the suggestion.

Accepting needs the right to change quality controls and an assistant you can see; the right to change the assistant itself is not needed. No run starts: you start one yourself.

Sharing a fact¶

The suggestion shows the fact as the note will say it, and whether it is a Fact or a Rule. Whose it was, or how many people said it, is not shown. Accept opens Share fact: adjust the text if needed, and under Who gets this note choose Everyone in the organisation, already chosen, or the members of one or more groups. After Share the fact is a note under Shared memory, where you change or delete it like any other note, and every assistant can look it up for whoever gets it.

Accepting takes the same right as writing a note in shared memory: changing assistants for the whole organisation. The right on improvements alone is not enough. Without a person, a fact is only shared if your organisation chooses so under Apply by itself.

Publishing a leaner assistant¶

A Leaner assistant suggestion shows the trial's two measurements side by side: Now and Leaner, each with its score, mean score and mean answer time, and the cost of each measurement where your organisation shows costs (Who sees usage and cost). Below them: how many scenarios were measured, how many turns a month the saving rests on, the estimated saving a month where your organisation shows costs, and what changes: the model, the tools left out or the shorter instructions. What a trial cannot show is given in English, as written: a trial measures only that evaluation's scenarios, once on each side.

• Publish version … publishes the measured version at once as the latest version: every conversation uses it straight away. Artific then finishes the assistant as it does for every saved version. Whoever may edit the assistant may do this.
• Dismiss throws the measured version away; the assistant stays as it is.
• You cannot edit the text: what was measured is what is published. If the assistant got a new version meanwhile, it can no longer go; a later analysis tries again.

Connectors and retiring skills¶

Have a connector built describes what a connector should do. Artific cannot build that connector: give the description to whoever builds your connectors, as a connector or in the AI Toolbox. With Retire the skill, Go to the skill takes you to the skill; if you find it redundant, delete it there. Then set the suggestion to Mark as done.

The panel on an assistant or skill¶

Whoever may change an assistant or skill sees the open suggestions about it in the Improvement suggestions panel, each with View. If there are none, there is no panel. If the analyses are off, there is one line, with Set up for whoever may change the settings.

If the form holds changes not yet saved, View in a new tab opens the suggestion separately. Save first before you accept, otherwise the suggestion writes a version without your changes. If someone accepts a suggestion in the meantime anyway, the form says on save Not saved: there is now a version … and that the version came from an improvement suggestion. Your text stays in the form; look at the new version and then save again.

Analyses¶

The Analyses tab shows every analysis, newest first: when it started, How (On schedule or By hand, never by whom), the status, the Period read and how many Suggestions it wrote. Model calls are shown only to whoever in your organisation may see costs.

A row opens the analysis, with what each check did:

Analyse now, top right on both tabs, starts an analysis at once. Only whoever may change the settings can use it, because every analysis costs money, and not while the analyses are off or one is already running.

What is recorded¶

The audit trail holds, under the name of whoever acted:

A text or a note is never in it. Changing an assistant also writes the usual assistant.changed, and sharing a fact is in it once, with who gets it. An analysis the clock starts is not in the audit trail; you find it on the Analyses tab.

Through the API¶

The routes are under /api/v1/organisations/{id}/improvements:

• GET and PATCH …/settings read and change the settings. If the environment offers no improvements, the read says available: false and every other route answers 409 improvements-not-available.
• POST …/runs starts an analysis (202), or answers 409 analyses-off or run-in-progress. GET …/runs lists the analyses.
• GET …/suggestions lists the suggestions, filtered by status, check, target_kind and target_id. PATCH …/suggestions/{suggestion_id} accepts, dismisses, reopens or marks as done, with a note of at most 1,000 characters.
• POST …/suggestions/{suggestion_id}/apply makes the change, optionally with your adjusted text, and for a shared fact takes audience: everyone or the ids of groups; POST …/publish publishes a skill again or a draft Artific wrote by itself, and GET …/draft reads that draft. PATCH …/settings also takes apply_level per kind, and trials with limit_eur and assistants. A refusal says why, among others suggestion-placeholders, suggestion-target-moved, suggestion-not-yet and newer-suggestion.