AI Studio Observability Overview - Appv5 Knowledge

What's in this article?

What Is Observability in AI Studio?

Transparency is critical when working with Large Language Models. The Observability screen uses the same underlying trace data as your home screen but allows you to inspect the exact step-by-step process of every single AI interaction.

Observability provides detailed insight into every interaction your users have had with your agents. Each agent interaction is logged as a trace, which can then be reviewed. A trace captures the input query and the output response, along with all reasoning, guardrails, tool calls, and other steps the agent takes while formulating its response.

Traces are powerful for understanding what your users and customers are asking, how your agent is responding, and for troubleshooting unexpected answers.

How to Access Traces in CXME?

You can find trace details by going to AI Studio > Observability. From here you will see aggregate stats for each of your organization's agents.

You can click into any agent to get the full list of traces for that agent. You can filter the list at the top or click into any trace to get the full details of that specific trace.

You can also find trace details in the footer of each agent response in the test chat panel in AI Studio agent configuration. This allows you to see the detailed trace information in real time as you are testing updates to your agent.

What You Will Find in a Trace

Each trace contains the following information:

Feedback, Scores, and Tags

Scores and tags are added to traces to help you sort through all recorded interactions and find the ones that need your attention.

Every response generated by an AI agent includes a standard Thumbs Up / Thumbs Down feedback widget. This allows internal users and agents to grade AI performance in real time. When a user leaves a thumbs up or down rating on a response, a corresponding score is applied to the trace, and you can easily filter for only traces with those scores.

When a user submits negative feedback (Thumbs Down), they can leave an optional comment detailing what went wrong (e.g., a missed workflow rule or gaps in content). AI Studio turns this feedback into action: you can draft new knowledge content or update agent rules directly from the feedback loop to continuously optimize your AI over time.

Tags are also added for certain responses — for example, if the agent could not answer a question and used its fallback response, that trace will receive a fallback tag. You can add filters to your observations list at the top of the observations screen once you click into a specific agent.

Feedback Loops > Turn Feedback into Action for Continual Improvement

At the top of each trace details page, you will find three actions that allow you to improve your agent without leaving the trace page.

Create Article

If your agent could not answer a question because it did not have the right content, you can draft an article so that the next time it will be able to answer the question. This action uses the context of the trace to draft the article, and you can add any additional details you want. The article is created as a draft so your content team can review and approve it before it goes live.

Suggest Rule

If your agent had the right content but answered in a way that you did not like, you can add a draft rule onto the agent specifying how it should have answered that question and similar questions in the future. This rule will appear on the agent configuration screen, ready for you to review, approve, and test the next time you are updating your agent.

Create Task

If you are not sure what content you need or how to answer a question, you can create a task and assign it to the appropriate subject matter expert in your organization to take action.