Knowledge Source Limitations in Copilot Studio | Day 74

Day 74 of 365 Days of Copilot Studio explores knowledge source limitations. Connecting knowledge is a starting point. Reliable answers also depend on content quality, retrieval, freshness, access, and testing.

Day 73 covered improving AI response accuracy. Today, we look at the gaps that can remain even after a source has been added, and how to design around them.

Available Knowledge Does Not Guarantee an Answer

A source may contain the information a user needs, yet the agent may fail to produce a useful answer. The relevant passage must be found and used in the right context.

Think of the path as knowledge source → retrieval → answer. Check each stage when a response is incomplete or incorrect. Adding more documents does not automatically fix a retrieval or content problem.

Poor Content Leads to Poor Answers

Unclear content makes it harder to retrieve and interpret the right guidance. A document with vague headings, mixed subjects, or missing conditions can support a weak answer even when it contains some correct facts.

• Use descriptive titles and headings that match the subject.
• Keep related rules and their exceptions together.
• Explain who a policy applies to and when it takes effect.
• Make important steps explicit instead of relying on assumed background knowledge.

For example, a leave policy should distinguish entitlement, eligibility, approval, and carry-forward rules. Improve the content before expecting the agent to explain it clearly.

The Agent Must Find the Relevant Content First

Available knowledge and retrieved knowledge are different. A user question must lead to the relevant information before that information can ground an answer.

When testing a question such as "Can I carry leave forward?", verify that the answer is supported by the carry-forward rule, including any conditions. A broad summary of the leave policy may sound useful while missing the actual question.

1. Confirm that the approved source contains the answer.
2. Check whether the content is clearly organized and specific enough.
3. Try realistic variations of the question.
4. Compare the response and its supporting source with the expected answer.

Sources Can Disagree

The lesson illustrates a simple conflict: Source A says 20 days of leave, while Source B says 25 days. Both documents are available, but they cannot establish one clear entitlement without more context.

Ask the content owner whether the difference reflects locations, employee groups, effective dates, or an obsolete policy. Document valid differences clearly and remove outdated guidance from the active knowledge scope.

Use an authoritative source for each applicable rule. Leaving a conflict unresolved asks the agent to work with uncertainty that should have been resolved in the content.

Old Knowledge Can Produce Old Answers

An agent can retrieve a passage successfully and still answer incorrectly if that passage describes a retired policy. Retrieval success alone does not establish that the information is current.

• Assign an owner to each important source.
• Review policies when business rules change.
• Update current guidance and retire superseded versions.
• Retest affected questions after changes are available to the agent.

For a policy agent, an accurate explanation of last year's rule is still the wrong answer to a question about today's entitlement.

Access Still Matters

Knowledge must be accessible through the configured source and authentication model. For sources that use the signed-in user's permissions, a user without access to the relevant content cannot rely on that content to ground an answer.

Test with representative users as well as the maker account. Check that authorized users can get the expected guidance and that restricted information remains protected. Resolve access issues through the intended permissions model.

If the required knowledge is unavailable, define a helpful response: explain the gap, request clarification where useful, or direct the user to the responsible team.

Knowledge Is One Part of a Capable Agent

The lesson separates three responsibilities:

Capability	Purpose	Example
Knowledge	Provides context and facts.	Explains the approved leave policy.
Instructions	Guide the agent's behavior.	Ask for location when the entitlement depends on it.
Actions	Perform work through configured integrations.	Submit a leave request through an approved workflow.

A document explaining how to request leave does not itself submit a request. Design the agent around the work users need to complete, with suitable instructions and actions alongside reliable knowledge.

Test What the Agent Cannot Answer

Expected questions are only part of a useful test set. Include edge cases, missing content, and known failure scenarios so you can find gaps before users encounter them.

Test case	What to verify
Expected question	The response matches the approved source and answers the question.
Edge case	The agent preserves exceptions or asks for missing context.
Missing content	The agent acknowledges the gap and offers a useful next step.
Conflicting sources	The conflict is identified and resolved with the content owner.
Outdated guidance	The response reflects the current policy.
Restricted access	Authorized access works and restricted content remains protected.

Record the question, expected result, actual response, supporting source, and improvement needed. After a change, repeat the failed question and related variations.

Design Around Knowledge Limits

Use this review routine when preparing an agent for real users:

1. Trust the content: confirm ownership, accuracy, and scope.
2. Check access: validate the source configuration and representative users.
3. Verify retrieval: confirm the relevant content supports the response.
4. Keep knowledge current: update changed rules and retire obsolete guidance.
5. Test continuously: cover successful answers, exceptions, and unanswered questions.

Key Takeaway

Reliable agents need trusted content, appropriate access, good retrieval, and continuous testing. Understanding knowledge source limitations helps you decide whether to improve a document, resolve a conflict, adjust access, clarify instructions, or add an action.

Know the limits. Improve the knowledge. Test the answers.