Agent Factory vs one-off prompt tuning

Prompt tuning is useful. A clear instruction can change tone, clarify a boundary or improve the order in which an agent asks questions. The problem begins when prompt editing becomes the whole improvement system. A team reads one transcript, changes a sentence, deploys it everywhere and hopes the next calls are better. That process is fast at the beginning and fragile at scale.

The difference between a prompt and an operating release is evidence. A release should identify the problem, explain the intended behaviour, add a test case, compare the candidate with the current version and record who approved the change. Dring's Agent Factory is built around this lifecycle: understand the work, assemble an agent, simulate difficult calls, release in stages and improve from live feedback.

Why one transcript is not enough

A single conversation can reveal an important failure, but it cannot tell you how often the failure occurs or what else a change might affect. Changing a greeting to fix one impatient caller can create a longer opening for every customer. Adding a broad instruction to avoid one policy mistake can make the agent refuse valid requests. Improving one language can introduce an unnatural phrase in another.

Start by clustering the evidence. Group calls by intent, language, customer type, failure reason and downstream outcome. Look for repeated misunderstandings, policy confusion, missed handoffs, tool errors and pronunciation problems. Then decide whether the right intervention is a prompt change, a glossary update, a system integration fix, a routing rule or a human boundary.

Factory thinking separates the layers

An agent has several layers that should not be mixed casually: conversation strategy, knowledge, policy, speech recognition, pronunciation, tools, identity, routing and post-call record. If a caller says a product name incorrectly, the solution may be terminology or speech adaptation rather than a new paragraph in the system prompt. If a CRM write fails, adding “always update the CRM” does not fix a permission or timeout problem.

Dring's agent configuration, quality controls and integrations reflect these layers. The goal is to make the next change legible. A supervisor should be able to say what changed, why it changed, what it was tested against and how the production result will be reviewed.

Define a release candidate

Give each meaningful change a release identity and a short brief. The brief should contain the affected workflow, target metric, known risks, new test cases, rollback condition and owner. A release candidate is not only a new prompt. It can include a knowledge revision, a tool permission, a language-specific pronunciation set or a revised handoff threshold.

Run the candidate against a fixed regression set and a fresh set of realistic conversations. Include ordinary calls, awkward calls and policy traps. Use independent scoring where possible so the person who wrote the change is not the only judge. The quality page should show both the score and the reason behind it.

Use live calls as signals, not automatic training

Production conversations contain valuable operational evidence, but not every utterance should become a new instruction. Reviewers should classify the signal, remove irrelevant noise and decide whether it reflects a durable issue. A customer's one-off phrasing may be useful for a test case without being suitable as a universal rule.

Be precise about data governance. Customer conversations may contain personal or sensitive information. Define access, retention, review roles and the approved use of anonymised or summarised signals with the customer and legal team. Dring's security controls and privacy information are starting points for that discussion; this article is not legal advice.

Measure improvement by trade-off

A candidate can improve resolution and worsen handoff quality. It can reduce average handle time and increase repeat contact. It can sound more natural while becoming less precise about policy. Track the intended KPI together with guardrails: policy adherence, tool accuracy, transfer reason, sentiment, repeat work and critical defect count.

Review results by workflow and language rather than only as one global average. If one change makes a visible improvement for a support queue but harms a sales queue, keep the scope separate. A factory makes that separation possible by treating agents and workflows as releaseable units rather than one giant prompt.

Make improvement visible to customers and operators

When an agent changes, operators should know what to expect. A short release note can describe the updated intent, new boundary, test coverage and monitoring period. Human teammates should hear how a handoff will arrive and which fields have changed. Customers should not be used as silent test participants for an unreviewed experiment.

Staged rollout keeps the organisation in control. Start at a small traffic percentage or a single queue, watch the evidence and expand only when the scorecard is stable. Keep a rollback path that restores the last known-good release without losing customer records or open tasks.

Prompt tuning still has a place

Prompt changes are often the right intervention when the model has the correct information but uses it in the wrong conversational order. They can improve empathy, clarify a question, state a limitation or choose a better handoff phrase. The factory does not make prompts less important. It gives prompt changes a home inside a more reliable process.

• Use prompt changes for behaviour that is clearly expressed and testable.
• Use glossary and pronunciation changes for names, brands and domain terms.
• Use knowledge changes when the answer itself is outdated or incomplete.
• Use integration changes when a tool or system result is wrong or unavailable.
• Use routing and human boundaries when judgement or risk exceeds the agent's scope.

One-off tuning asks, “What sentence should we change?” Agent Factory thinking asks, “What did the operation learn, what intervention fits the failure and how do we know the next release is better?” The second question is what allows a digital call-center teammate to improve without becoming unpredictable.

Choose the smallest useful intervention

Before changing a release, name the layer where the failure actually lives. A caller misunderstanding a product name may need terminology or pronunciation work. A correct answer delivered with the wrong tone may need a conversation-strategy change. A missing CRM note may point to a field definition, permission or timeout rather than a prompt. This diagnosis keeps the fix narrow and makes the result easier to attribute.

Compare the candidate with the current version on the same scenarios, then watch the live slice for the agreed review period. Record both improvement and collateral change: fewer repeated explanations is useful only if policy accuracy and handoff quality remain stable. When the evidence is mixed, keep the release small, explain the trade-off and create the next test instead of hiding the uncertainty inside a larger prompt.

Further reading

• OpenAI on orchestration, visibility and evaluations for production agents
• NIST AI RMF Core on govern, map, measure and manage
• NIST AI RMF Playbook on practical lifecycle actions