The rapid evolution of artificial intelligence (AI) has led many organizations to prioritize generative AI initiatives, often driven by a fear of missing out (FOMO). While the technology holds transformative potential, the rush to implement it can lead to common pitfalls seen with other complex systems. AI is a specialized field, and diving in without the necessary expertise can lead to challenges that overwhelm businesses.
In fact, nearly 75% of organizations attempting to build AI agents internally are predicted to fail, according to Forrester. This underscores the importance of careful planning and understanding when embarking on AI initiatives.
The Complexity of Building AI Architectures
AI agents rely on intricate architectures that require multiple interconnected models, retrieval-augmented generation (RAG) systems, and advanced data frameworks. Forrester analysts Jayesh Chaurasia and Sudha Maheshwari describe these architectures as "convoluted," often demanding extensive technical expertise.
One significant hurdle for companies attempting to develop AI agents in-house lies in handling RAG systems and vector databases. According to Forrester's senior analyst Rowan Curran, generating accurate results within tight timeframes is a challenge. For instance, processes like re-ranking data—identifying the most relevant documents or information—are often misunderstood or overlooked.
As an example, feeding a model with 10,000 documents might result in the system identifying the 100 most relevant items. Yet, due to context window limitations, a human might need to select the final 10, which can reduce the model’s overall accuracy. Curran notes that building and optimizing RAG systems typically requires 6 to 8 weeks, with initial accuracy rates as low as 55%, gradually improving through iterations.
To succeed, developers need to focus on:
- Data Quality and Availability: Ensuring input data is accurate and well-structured.
- Re-ranking and Evaluation: Fine-tuning outputs to align with verified sources.
- Continuous Optimization: Iterating and adjusting based on feedback and testing.
- Creative Model Tuning: Adjusting temperature settings to balance creativity and precision.
Despite advancements, Curran warns against viewing AI implementation as a simple process. "There is no easy button," he stresses, highlighting the extensive human effort required for deployment and ongoing support.
Navigating Deployment Options: Build, Buy, or Use Open Source?
When exploring AI agent deployment, enterprises face a crucial decision: whether to develop in-house solutions, leverage third-party providers, or use open-source tools. Each approach has its benefits and challenges, and selecting the right path requires a structured strategy.
Andreas Welsch, founder of Intelligence Briefing, recommends organizations start by evaluating their specific needs:
- Assess Workflow Pain Points: Identify tasks or processes consuming significant time.
- Analyze Task Complexity: Determine whether the tasks involve IT systems or accessible data.
- Set Clear Benchmarks: Understand how improving these tasks can deliver measurable value.
Additionally, companies should evaluate their existing software licenses to determine if they already have access to AI agent capabilities. Many software solutions include AI add-ons or premium features that can be activated with minimal effort.
Starting Small and Scaling Strategically
Rather than attempting widespread adoption, enterprises should focus on implementing AI agents in one business function. Welsch advises identifying areas where teams spend time on repetitive manual tasks that are not easily automated with traditional code. By testing AI capabilities in these controlled environments, organizations can identify potential gaps and set realistic expectations.
It’s equally important to involve employees in the process by educating them about AI agents' benefits and limitations. This fosters a collaborative atmosphere and helps manage expectations for performance.
You may also like: The Rise and Struggles of Apple’s Vision Pro: A Look at the Mixed-Reality Landscape
Building a Cross-Functional AI Strategy
A successful AI strategy requires collaboration across departments, says Curran. Business leaders, software developers, data scientists, and user experience teams must align on goals and contribute to a comprehensive roadmap. The roadmap should answer questions like:
- What are the organization’s long-term objectives?
- How can AI support these goals effectively?
Curran emphasizes that developing AI agents is not just about the technology—it requires understanding how AI integrates into business workflows. "The skills to build and maintain these systems are still being developed," he notes, which can slow down internal projects and lead organizations to seek external help.
Leveraging Third-Party Expertise
Many enterprises find success by partnering with third-party providers. These vendors often have the resources to stay updated on the latest AI technologies and build solutions tailored to clients' needs. Working with a third-party provider also ensures access to expertise in areas like post-deployment maintenance, governance, and compliance.
That said, in-house development is still an option for companies with robust technical teams, well-structured data, and clear API strategies. Organizations with well-governed, tagged, and documented data can leverage internal capabilities to build custom AI agents effectively. However, they must prepare for the continuous effort required after deployment to ensure long-term accuracy and reliability.
Final Thoughts: Planning for Long-Term Success
AI agents represent a powerful tool for modern enterprises, but successful implementation demands a thoughtful approach. Companies must carefully weigh the pros and cons of building versus buying, start with small-scale deployments, and involve cross-functional teams to create a unified strategy.
As Curran aptly puts it, "There is no free lunch post-deployment." AI systems require ongoing maintenance, testing, and refinement to deliver value consistently. By investing in the right resources and partnerships, organizations can harness AI’s potential to transform their operations and achieve their goals.