AI Center of Excellence: A Primer

Large Language Models (LLMs) are increasingly essential in the realm of artificial intelligence, propelling innovation in everything from chatbots to advanced data analysis. Yet, the effectiveness of LLMs relies on selecting a toolset that matches your requirements. This blog post is designed to assist business and tech leaders in selecting the proper tools for LLM implementation by offering clear, objective ways to evaluate them, simplifying the process and speeding up the integration of LLMs into your projects.

To begin, we will explore the various data storage solutions, examining their suitability based on your LLM’s needs, such as real-time chat capabilities, lasting memory, or efficient information retrieval. This will help you understand the importance of choosing the right data storage option to ensure that your application runs smoothly and efficiently.

Next, this section will provide you with the insights needed to choose the appropriate compute resources, ensuring that your LLM performs optimally. We will consider factors such as the necessity of pre-trained models, the potential for fine-tuning, and the applicability of OpenAI models.

Lastly, we will focus on strategies for integrating your LLM into existing infrastructure. We will discuss considerations for multiple UIs, containerization, and event-driven architectures. Understanding these aspects will help you seamlessly integrate your LLM into your current setup, making the most out of your existing resources while leveraging the power of advanced AI.

While this blog post focuses on the foundational aspects of choosing the right toolset for your LLM application, it is important to note that other capabilities, such as version control systems (VCS) like GitHub and infrastructure as code (IaC) tools like Bicep, play a significant role in the broader context of DevOps and MLOps. These aspects, while crucial, are not the focus of this post. Instead, we concentrate on defining the core strategies and reasons for selecting one approach over another in the context of LLM operations.

By the end of this blog post, you will have a comprehensive understanding of how to choose the appropriate data storage, compute environment, and management platform for your LLM application, ensuring that your implementation is both efficient and effective. Let’s dive into the first section: choosing the proper data storage.

When does the LLM Tooling happens?

One of the key factors that influences the choice of Large Language Model (LLM) solutions is the timing of the decision. Ideally, the solutions that encompasses LLMs should be selected during the design phase of the LLM application, where the value stream, operational costs, and roadmap are being defined. This is because the services, platforms, frameworks and technical strategies have a significant impact on the feasibility, scalability, and performance of the LLM application, as well as the development and maintenance effort required.

Choosing the right LLM tooling (the set of services that support better LLM Apps) early on can help prioritize the projects that have the most potential and avoid costly rework or migration later. Therefore, it is advisable to assess the LLM tooling options before starting the implementation of the LLM application.

Drawing a parallel with traditional DevOps, the selection of tools is akin to making architectural design decisions. In DevOps, tool selection typically occurs concurrently with the architectural design phase. Similarly, in LLM Ops, the selection of Azure tools should happen while the architecture of the LLM application is being drawn up. This alignment is crucial for several reasons:

· Functionality: Ensure that the selected tools are the right ones for each part of the functional requirements and deliver the proper functionality for the task.

· Integration: Ensuring that the chosen Azure tools integrate seamlessly with the architecture of the LLM application.

· Scalability: Selecting tools that support the scalable design of the application.

· Performance: Choosing tools that meet the performance requirements of the application.

· Cost Efficiency: Considering the cost implications of the tools in relation to the overall budget and operational costs.

Defining the Value (Stream and Proposition)

The definition of the value stream in LLM Ops should follow the value proposition of the application. The major difference here with traditional DevOps is that we’re also looking into how the inclusion of an LLM could change the features of a given product and improve the value it delivers. This dual focus requires a comprehensive understanding of both the business and technical aspects of the application.

• Business Value: Understanding how the LLM application will add value to the end-users and stakeholders.
• Technical Value: Identifying the technical capabilities and features that will enable the LLM application to deliver this value.

Steps of the Value Proposition

Begin by defining who your users are, both external and internal. For external users, specify the demographics, psychographics, and behavioral characteristics of your ideal users. For internal users, identify the stakeholders within your organization who will benefit from the product. Creating detailed user personas can help in this process. These personas should include background information, goals, challenges, and preferences, providing a comprehensive understanding of both external and internal audiences.

After properly defining the final users of the application, conduct interviews and surveys to gather insights into the needs, challenges, and expectations of your audience. Complement this primary research with external research that enables you to understand what are the main features that your application should have to improve the target operation. This external information might include market research to analyze market trends, competitor offerings, and industry reports. This will help you understand the broader landscape and how your application can address the needs of all stakeholders. You could rely on the Ishikawa diagram to map what are the major problems, and their reasons to be a problem. Take this example of a Ishikawa diagram that would point the operational root causes for using a LLM to manage the unstructured data pipelines:

After that, clearly articulate the specific problem or set of problems your LLM application aims to solve for both external and internal users. This involves identifying the problem and explaining its impact on these groups. Understanding why solving these problems is important will help in crafting a compelling value proposition that resonates with all stakeholders. A good way of implementing a structured and standardized framework for this is using the 5W2H framework to frame the features and specifications of the solution presented to each problem statement or dimension. Take this example that extends the previous Ishikawa diagram and maps each problem into a solution envisioning:

A 5W2H example for the Ishikawa Diagram

The 5W2H framework describe the tangible outcomes users will experience, such as improved efficiency, cost savings, or enhanced accuracy. For internal users, this might include streamlined workflows, better decision-making support, or reduced operational costs. In addition to these functional benefits, you may include highlights of the emotional or psychological benefits, such as reduced stress, increased confidence, or greater satisfaction for both external and internal users. This dual focus on functional and emotional benefits can make your value proposition more compelling, and the usage of 5W2H easier to see it.

After that, identify what makes your LLM application unique compared to competitors. This could be advanced features, superior performance, ease of use, or better support. Explain how your LLM application offers better value or solves the problem more effectively than other solutions. These unique selling points (USPs) and value differentiators will set your application apart in the market and within your organization.

Finally, ensure the value proposition statement is simple, clear, and easy to understand. Focus on the specific benefits that are most relevant to both your external and internal audiences. Make the statement compelling and persuasive, encouraging all stakeholders to support and use your application.

An example of the Value Proposition Statement

Implementing Large Language Models (LLMs) in our company’s operations will revolutionize content creation and management processes, delivering high-quality, contextually rich content to our external users while significantly enhancing productivity and reducing workload for our internal teams. By leveraging advanced AI-driven tools, we will ensure precise, engaging, and semantically rich content that resonates with our audience, boosting user satisfaction and retention. Internally, streamlined workflows and automated content enhancement will prevent burnout and increase efficiency, enabling us to meet deadlines with superior quality output. This transformation will not only enhance user experience and brand perception but also drive operational excellence and sustainable growth.

Key Benefits

1. Enhanced User Experience:

· High-quality, contextually accurate, and engaging content.

· Increased user satisfaction and retention.

· Improved brand loyalty and reputation.

2. Operational Efficiency:

· Reduced workload and burnout for content creators and editors.

· Streamlined content creation and review processes.

· Timely delivery of superior quality content.

3. Cost Savings:

· Lower costs associated with user attrition and poor engagement.

· Reduced rework and productivity losses.

· Efficient resource allocation and utilization.

A framework for decision making

One of the tools that can help you with this challenge is TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution), a multi-criteria decision analysis method that ranks different alternatives based on their closeness to an ideal solution and their distance from a negative-ideal solution. TOPSIS can help you evaluate different LLM tools according to multiple criteria, such as:

Ease of implementation: How easy is it to integrate the LLM tool into your existing workflow and infrastructure? How much coding and customization is required? How well does it support your preferred programming language and framework?

Resiliency and versatility: How robust and reliable is the LLM tool in handling different types of inputs and outputs? How well does it adapt to changing requirements and scenarios? How scalable and flexible is it to accommodate your current and future needs?

Time-to-market: How fast can you deploy the LLM tool and start generating value from it? How much time and effort can you save by using the LLM tool instead of building your own solution from scratch or using other alternatives? How competitive is the LLM tool in terms of quality, performance, and accuracy?

By using TOPSIS, you can assign weights to each criterion according to your priorities and preferences and score each LLM tool based on how well it meets each criterion. Then, you can calculate the similarity and dissimilarity of each LLM tool to the ideal and negative-ideal solutions, respectively, and rank them accordingly. This way, you can have a clear and systematic framework for decision making that considers multiple aspects of LLM integration.

What are the decision drivers for LLM tooling?

Navigating the landscape of integrating Large Language Models (LLMs) into your applications can be complex, full of considerations and possibly many implementations that would match a good solution for each case. Let’s delve into the most important decision-making drivers you need to consider.

Memory: The Backbone of LLM Applications

First and foremost, memory is a crucial aspect of any LLM application, regardless of its architecture. Whether you’re building a simple chatbot or a complex agent-driven system with multiple capabilities, you must determine if your app requires long-term or short-term memory. Long-term memory involves a persisted, well-structured, scalable, and resilient database. This is essential for applications that need to retain information over extended periods, such as customer service bots that need to remember user preferences or historical interactions. The choice of database technology here is critical options like Azure SQL Databases, Azure Cosmos DB (e.g., for MongoDB or NoSQL), each offer different strengths. For instance, the NoSQL distribution for Cosmos DB provide flexibility and scalability, making them ideal for dynamic data storage needs.

On the other hand, short-term memory requires a fast, malleable, and scalable database, suitable for applications that need to process and retrieve information quickly without long-term storage. In-memory databases like Azure Redis are often used in these scenarios because they offer low-latency data access. However, the trade-off here is persistence; data stored in-memory can be lost if the system crashes. Thus, you might need strategies to periodically snapshot the data to persistent storage. Another option for this is using Azure AI Search for short-term storage, but be aware that, being a vector database, its writing capabilities are limited and is not a good solution for workloads where transactions are critical.

Choosing the right memory strategy also involves considering the trade-offs between speed, scalability, and complexity. Long-term memory solutions can add complexity to your architecture, requiring robust data management and backup strategies, as well as more security layers to keep the data safe. Meanwhile, short-term memory solutions, while faster, might necessitate additional mechanisms to handle data persistence and recovery.

Balancing these factors is key to ensuring your LLM application performs optimally while meeting its functional requirements. To improve the driving experience of this decision-making process, we provide a comprehensive TOPSIS scoring strategy that should be adapted to your current use case. In this strategy, we consider three dimensions to compose the scoring of each criterion and tool.

The first criterion is scalability, that is, how resilient the tool is for general memory operations within the LLM tool, which encompasses retrieval and delta operations on existing data, as well as the creation of new data in general transactional environments. By addressing the scalability in terms of resiliency, consistency and capacity to keep the performance stable on increasing data management operations, we can guarantee, or understand, if the memory operations will become a bottleneck of the app performance, and when under which workload that might happen.

The second criterion is the overall maintainability of the solution as a memory management system (MMS). This is similar in capacity of a database management system, with the inclusion of the precision of the retrieved content based on ai generated queries. This is important because memory operations might occur in environments where the human supervision is limited or inexistent, like under the usage of a Chain (using tools like Prompt Flow and Semantic Kernel) or under complex agent integration, and is critical for the outcome of most (if not all) LLM Apps.

The third criterion is adaptability to the current data ecosystem that the AI operation relies on. To guarantee the consistency of data pipelines in production, is nice to have easy integration tools on each data source that the LLM App relies on, because it enables easier maintenance and better ownership of the data pipeline. This is the most business-oriented metric and, although not explicitly stated, should also consider FinOps in the evaluation of the total cost of ownership.

Based on those three criteria, we formulate the following table. Consider the triple on each row the score for each criterion stated above, in the same order they were stated (scalability, maintainability, adaptability), which scores from 1 to 5, where 1 means totally unfit and 5 means totally fit:

TOPSIS reference scoring table for data tool choice

We deep dive through those criteria in a separate post (which we’ll add the link here after it’s ready).

Compute: Powering the LLM Engine

Next, compute resources are closely tied to the ability to deliver prompt answers in various scenarios. The main drivers here are like those that govern standard application compute instances. If your LLM application’s processing — essentially, the implementation of its operational reason-to-exist — needs to run asynchronously, whether in batch or real-time, your compute instance should be oriented to handle non-HTTP requests.

This might involve using message queues like Azure Service Bus or Azure Event Hub to manage task distribution. The type of compute will vary depending on throughput needs and the nature of the tasks. For example, high-throughput, batch-processing tasks might benefit from using scalable cloud services like Azure Functions or other serverless compute on Azure, which automatically scale with demand.

Conversely, if the processing is meant to be synchronous, like in live chat or live summarizer applications, you should opt for synchronous solutions. This could involve using traditional web servers or serverless architectures designed to handle HTTP requests efficiently. In these cases, low-latency performance is critical, and you may need to consider using high-performance compute instances or even specialized hardware like GPUs for intensive tasks.

In any case, it’s fundamental to use containerized components to ensure flexibility, scalability, and ease of management. Containers, orchestrated by tools like Azure Kubernetes Service, allow you to deploy and manage applications consistently across different environments. They also make it easier to scale your application horizontally by adding more container instances as demand grows.

However, containerization comes with its own set of challenges, such as managing container orchestration and ensuring security across multiple instances. Addressing these challenges effectively can significantly enhance the reliability and performance of your LLM application. In particular, the orchestration within containers should be followed by a smart hub for handling the different models used on different containers, heavily relying on Azure AI Studio and Azure API Management for that.

We also proceed here with providing a TOPSIS scoring strategy that should be adapted to your current use case for fast implementation of a decision-making process for the compute strategy to use on the App. In this strategy, we consider two dimensions to compose the scoring of each criterion and tool.

The first criterion considered is the support for different software and architectural patterns in different languages and orchestration strategies. LLM Applications usually require the handling of request in a complex way, majorly due to software patterns where Chains and Agents are implemented, and the capacity to support an environment capable of handling different patterns in a consistent way is critical.

The second criterion considered is the resiliency in handling requests with different nature (synchronous or asynchronous) under different protocols. To guarantee the best practices on software engineering, including (but not limited to) SOLID, DRY, KISS and Clean Code, the compute strategy should consider the execution of the task under different environments, favoring implementations that may define components that are simple and have well-defined responsibilities.

Based on those two criteria, we formulate the following table. Consider the tuple on each row the score for each criterion stated above, in the same order they were stated (support, resiliency), which scores from 1 to 5, where 1 means totally unfit and 5 means totally fit:

TOPSIS reference scoring table for compute tool choice

We deep dive through those criteria in a separate post (which we’ll add the link here after it’s ready).

Agentic-Related Engines: Ensuring Seamless Interaction between AI and Users

Finally, agentic-related engines are vital because every application ultimately serves a user, solving a specific problem for them. The quality of the human interaction is a critical factor to LLM-powered solutions because it is the core value generated by the technology, majorly due to the sensation that the user is talking to a person, bringing familiarity

For LLM Apps, there is an additional complexity where the LLM itself acts like an agent, enabling human-like interaction under different data modalities and circumstances. Depending on how you interact with the end user, you’ll need different tools for monitoring, evaluating, orchestrating, and enabling interactions.

For instance, in a chat-with-your-data app, you should consider whether you’ll create a feedback loop with human interaction, such as reactions in Microsoft Teams, and how frequently you’ll improve your prompt or fine-tuning strategies based on those interactions using Azure AI Studio. This feedback mechanism is essential for refining the application’s performance and ensuring it meets user expectations effectively and the way to handle it widely depends on the LLM and the strategy to optimize its responses.

Agent-related engines encompass a variety of tools and strategies aimed at enhancing the user experience and ensuring the application remains responsive to user needs, encompassing not only User Interface, but User Experience tools and LLM Models as well. Monitoring tools, such as application performance management (APM) solutions like Azure Monitor or Azure App Insights, can provide real-time insights into how users interact with your application and where any bottlenecks or issues may arise. These insights allow you to make data-driven decisions to improve performance and user satisfaction, both on functional and non-functional requirements.

Evaluation mechanisms are equally important. Implementing A/B testing frameworks can help you understand how different versions of your application and models perform with users, enabling you to iterate and enhance the user experience continually. In terms of orchestration, those tools play a critical role in managing complex user interactions. For applications with multiple user touchpoints or functionalities, orchestrating these interactions smoothly is crucial. Workflow automation tools like Microsoft Power Automate can help manage these processes, ensuring that user interactions are handled efficiently and effectively.

Enabling interactions often involves integrating various communication channels and user interfaces, as well as response quality assurance. Whether your application is accessed via a web interface, mobile app, or integrated into a platform like Microsoft Teams, ensuring seamless and intuitive interactions is key, as well as guaranteeing that the Model response is suitable for the use case that it is being applied.

This might involve using UI frameworks like React or Angular for web applications or tools like Flutter for mobile app development. Additionally, incorporating natural language processing (NLP) capabilities with Azure AI Studio models, both proprietary and open-source, is itself a can enhance user interactions by making them more intuitive and responsive. In a edge case, you would rely on Azure Machine Learning for traditional ML workloads for improving model response.

Creating a feedback loop with users is another important aspect of user-related engines because it directly and strongly affects the User Experience (since everyone likes to be “listened” and have the LLM to learn its own preferences). By enabling users to provide feedback directly through the application, you can gather valuable insights into their experiences and identify areas for improvement.

This might involve implementing features like in-app surveys, feedback buttons, or direct communication channels with support teams. Regularly updating and fine-tuning your LLM based on this feedback can significantly enhance its effectiveness and user satisfaction, and you will likely rely on Azure AI Studio for that integration and management purpose.

Agent-Related engines encompass a widely different set of tools that might be considered on the decision flow but may be reduced in two big subsets: User-Facing tools, encompassing everything that is associated with UX/UI, and Feedback Loop tools, which encompass operational tools that enable improvements oriented by the User feedback. The comprehensive TOPSIS scoring strategy that we provide at this topic also should be adapted to your current use case. In this strategy, we consider three dimensions to compose the scoring of each criterion and tool.

The first dimension is the resiliency under different operational scenarios that should be covered by the app. This includes the considerations of when, where and why the app should be used by a person and describes the situation where a problem that the app solves arises, as well as how the person is expected to interact with the app (including the AI). For instance, a chat app (which corresponds to 90% of the current use cases) should be enabled with streaming texts to reduce the sensation of time to get answer from the app, while a analysis-oriented app should consider a BFF architecture and run on background jobs.

The second dimension is the capacity to adhere to different, possibly ever-growing patterns of feedback and human interaction that should be considered by the solution, which strongly enhances User Experience (since it has directly impact on matching what the user wants to see and what it actually sees). This implies that the app should present features for automatically improving the responses from the AI with enhanced contexts, and depends on CI/CD cycles, data assets, and monitoring strategies to know when and where the triggers for dropping data, retraining, and model update should happen.

The third one is the capacity to escalate the operation horizontally and vertically. Scalability is critical to the business because it enables fast growth and enablement of the app after it is tested and validated, that is, when risks are mitigated and the value proposition is confirmed with a small set of users. After that happens, the business would benefit a lot from fast and controlled growth of the application because the value delivered — and thus the competitive advantage generated by it — is exponentially increasing. We start with the evaluation of User-Facing (UI) Tools:

TOPSIS reference scoring table for UI tool choice

Meanwhile, the de-facto solution for enabling user-oriented evolution of the software relies on a set of services, and not on a single one. For the LLM evaluation, Fine Tune triggering, and tracking of the LLM answers and interactions, it is important to have a plataform such Azure AI Studio that supports several different LLM-related tasks. For monitoring live workloads and providing insights and triggers, it’s interesting to put in place monitoring tools such as Azure Monitor, so that you can not only track, but gather evidence of what might be optimized in the application. Finally, for the data, it is important to have a consistent and well-integrated database solution such as CosmosDB for historical tracking and management.

Deciding which road to take

Navigating the integration of Large Language Models (LLMs) into your applications involves a series of critical decision-making steps, each tailored to ensure the optimal performance and user satisfaction of your implementation. Let’s walk through these steps, focusing on the reasoning behind each decision-making point. Look at this decision-making flow:

LLM Tooling decision flow

In this flow, we represent several decision-making steps guided by the needs of the project. We will dig into every step, but is important to consider the following major decisions during your planning phase for LLM Apps:

Step 1: Assessing Memory Requirements

The first step in the decision-making process is to determine the memory requirements of your LLM application. This involves deciding whether your application needs long-term or short-term memory. Long-term memory is essential for applications that need to retain information over extended periods, while short-term memory is suitable for applications requiring quick data processing without long-term storage. This decision will influence the choice of database technologies and the overall architecture of your application.

Step 2: Evaluating Compute Resources

Next, you need to evaluate the compute resources required to power your LLM application. This step involves determining whether your application’s processing needs to run asynchronously or synchronously. Asynchronous processing is ideal for tasks that can be handled in batch mode or real-time without immediate responses, while synchronous processing is necessary for real-time interactions like live chats. The type of compute instance will depend on the throughput needs and nature of the tasks. Regardless of the processing mode, containerization is essential for ensuring flexibility, scalability, and ease of management.

Step 3: Integrating User-Related Engines

The final step involves integrating user-related engines to ensure seamless interaction with end-users. This step focuses on the tools and strategies needed for monitoring, evaluating, orchestrating, and enabling user interactions. Depending on how you plan to interact with users, you may need various tools for real-time monitoring, performance evaluation, workflow orchestration, and communication channel integration. Creating a feedback loop with users is crucial for continually refining and improving the application based on user experiences and feedback.

General Considerations on Governance and Compliance

Governance and compliance are essential aspects of any LLM application, as they ensure that the application meets the quality standards, ethical principles, and legal regulations of the domain and the organization. By defining clear roles and responsibilities, establishing best practices and guidelines, implementing quality control mechanisms and evaluation metrics, ensuring data privacy, security, and protection, aligning the LLM goals and outcomes with the organizational vision and values, monitoring and auditing the LLM performance, behavior, and impact, and addressing any issues, risks, or challenges that may arise during the LLM lifecycle, governance and compliance can help to achieve business value and operational efficiency.

Data is the core component of any LLM application, and therefore LLM Ops, as well as ML Ops, heavily rely on the DataOps maturity and structure. DataOps refers to the process of creating, managing, and delivering data pipelines that enable data-driven decision making and innovation. DataOps can facilitate LLM Ops by providing reliable, consistent, and high-quality data for training, testing, and deploying LLM models, automating and streamlining data workflows and processes across different stages and teams, enabling collaboration and communication among data engineers, scientists, analysts, and other stakeholders, supporting data governance and compliance by ensuring data traceability, provenance, and lineage, and leveraging cloud computing and distributed systems for scalable and efficient data storage and processing.

The LLM governance should account for the security, scalability, and integration capacity of the LLM environment with all other business areas within the company. This means that the LLM governance should adopt robust and secure data encryption, authentication, and authorization methods to prevent unauthorized access or misuse of the LLM data and output, design and implement scalable and flexible LLM architectures and infrastructures that can handle large volumes of data and requests, as well as adapt to changing needs and demands, integrate the LLM application with various communication channels and platforms, such as web, mobile, social media, or voice, to enable seamless and convenient user interactions, and coordinate and align the LLM objectives and strategies with the overall business goals and processes, as well as with other existing or potential applications and systems.

Concluding Remarks

Choosing the right toolset for your LLM application is a pivotal step in harnessing the full potential of Large Language Models. By carefully evaluating data storage solutions, compute resources, and integration strategies, you can ensure that your LLM implementation is both efficient and effective. The insights provided in this blog post are designed to help business and tech leaders make informed decisions, paving the way for smoother integration and faster realization of LLM benefits in your projects.

The timing of selecting LLM tooling is critical, ideally occurring during the design phase. This ensures alignment with the overall architecture and operational strategy, like how tool selection is managed in traditional DevOps. By addressing functionality, integration, scalability, performance, and cost efficiency early on, you can avoid costly rework and ensure your application meets its objectives.

Defining the value proposition for your LLM application involves understanding both the business and technical benefits. By focusing on who your users are, gathering insights into their needs, and mapping out problems and solutions using frameworks like 5W2H and Ishikawa diagrams, you can articulate a compelling value proposition that resonates with all stakeholders.

Our detailed exploration of decision drivers for LLM tooling emphasizes the importance of memory, compute, and user-related engines. Each of these aspects plays a crucial role in the performance and user satisfaction of your LLM application. By carefully navigating these decisions, you can ensure your application is well-positioned to deliver exceptional value and transformative results.