The Aspiration for a Modern Data Experience — part 2

We will continue expanding on the experiential goals we discussed in part 1 (https://medium.com/@DataEnthusiast/the-aspiration-for-a-modern-data-experience-part-1-3233e179cf0a), which focused on the aspirations for ad-hoc analysis. The current modern data stack consists of impressive individual components but lacks coherence. Specific responsibilities, such as managing business logic, are spread across multiple BI tools, and essential tasks, such as tracking post-answer activities, are frequently neglected, with no tangible ways to understand what is available and reliable. Additionally, simple inquiries like "Where should I start to ask a question?" often lack clear and direct answers. The expectation for everyone is to come to the modern data platform, write SQLs/data wrangling tools, and find their own answers in the name of Self-service analytics, which doesn’t sell well with non-technical stakeholders. Instead, the goal should be to create a seamless experience for data consumers who want to enjoy the insights and answers without worrying about the intricate processes of data cleaning, writing SQLs, or code-free tools for analysis, much like how diners at a restaurant savor their meals without needing to know the details of how the ingredients were prepared or who did the chopping.

Empower people to play with data based on their Skills instead of Expecting them to be Analyst

Data democratization and self-serve analytics have been popular concepts with specific ambitions: to make data accessible to everyone and to free up data teams for more strategic projects. However, these concepts have sometimes been misinterpreted as a solution to make everyone an analyst using code-free tools, which has not been very successful. In a modern data environment, the focus should be on integrating data into existing operational systems to assist people in their current roles rather than burdening them with new responsibilities.

Not everyone has the same level of expertise with data in an organization, just as not everyone has the same culinary skills in a kitchen. The goal is to empower employees to utilize their skills effectively without requiring them to become experts (like Analysts) in a field they are not trained in.

Personas with different Experiential Aspirations with Data (Credit: Co-Pilot)

Let’s draw a parallel analogy to understand the aspiration better here.

Different Skill Levels in the Kitchen:

• Home Cooks: People who can make simple meals using easy recipes and regular kitchen tools.
• Intermediate Cooks: People with some cooking experience can tackle complex recipes and use fancy kitchen tools.
• Gourmet Chefs: Highly trained professionals who can create fancy dishes utilizing various specialized tools and techniques.

Organization Scenario (Different Skill Levels in Data Handling):

• Business Users: These employees can perform simple data tasks, such as running standard reports, entering data into spreadsheets, and performing rudimentary analysis. These are non-technical personas with business expertise who must be empowered with capabilities that provide data-informed decisions, enabling them to do their jobs without expectations of learning new skill sets (SQL, Code free tools, etc.).
• Business Analysts/Intermediate Users: Employees with some data literacy can use more advanced data-wrangling tools to analyze data, create visualizations, and draw insights.
• Data Engineers/Data Analysts/Scientists: Highly skilled professionals who can handle big data and complex data sets, use advanced analytics tools, create predictive models, and derive deep insights from data.

Organizations can foster a more efficient, productive, and satisfied workforce by equipping employees with the right tools based on their skill levels. Just as a kitchen operates smoothly when everyone uses tools they are comfortable with, an organization can thrive when employees are empowered to handle data in ways that align with their expertise. Conversational AI could be the future for non-technical users, which can help interpret natural language queries and provide instant, relevant responses, making data access more intuitive and less intimidating.

The most underrated trait in the world of data is the “Ease of Use”, which will inevitably become the key selling point for data teams developing solutions.

Conversational AI to the Rescue

Conversational AI is a cutting-edge technology that empowers machines to understand, process, and respond to human language naturally. This powerful tool is incredibly beneficial for non-technical users, enabling easy access and effective data utilization. Conversational AI allows users to ask questions in plain language, bypassing the need for complex query languages like SQL. For instance, a user can ask, “What were the sales figures for last quarter?” instead of writing a query. They can also ask AI to automatically generate charts, graphs, and dashboards in response to user queries, making it easier to understand complex data. For example, “Show me a bar chart of last month’s sales by product category.” Conversational AI can help bridge the gap between non-technical users and complex data systems, making data more accessible, understandable, and actionable. There are options for integrating this capability into the operational systems.

Discovering Data and Ensuring Reliability Should be Tangible

Let’s try to understand these experiential aspirations using the analogy of kitchen skills. We can see how data discovery and reliability are analogous to exploring and preparing ingredients in the kitchen. Data discovery is about identifying, understanding, and organizing your data, like how a chef selects and prepares ingredients.

Ingredient Identification:

• Kitchen: You start by identifying your ingredients—vegetables, spices, meats, grains, etc.
• Data Discovery: Similarly, in data discovery, you identify the different data sources and datasets available within your organization. This involves understanding what data exists, where it is stored, and its fundamental attributes.

Understanding Ingredients:

• Kitchen: Once you know your ingredients, you must understand their characteristics — flavors, textures, and how they combine with other ingredients.
• Data Discovery: In data discovery, you delve deeper into understanding the data’s characteristics — structure, format, relationships with other data, and potential uses.

Finding the Best Ingredients:

• Kitchen: You select the best ingredients for your dish, ensuring they are fresh and suitable for the recipe.
• Data Discovery: In data discovery, you select the most relevant and high-quality semantically relevant datasets for your analysis or project, ensuring they are accurate and up-to-date.

Organizing Ingredients:

• Kitchen: Before you start cooking, you organize your ingredients, making them easily accessible and ready to use.
• Data Discovery: Similarly, in data discovery, you organize and catalog the datasets, making them easily accessible for analysis and decision-making.

Centralized data discovery allows for a unified access point where all data assets can be easily located and explored. This tangibility means users can see the breadth and depth of available data, fostering a comprehensive understanding of the organization’s data landscape.

Self-Reliant Data Consumers (Credit: Co-pilot)

The question, “Can I trust this?” is a common concern when dealing with data and often leads to frustration. Our ability to trust data relies on implicit signals, such as considering who created it, whether it has been recently modified, and its appearance. When we use information, it can take a lot of time to ensure it’s accurate and up-to-date. Data reliability in the data space can be likened to ensuring the quality and consistency of the dishes you prepare in the kitchen.

Quality of Ingredients:

• Kitchen: The quality of your dish depends on the quality of your ingredients. Fresh, high-quality ingredients lead to better-tasting meals.
• Data Reliability: The quality of your data is crucial in data reliability. Accurate, clean, high-quality data leads to more reliable insights and decisions.

Consistent Preparation:

• Kitchen: Consistency in preparing ingredients — like chopping vegetables uniformly or cooking meat to the right temperature — ensures that your dish turns out well every time.
• Data Reliability: In data reliability, consistency in data processing — like consistent data cleaning, validation, and transformation — ensures that your data remains reliable over time.

Regular Checks and Balances:

• Kitchen: Regularly checking your ingredients for freshness and ensuring your cooking methods are sound helps maintain the quality of your dishes.
• Data Reliability: In data reliability, implementing regular checks and validation processes helps maintain the integrity and accuracy of your data. This includes data quality assessments, anomaly detection, and error correction.

Adapting to Changes:

• Kitchen: If you notice an ingredient is no longer fresh or a cooking method isn’t working, you need to adapt and make changes to maintain the quality of your dish.
• Data Reliability: When you identify issues such as data decay or changes in data sources, you need to adapt your data management practices to maintain data quality. This might involve updating data sources, revising data cleaning procedures, revisiting the data contract, or implementing new validation rules.

Tangible Monitoring and Alerting

• Kitchen: Tracking a dish's progress is an art in itself. We watch its changing color, note how long it's been cooking, and, of course, sneak in taste tests at regular intervals to ensure it's just right.
• Data Reliability: To convey a system's reliability effectively, it's essential to present key metrics and indicators clearly and understandably through a dashboard with time series changes. This approach allows users to grasp the current status of the data system at a glance. Visual alerts like color-coded notifications and status indicators can help users recognize and prioritize issues promptly.

We need better ways to know if the processes that provide the information are working well or if the information is still being worked on.The main goal is to spend more time using the information to make decisions, and less time checking if it's right. Tangibility provides clear, visible benefits and real-world applications and processes, which fosters active participation, trust, and consistency.

There is still much more to discuss regarding these topics. Engaging in further conversation to refine this list and thoughtfully implement the delivery of the experience is the key. Keep an eye out for Part 3 of this article, where we will continue delving deeper into the experiential aspirations of a Modern Data Experience.

Check out our blog post on how to boost your data reliability by implementing Data Observability in your system! (https://medium.com/@DataEnthusiast/discovering-the-perfect-data-observability-tool-for-your-ecosystem-a3eebcaddb05)

I hope you found it helpful! Thanks for reading!

Let’s connect on Linkedin!

Link to Blogs on various topics in Data Space

https://medium.com/@DataEnthusiast

Authors

Himanshu Gaurav — www.linkedin.com/in/himanshugaurav21

Bala Vignesh S — www.linkedin.com/in/bala-vignesh-s-31101b29