The impact of data transforming how business and society functions have progressed enormously over the last few decades, yet recent advances have fallen far short of their potential because only a very few – the data experts in our organizations – can “deal with data.”

Organizations know that enabling their business professionals to be “information workers” will herald new waves of innovation.  The buzzwords “citizen analyst” and “citizen integrator,” and “data democracy” all point to this desire.   The objective is to arm business users with a means to experiment and construct new data insights and data processes on the fly.   It is no accident that this aligns with the corporate mantra of “digital transformation,” which seeks to harvest data and reimagine business processes and customer outcomes.

We need a completely different framework to make data valuable and accessible to a “business engineer” – a person that improves how any work activity is performed.  This person could be reducing exceptions in the submission of insurance claims or improving delivery truck patterns to minimize fuel consumption, or in life-changing ways such as identifying people with risk of diabetes so that interventions may be applied before it’s too late.

These business engineers innovate by analyzing and manipulating data, which nowadays holds the “keys” to every process and interaction in modern societies. The problem then is how to make the variety of data much more relevant and reachable, something typically solved by “technology engineers,” with its inherent bottlenecks and impedance (what I need and what you gave me are not the same).  

Businesses are not just ready for it; they are asking for it.  Allow me to share a story using the world of healthcare.   Before the pandemic, we ran a survey collecting input from over 200 people in health care who are leading improvement teams or processes.   We spoke to physicians in leadership roles, heads of nursing, heads of population health, heads of clinical quality, financial leaders, and IT leaders.  The purpose: to zero in on the data challenges most impacting their work. 

The #1 issue from the survey – Data Governance.  What’s striking is that people like the “head of nursing” are talking about the need for data governance.  This was not so five years ago.  The importance of data is top of mind in almost every facet of the workplace.  Digital Transformation – shorthand for “data is at the center of modernizing business processes” – is so rooted that everyday people talk about data’s importance; it sounds small, but it is seismic.

The premise is simple – good data habits are extremely impactful to business success, and data governance establishes those habits.  If we go back to our survey participants, this is their real gripe; people living these “digital transformations” are saying their work is undermined if the data, its meaning, its provenance is ambiguous or uncertain.

This can change – we can arm business engineers with the freedom to explore the evolving and changing nature of data to solve problems faster.

We need a new model for data governance, new capabilities for interacting with data, and a new playbook for IT, business engineers, and business leaders to make it happen.  This collective framework – a Governed Data Ecosystem – has emerged from our work leveraging data to improve the business performance for our customers and rethinking the evolving technological approaches that work well and eliminating the hype where they don’t.

The Failure of Data Lakes

The Data Lake arrived on the scene because our data technologists recognized that the classic data assembly workhorse – the data warehouse – lacked the necessary agility.  Yet despite significant efforts, data lakes failed spectacularly at meeting expectations.

There are two reasons why.  First, the baseline technology (Hadoop and its cohorts) was too complicated.  Despite heroics to surmount that challenge, the fatal blow is the second reason:  no governing architecture.

While some organizations tout success, what they’ve achieved is enabling a small group of experts.  They may be doing important work, they may have breakthroughs, but just like a lab researcher, no one else can participate.  The collective contribution of knowledge workers will drive far broader business innovation, much more than the lone researcher.

The Cloud vendors have addressed the technical challenges by abstracting, templating, and simplifying the data lake platform eliminating the IT engineering burden.  Technology, therefore, is not the issue anymore; it is our approaches to data governance that is failing us.

The data lake objectives – data variety, immediacy, capacity, agility – with little thought about the data itself left a gigantic mess.   Inherent in the overhead of the data warehouse was a deliberate evaluation and structuring of data – cumbersome and slow-going, but “consumption ready” as a result.

We need to find a way to marry the agility of the data lake with equal agility in making it “consumption ready.” 

Agile Data Governance

Here’s the problem: most governance initiatives are organized as programs.  Anything organized as a “program” implies a top-down organizational structure is driving the change. The governance effort misses the organic vitality of connecting with the day-to-day reality of people getting things done.

Know what I need to know about the data that matters most to me and make it easy to get my hands on it.  What’s so hard about that?   Not much, if I am the expert in the data – where it lives, its weird codes, how it was generated, and such.  But if I am the expert, shouldn’t I help to govern it?   And shouldn’t someone else who is an expert help as well?  And wouldn’t it be best if I can “govern” as I work with the data?  That is when I am most intimate with its details and idiosyncrasies.

This leads us to a critical maxim: data governance programs struggle because:

We need to foster a “govern at the point of use” self-sustaining program of good data habits, a culture of Community Governance… 

The experts in the data – in this case, a population health analyst – is with whom the community-driven governance occurs.  As they do their work, essential “tribal knowledge” regarding the data is captured.  Meanwhile, the department leaders – in this case the VP of Population Health – oversees a process for creating consensus and clarity on the captured tribal knowledge as they focus the organization on its priorities.   Both parties propagate valuable governance outputs as they go about their day-to-day work.

The other wonderful aspect is that the community governance process is inherently self-sustaining: governance is integrated into how people work from the start rather than being a separate endeavor.

Data Governance is an investment in organizational infrastructure and accumulates with an amplified impact over time.  Except for compliance purposes, governance processes should be:

• User-centric (rather than bureaucratic),
• Adaptive, and
• Selective

We help our clients realize this combination of data with governance agility by intertwining the technologies of real-time data integration, data lakes, data catalogs, data governance, and data security as an architecture. We overlay a set of processes and reposition the critical roles that IT assumes to make this possible.

The last decade has established the necessary technology innovations and their use will fast become mainstream this decade.  Our work with forward-thinking organizations to help them apply and scale the methods we have developed continues to evolve and demonstrate distinct competitive advantages.

Kurt Rosenfeld is Chief Technology Officer at CTI.