What is data lineage and how does it differ from the better-understood discipline of data management of which it’s becoming a crucial component in forward-thinking circles?
In a complex world of interconnected business systems – with data flowing within and between applications, and against an enhanced backdrop of regulations – it’s becoming ever-more important to take a ‘lineage-first’ approach when making any decisions relating to data and business systems.
Philip Miller and Philip Dutton, Co-Founders and Co-CEOs of Solidatus, explain their vision of data lineage, and put it in context for different applications.
With data becoming an increasingly important area for risk assessment, we ask Miller and Dutton how data lineage capabilities are being adopted by companies in today’s ecosystem, and kick off a podcast discussion that takes in talking points as varied as:
- a useful analogy of the living, breathing blueprint needed to build an ambitious and technically complex skyscraper like The Shard in London, versus a simple isolated house built more than a hundred years ago
- the equally useful analogy of Google maps and an Ordnance Survey map: one is highly accurate at a point in time; the other is broadly accurate all the time
- the joy of getting lineage right within a regulated data ecosystem, which allows you to simulate all of your proposed changes in one go and see how they affect your landscape, if you’re compliant and whether you can write policy on the top of it – or whether to ‘fail fast’
A review of the many industries beyond banking and finance for which data lineage can help, such as pharmaceuticals and healthcare, as well as any company with an ESG program in place.
With data systems so organic and evolving, lineage is all about keeping your view up-to-date and living. As Dutton said in the recording: “Do you want to know that there was a traffic jam yesterday? Or do you want to know there’s a traffic jam in front of you? That's the difference.”
Put like that in yet another pithy analogy, it’s hard to argue against the approach that he and Miller advocate.
As covered in the podcast, data lineage is in essence far more holistic than traditional ways of viewing and interpreting data. It allows you to track, manage and understand it in a far more 360° way.
Listen to the full discussion and get the complete lowdown:
You may also be interested in:
7 things the Chief Data Officer needs to know about ESG
A-Team regulatory data handbook 2021/22
The power of data lineage to deliver compliance and business insight
Data isn't expensive, complexity is