What we learned from data marts and consolidation
IBM’er Nancy Kopp talks about the trade-offs between data marts and consolidation. Enter Nancy:click here to read this post in Portuguese ———————————————————————————————-
Why we built data marts in the 90s
Business intelligence at IBM started with data marts, which helped solve the problem of reporting and analysis for specific areas of the business- like financial or marketing. However, we also found they created problems, like the ability to deliver compliance, security, governance and the ever popular and dreaded data redundancy! When CFOs started going to jail for not having their facts straight, we had to get serious about compliance and governance.
Evolving from Data Marts to the Enterprise Data Warehouse
IBM and its competitors began to focus on building Enterprise Data Warehouses (EDW) to solve the problems of data marts and data sprawl. We built consolidated and monolithic infrastructures that were really good at being that “single source of truth” we were aiming for. And as we pulled all the marts into the EDW, the concept of virtual marts came along with new technology that allowed us to have cubing-like capability in the engine of the database.
We wanted control over the data so we tried consolidating, but maintaining control caused other problems to appear. It was far more difficult to maintain performance in these systems, especially for some analytical workloads. Smaller vendors in the market place were building cheaper appliance-like solutions to go after the analytics workloads and offload that functionality from the EDW. The appliances really appealed to the line of business who wanted things FAST and cheap and that demand created a big surge of appliances in the marketplace starting in about 2003.
But Data Marts kept popping up! Like playing “Whack a Mole”
Ten years later, and after all the things we’ve learned, we’re once again seeing this trend of marts popping up. Have we figured out a way to solve the original problems of marts? Well kinda, but we made some tradeoffs along the way. One of those tradeoffs was having control over the data but we found the EDW wasn’t catering to business needs, like ROI and fast time to value. I have seen ROI numbers as high as 145% – no kidding. If you were a CFO, would you choose fantastic ROIs or the EDW?
Why CFO’s keep choosing data marts
Ok, Ok so I know marts happen but it’s WHY they happen that keeps me up at night. How is it that we built these wonderful EDW structures but still somehow failed our business counterparts? How could they not care about governance and compliance? And if we don’t manage the deployment of these marts, we’ll end up back where we started a twenty years ago.
Not to be forgotten though, we can now declare data warehousing a success because there is more pressure than ever to deliver more of what the business wants. Quite the conundrum eh? How do we manage the trade offs of data governance and the ability to respond to the needs of the business?
A Smarter way forward – learning from both data marts and the EDW
I don’t think the answer lies in simply saying that throwing everything on one box will get you to where you need to be. Sure it’s simple, sure it sounds good – in theory. Oracle is still pushing a strategy of one size fits all – what that tells me is their maturity level in data warehousing is about IBM circa 2003 – but hey that’s ok they will catch up.
In reality the monolithic architectures have fixed some issues and created others. We fixed some governance, metadata, security and compliance issues. But we created such complexity in our modeling and methods that we aren’t as useful to our business sponsors.
In a nutshell, we kept our architects and DBA teams so busy they lost their connection to the business. I think you have to be far more practical about how to approach data warehousing.
The answer to what lies ahead is being able to take advantage of consolidation and also to be able to meet time-to-value, in particular for analytical applications. We must match the needs of the business to the right computing technology.
IBM’s evolving business strategy for the EDW
In my next post I’ll look at IBM’s new business strategy that has evolved from what we’ve learned from data marts, the EDW, and 20 years of experience in business intelligence.
See more from Nancy on her blog at NancyKoppOnDataWarehousing.wordpress.com.