Why metadata matters
If data is the ‘Cinderella’ of corporate IT, then metadata is probably the ‘pumpkin’, because without the pumpkin, Cinderella cannot go to the Ball! Without metadata, you cannot ‘use’ data or realise the value of data. Even a simple tweet of 140 characters may map to dozens of lines of information code.
“Metadata unlocks the value of data, and therefore requires management attention.” [Gartner 2011]
“Data discovery and profiling (DDP) is the most important data management task that enterprises are still not doing. DDP is crucial for data-intensive initiatives such as data warehousing, business intelligence, master data management, and data integration.” [Gartner, 2013]
Metadata is a corporate ‘blindspot‘: poor, incomplete or missing metadata has considerable consequential impact and cost, much of which may be hidden and therefore realisation of the cost and impact is not fully appreciated until well after the event:
- Search functions are slow and may not produce meaningful results;
- Projects using AI rely on meaningful data for meaningful outcomes;
- Finding data consumes a lot of manpower and time and may not produce reliable or timely results;
- Big data analysis is slow, tediously behind the curve and again, may not produce meaningful results;
- Data governance, which needs to be granular, especially in regulated environments, is likely to be challenging if not impossible;
- Different departments or divisions of the organisation have different and irreconcilable data models;
- Some packaged applications will have no data model or no usable data model which usually means intensive manual effort and delay – and even labourious reverse engineering may still not provide you with a view of the actual data;
- Data modelling tools will still need metadata that is fit for purpose and if table or column names are not useful, it will be difficult to determine which are relevant to use within the tool – who would imagine, for instance, that ‘T005U’ referred to country names? Not obvious, is it!
- Migrating legacy databases will be painful and tedious, effectively working blind;
- If applications lack a coherent enterprise data model (because they are the result of corporate acquisition) then consolidating instances, even of the same technology, may be a minefield;
- If migrating to Cloud, then at some point, the organisation will have to face the challenge of determining how it intends to reconcile different naming conventions from different application vendors into a single cloud, SaaS or other target repository;
- Reverse engineering may eventually yield the missing data model, but it will not necessarily give you the opportunity to view the data itself (subject to appropriate permissions).
Slow, inaccurate metadata discovery (the ‘metadata bottleneck’) itself gives rise to further negative consequences such as:
- Cost overruns;
- Low productivity;
- Quality issues;
- A demotivated business that loses faith in the IT department and in the data itself. This can be very culturally corrosive as time goes on.
For most organisations, metadata should answer the simple questions:
- Where’s the data?
- What does the data mean?
- Where do I put the data?
For organisations with a strong regulatory environment, the metadata may also need to answer:
- Where the data has been?
- Who owns it?
- Who had access to it?
- Who should have access to it?
- How long for?
- What they did with it?
- What the original or raw data was?
- What the known state of the data was at each stage of its lifecycle?
- What data is to be destroyed?
- When is the data to be destroyed?
- When did erasure happen?
And, not all metadata is created equal. Metadata sources may include the following, which are harder to interrogate:
- Microsoft ERP and CRM;
- Salesforce (increasingly);
- Other large packages;
- Unstructured data.
How can I reduce the impact and cost of metadata issues?
If you have metadata issues, in packaged enterprise applications (such as SAP, Salesforce, JDEwards, Peoplesoft, Siebel, Oracle EBS, or Microsoft Dynamics) then using Safyr, designed by Silwood Technology, will save you a huge amount of time and effort.
This is what Bloor said and why Incorvus recommends Safyr:
“In our opinion, every major user of an Oracle or SAP application should be a Safyr customer…. If you are a large enterprise and you use any of the relevant SAP or Oracle applications then you will almost certainly have data marts and warehouses, integration issues, master data management, archival or data migration projects to take just a few examples that would benefit from the use of Safyr.” [Philip Howard, Bloor Research, 2011]
The Safyr metadata discovery tool reduces the cost and effort of many hours of manual labour significantly; automating the process and producing more reliable results than human intervention alone can achieve. Other approaches simply are not effective due to:
- Inaccessible metadata: nothing useful in the system catalogue that can help you untangle the data dictionary;
- A very large data model: e.g. SAP > 90,000 tables which will take some time with manual intervention even if you deploy additional headcount;
- A complex data model: which needs to be multi-dimensional;
- A customised data model: which may not be supported by original vendor or which does not align with newer models;
- The inability to port the metadata structure, once discovered and analysed, into its next ‘home’. This may otherwise require a lot of manual intervention, re-keying, or expensive consultancy in order to reproduce it correctly. Safyr saves a great deal of this labour otherwise required to present metadata results.
Safyr customers explain, with devastating clarity, the key issues they faced before using Safyr, and the benefits that Safyr delivered:
“I am still puzzled why so many developers spend countless hours trying to decipher the 90,000 tables in SAP, instead of using Safyr.” [Adrian Farcas, QlikView Developer, Virtek]
“Frankly we simply could not have done what we did without some way to extract that metadata automatically. To discover it and hand enter it manually would have taken thousands of hours.” [Lorin Yeaton, Information Architect, US Aircraft Manufacturer]
“After doing a quick prototype metadata extract from SAP, the response to the insights SAFYR provides has been very positive! I’m really grieving for the lost years without access to this tool. It has met and exceeded my fairly lofty expectations… If you are running SAP and don’t have this tool in-house…mistake” [Brian Farish, IT Architecture Senior Manager, AMD]
“The team was originally informed that no data model was available for the SAP application or for SAP BW… Having been told by our ERP implementation partner that they couldn’t deliver a data model for SAP and even by some folks at SAP that no such thing existed we were on the look out for anything that would help with our understanding of SAP’s data structures. Once we found [Safyr] making a case for purchase was pretty straightforward. At Safyr’s price point the payback in our case is only a few months when offset against the Google & trial and error approach we were having to employ to get information out of SAP to answer the questions our business colleagues were asking. … As a result of our investment in Safyr we are able to take a more agile approach to meeting the demands for new reports and data within acceptable timescales and the business’ trust in the information they provide is growing.” [Scott Delaney, Business Intelligence & Data Integration Manager, Hydro Tasmania]
“The data in these (ERP) systems makes sense and are useful, but only in the context of the hard-coded processes. In short, the data is trapped inside a complex web of thousands of database tables whose integrity is solely controlled by a rigid fossilized collection of software algorithms. If you don’t believe me, just ask your SAP support staff for access to directly update (or even read) a data table.” [John Schmidt (vice president of Global Integration Services at Informatica Corporation]
“RS are succeeding in achieving a level of understanding of data in SAP that we previously thought impossible We have quickly assembled a set of detailed subject area data models which we can now use to guide project activities. The Safyr models deliver a level of detail that we would not otherwise be able to achieve without extensive user research (and a large helping of guesswork) We have high confidence in the detail in each model as it is coming directly from SAP itself.” [RS Components]
“It’s difficult to imagine how we could have integrated complex SAP, Siebel and J.D. Edwards metadata structures into our corporate information architecture without Safyr. The cost was low compared to other methods and a fraction of the overall project budget.” [Global Retail Brand]
“Safyr was a clear enabler for our SAP to SAP DFPS migration and related Data Warehouse redevelopment project. We could not imagine how we could have solved all the issues in time without it!” [European Defense Organisation]
“Safyr allowed us to quickly analyse intricate SAP tables, relations, and data – like a knife cutting through butter. It has saved us a considerable amount of time in developing a cross-application program that links R/3 and our legacy application.” [US Food Company]
How Safyr can release your data: see Bloor – Safyr enhances catalogues.
Metadata services from Incorvus: