Andy on Enterprise Software

Some rare common sense

2006-07-18T07:34:00.001-07:00

Ad Stam and Mark Allin have written an excellent piece in DM Review this month covering data stewardship and master data management. They correctly point out that, with regards to business intelligence systems, that “change will occur, and time to react will decrease” and lay out a sensible architecture for dealing with this issue. I very much like the way they put emphasis on the need for a business unit to deal with data governance as a key building block. In the article they explain the key requirements of such a group and make the interesting analogy of logistics, which is usually sourced these days to a separate unit or even separate company. Similarly they believe that the management of master data should be managed by a non-IT business unit.

The article also correctly distinguishes between “golden copy” data held in the data warehouse and a master data management repository, which in addition will hold master data in all its stages. The master data repository should be linked to the data warehouse, but are not the same physical entity since the master data repository has to handle “unclean” data whereas the data warehouse should have only fully validates data stored in it.

It is a pleasant change to read such a sensible article on best practice in data management, but this is because Ad and Mark are real practitioners in serious enterprise-wide projects through their work at Atos Origin e.g. at customers like Philips. They are not people who spend their lives giving slick Powerpoint presentations at conferences but are close to the action in real-world implementations. I worry that there are too many people on the conference circuit who are eloquent speakers but haven’t actually seen a real live project for a long time. I have known Ad Stam for many years and can testify that his team at Atos are an extremely competent and experienced set of practitioners who focus on customer delivery rather than self-publicity. If you have a data warehouse or MDM project then you could do a lot worse than use Ad’s team.

Vive la France

2006-07-17T06:06:00.000-07:00

For some time I have been involved with an EU project that wrapped up last week in Brussels. With the unpromising name Sun&Sup it tried to identify the issues that hold back hi-tech start-ups in Europe, and to make recommendations that could improve the current situation. The project invited periodic input from selected hi-tech start-up companies across the EU (along with various service providers to start-ups) and I represented the UK on this project.

Make no mistake that there is a problem: once you get beyond SAP, Business Objects and Sage you will be hard pressed to name a large European software company. Israel has done a better job than the combined resources of Europe, with companies like Check Point Software, Amdocs, Mercury Interactive and many others. Israel has the second highest ranking for VC investment, and even in absolute terms has the second highest number of start-ups after the USA, yet it has a population of just over 6 million. There are many reasons for Europe’s hi-tech malaise, and few easy answers. The Sun&Sup project tried to deliver some very low-key, pragmatic services in pilot form, such as a self-help network of companies wishing to expand across borders, an expert system to help companies assess their business plans, a mentoring program to provide non-executive directors for start-ups, amongst others. Its most ambitious recommendation was to lobby to replicate the US system in government procurement, which sets aside USD 100 billion of government spending for small companies. European government procurement favour large companies: 50% of economic activity in Europe is from SMEs, yet only 30% of government spending is with SMEs. Of course opening up more government business to SMEs would not be a panacea, but it would help, as the successful federal Small Business Act has demonstrated for many years.

The highlight of the wrap-up session of the project in Brussels was to hear the French Trade minister Christine Lagarde making an eloquent case for the need for change in public procurement. It was indeed refreshing to an Anglo-Saxon ear to hear a small business initiative being championed by a French minister. Ms Lagarde was an extremely impressive speaker, yet clearly faced entrenched opposition from the Commission and indeed from several member countries in trying to open up public procurement. Indeed, from the way that several of the modest Sun&Sup initiatives ended up being buried or transferred to other EU projects, it seemed clear that the lack of high-tech competitiveness in Europe is something that will remain the subject of much hand-wringing for a long time to come.

SeeWhy real-time event monitoring makes sense...

2006-07-13T07:19:00.000-07:00

Given the consolidation in the business intelligence sector, and the recent share price dips even of leaders Cognos and Business Objects, you might wonder why anyone would bring out a new BI product. Certainly there is no shortage of reporting tools, data mining has yet to break out of its statistician niche, while visualisation tools have again failed to become a mass market. However one area that does make sense for a new entrant to focus on is real-time event monitoring, which is typically today addressed (poorly) by the vendors of major applications.

SeeWhy software is a UK start-up which has managed to get over the key hurdle of signing up initial high class customers such as Diageo. It is run by Charles Nichols, previously an executive of Business Objects. Charles is a smart guy who understands the space well. The software pulls data out of real-time message queues, enabling alerts to be generated e.g. for supply chain data in the case of Diageo. The company should continue to focus on this niche in my view, and ovoid trying to be "all things to all men". For example it would be natural to extend its capability to data mining in order to spot anomalies or trends, but would be wise to partner with existing data mining tools in order to do this. Similarly, if they start to build up repository capabilities and looking at trends in their customer data they should avoid trying to compete with general purpose data warehouse technology, or they risk undermining their message of "real time" analysis. I have written elsewhere how EII vendors struggle when they try and position themselves as general purpose business intelligence tools, since fundamental issues like data quality get in the way if you do not have a persistent store of data such as a data warehouse. This has led to pioneer EII vendor Metamatrix stalling in the market, with virtually no growth in revenues last year. By concentrating on drawing data from real time message queues, marketing to that niche and by selective partnering in other areas SeeWhy should be able to prosper in an apparently crowded market.

Managing software risk

2006-07-12T10:00:00.000-07:00

An interesting, common sense debate was featured in Silicon.com last week. A panel of CIOs was asked whether they felt comfortable buying from small suppliers or whether they preferred dealing with the big players. There was a surprising degree of consensus that, in general, CIOs felt OK about small suppliers: indeed in some cases they actively preferred them. Perhaps this is a sign of a steady recovery in economic confidence, with CIOs preparing to come out of their shells into which they retreated in 2001 and 2002. As I have written about before, buying software from small suppliers carries risks, but this is true of big suppliers also. Just because a giant company may not go bust does not stop them dropping products for any number of reasons, as I can testify from personal experience.

The one element in the article that did make me smile was the assumption that code escrow was a form of insurance against a small vendor folding. Indeed code escrow arrangements have become quite standard in contracts, and generate modest fees for those organisations that provide the service. I hate to disillusion those CIOs, but code escrow is not the panacea it may seem. Sure, so you get the source code, but then what? Firstly, you have to hope that the vendor has been diligent about keeping their escrow up to date with the version of software that you are actually using. But more to the point, the raw code itself is of limited use without the design specifications that go along with it (at least assuming you actually want to continue developing it). Even if you are looking at basic support only, how well documented is the code? I had the misfortune to try and execute an escrow contract once when I was working at Esso. The tape of source code duly turned up and it was 3 million lines of undocumented assembler code. While my colleague (an expert at assembler code) got a misty gleam in his eye as he could see a job for life coming up, we concluded that we simply couldn’t justify taking this on, and opted to go for a complete replacement instead. So, if you are insisting on source code escrow from your vendor, be aware of the pitfalls and ask some searching questions about documentation.

What's next for ERP?

2006-07-07T04:22:00.000-07:00

The ERP landscape became simpler this week, when SSA was swallowed up by a private equity company called Golden Gate Capital. This group (and its subsidiary Extensity) has now absorbed Baan, Comshare, Dun & Bradstreet Software, Epiphany, Infinium, Marcam and Systems Union. As Ventana points out this means that the choice boils down to SAP, Oracle, Microsoft and this new amalgam under GoldenGate/Extensity. It is interesting that this private equity group seems to be performing the role CA used to play: hoover up under-performing companies, slim them down and milk the support revenue stream. What the article implies is that this is pretty much the endgame for the ERP space, but I am not so sure. The one dimension missing here is the hosted model.

Salesforce.com showed what could be done with a hosted software model. In the ERP world we are seeing new entrants like Intacct and Ataoi, which while they are small so far are making solid inroads into their chosen markets. At present this approach may appeal more to SMEs, but remember that salesforce.com started this way as well, only later taking on Siebel more directly. I know the CEO of one of these emerging hosted ERP vendors, who was amused to be in a competitive bid with SAP at one prospect. His company’ bid was less than one-tenth that of the behemoth. I’m not suggesting that these hosted ERP systems compare in functionality with SAP and Oracle, but perhaps that after all is the point. Traditional ERP systems have become so bloated that large parts of them remain unused and having systems hosted avoids all the environmental installation problems that ensure with traditionally installed software, where there are so many combinations of operating system, DBMS, transaction monitor etc that the vendors have to spend as much time testing combinations of software than actually writing new functionality.

It may take some time but I think the next change in the ERP market will be via this hosted model. When you start to see defensive market offerings by the giant vendors, just as Siebel started an on-demand offering (but too late), then we will know that this prediction has been fulfilled.

Don’t prototype: iterate

2006-07-06T06:01:00.000-07:00

Stephen Swoyer makes a case for using EII as a prototype for data warehouses in his recent Enterprise Systems article. As the article reflects, there are some dangers as well as benefits here e.g. the prototype may just be extended and a proper data warehouse system never built. This is a problem because, as I have argued elsewhere. EII is suitable only for a small subset of business intelligence needs. However the valid point is that business users do want to see prototypes, especially in an area like business intelligence where the requirements tend to be fluid and ill-defined. However there is an alternative to buying an EII tool, knocking up a prototype and then building the proper warehouse.

These days you do not have to build a data warehouse, since you can buy one. Packaged solutions can be deployed much more rapidly than data warehouses that have to be designed and built by hand, and if they are flexible enough then an iterative approach to the warehouse can be taken. A great example of this was at Owens Corning, who deployed a data warehouse package over a 90 day period, using weekly builds of the business model. Each week a new piece of the business problem was tackled, the business model was updated in the package and the results presented back to the users. Issues would arise, feedback taken, and the next week the model would be refined, and a new business area started. This highly iterative approach ensured that the business users were fully engaged with the project, and could see a visible improvement in what they were going to get week by week.

After a few weeks the problems became less technical and functional, and more business related e.g. issues of data quality and how certain business model issues were to be resolved. After 90 days the application was delivered, and this was no prototype: it was a fully working, deployed production warehouse. The insights this application gave saved Owens Corning a great deal of money, so much so that the project had a three week payback period. Indeed the project became an Infoworld 100 winner.

Data warehouse project leaders need to rid themselves of the notion that data warehouses have to be long, custom build projects. At present TDWI reckons they take 16 months to deliver on average. This is far too long if using a traditional waterfall methodology, and indeed needs a more iterative approach. But why build a throwaway prototype when you can implement the real thing via a data warehouse package?

The long and winding data quality road

2006-07-03T08:55:00.000-07:00

As Rick Sherman quite rightly points out in his commonsense article in DM Review, data quality is not something that you can realistically fix before you build a data warehouse. Data quality in operational systems is often scarily low, but often the only way that this will be highlighted is when it is brought together at a data warehouse. People often assume that the data inside their ERP systems is somehow sacrosanct and immune to data quality issues, and it often comes as a big disappointment when they discover that this is not so.

In one example it turned out that a product was mis-priced in the SAP system in a region, resulting in the product being sold at cost price. This anomaly went undetected for over a year before a data warehouse project brought this to the surface (by comparing gross margin by product by country). Initially everyone assumed it was a bug in the warehouse software, but it was not. Indeed this insight alone pretty much paid for the project.

If a data warehouse can be implemented rapidly and in an iterative fashion then it can quickly highlight business issues such as this one, which may be as a result of data quality or could be as a result of new insight that was previously unavailable: the "wood for the trees" problem. Eventually data quality needs to come out of the closet and be treated as a serious business issue, dealt with by a corporate business function that have the political clout to fix the problems at source. Some progressive companies have set up such organisations, which may report into finance or another corporate function, but never into the CIO.

However, to show just how long a journey data quality can be, I can recall working with such a function in Esso UK in the mid 1980s. The issue is only now dawning on many companies, and still has to surface in most.

A multiple view of the truth

2006-06-29T09:16:00.000-07:00

Ragy Thomas highlights the plight of marketers in his recent article in DM News. As he points out, marketers are usually poorly served by corporate IT systems. A critical thing for them is to get a good understanding of their customers, is that they can craft carefully targeted campaigns, yet corporate CRM systems have failed to deliver the much talked about, yet elusive "single customer view". He sets out a sensible approach which is essentially a recipe for master data management in the specific context of customer data. In particular he talks about the need to document the existing data sources, to map the differences between them, to set up processes to improve data quality and then to consider how to deal with integrating and automating the processes.

I would add that such an initiative should sit within the broader context of an enterprise-wide approach to master data management, else we will see a new generation of silos developing, with a customer hub, a product hub, a hub for various other types of data, all in themselves duplicating and so having to be synchronized with data in underlying transaction systems. Nobody wants to see different technologies and approaches used for the marketing data, for the finance master data, for the supply chain data etc. That marketing departments are having to resort to such initiatives shows just how hollow the "single view of the customer" promises from giant application vendors have turned out to be.

Operational BI and master data

2006-06-28T06:35:00.000-07:00

In an article in the beye network Mike Ferguson makes an interesting observation which many seem to have missed. A current "theme" is that business intelligence needs to be embedded into operational systems. This innocent sounding notion is of course entirely unrelated to the need for BI vendors to sell more licenses of their software in what has become a slightly saturated market. As I have written about earlier, it is by no means clear that everyone in an organization needs a BI tool, whatever the wishes of BI vendor sales staff. So trying this new tack of bundling BI in with operational systems (which many people do need, or at least have to use) is a cunning ploy. However, as Mike Ferguson notes, if this is done without a basis for common master data and common business rules, then all that will happen is that we will get lots of competing BI reports, all with different numbers and results. The whole notion of data warehousing was created in order to resolve the differences in definitions that exist between operational systems. Each separate ERP implementation will usually have slightly different data, never mind all the other operational systems. By going through a process of trying to get to an enterprise-wide common subset of definitions (Mike calls it a "shared business vocabulary") then these differences can be understood, and data quality improved back in the source systems. Without such an underlying basis we merely have snapshots of operational data without resolving the data quality issues, and without resolving the inconsistencies. In other words, we will be pretty much back where we were before data warehouses.

There certainly may be valid examples where it makes sense to embed some simple BI capability on the top of operational data, especially in the context of operational reporting where you are only interested in the data in that particular system. However as soon as you want to take a cross functional or enterprise view, then that pesky data inconsistency and data quality has to be dealt with somehow. Putting complex BI tools in the hands of the front-line staff doing order entry is not going to resolve this issue -it may just confuse it further.

Classifying MDM products

2006-06-27T01:38:00.000-07:00

In his article on the IBM MDM conference Philip Howard makes the distinction between different MDM approaches, which is useful but I feel does not go quite far enough. As he says, at one extreme you have "hubs" where you actually store (say) customer data, and hope that this is treated as the master source of such data (a non trivial project then usually ensues). At the other end you have a registry, which just lists the various places where master data is stored and maps the links to the applications which act as the master, and a repository, which goes further in picking out what he calls the "best record" and is sometimes called the "golden copy" of data. The distinction between a repository and a regiistry is a subtle one which Bloor makes. I feel that there are other aspects which are useful to categorize though, beyond just the storage mechanism. Some MDM products are clearly intended for machine to machine interaction, synchronizing master data from a hub back to other systems e.g. DWL (now bought by IBM). However there are other products which focus on the management of the workflow around master data (managing drafts, authorizing changes, publishing a golden copy etc), and so deal more with human interaction around master data. Kalido MDM is one example of the latter. This is another dimension which it is useful to classify tools by, since a customer need for synchronized customer name and address records between operational systems is very different from workflow management.

The article notes that IBM does not score well in the recent Bloor report on MDM, but hopes for better things in the future. Certainly IBM did something of a shopping spree once they decided to tackle MDM, and bought a PIM product, a CDI product and a few others while they were at it, so it is perhaps not surprising that it is difficult to see an overall strategy. I absolutely concur with Philip Howard in that MDM needs to be treated as an overall subject and not artificially segmented by technologies that deal with product, customer or whatever. In one project at BP we manage 350 different types of master data, and it is hard to see why a customer can reasonably be expected to buy 348 more technologies to go beyond product and customer. This example illustrates the absurdity of the technology per data type approach which is surprisingly common amongst vendors.

Software is hard to rearchitect, and customers should always look carefully at vendor claims of some overall gloss on top of multiple products, compared to something which was designed to handle master data in a generic fashion in the first place.

Mergers and Measurement

2006-06-26T02:15:00.000-07:00

Margaret Harvey points out in a recent article that the effort of integrating the IT systems of two merged companies can be a major constraint and affect the success of the merger. Certainly this is an area that is often neglected in the heat of the deal. But once the investment bankers have collected their fees and an acquisition or merger is done, what is the best approach to integrating IT systems? What is often missed is that, in addition to different systems e.g. one company might use SAP for ERP and the other Oracle, the immediate problem is that the two companies will have completely different coding systems and terminology for everything, from the chart of accounts, through to product and asset hierarchies, customer segmentation, procurement supplier structures and even HR classifications. Even if you have many systems from the same vendor, this will not help you much given that all the business rules and definitions will be different in the two systems.

To begin with the priority should be to understand business performance across the combined new entity, and this does not necessarily involve ripping out half the operational systems. When HBOS did their merger, both Halifax and Bank of Scotland had the same procurement system, but it was soon discovered that this helped little in taking a single view of suppliers across the new group given the different classification of suppliers in each system. To convert all the data from one system into the other was estimated to take well over a year, but instead they put a data warehouse system in which mapped the two supplier hierarchies together, enabling a single view to be taken even though the two underlying systems were still in place. This system was deployed in just three months, giving an immediate view of combined procurement and enabling large savings to be rapidly made. A similar appraoch was taken when Shell bought Pennzoil, and when Intelsat bought Loral.

It makes sense initially to follow this approach so that a picture of operating performance can quickly be made, but at some point you will want to rationalize the operational systems of the two companies, in order to reduce support costs and eliminate duplicated skill sets. It would be helpful to draw up an asset register of the IT systems of the two companies, but just listing the names and broad functional areas of the systems covered is only of limited use. You also need to know the depth of coverage of the systems, and the likely cost of replacement. Clearly, each company may have some systems in much better shape than others, so unless it is case of a whale swallowing a minnow, it is likely that some selection of systems from both sides will be in order. To be able to have a stab at estimating replacement costs, you could use a fairly old but useful technique to estimate application size: function points.

Function points are a measure of system "size" that does not depend on knowing about the underlying technology used to build the system, so applies equally to packages and custom-build systems. Once you know that a system is, say, 2000 function points in size, then there are well established metrics on how long it costs to replace such a system e.g. for transaction systems, a ballpark figure of 25-30 function points per man month can be delivered, which does not really seem to change much whether it is a package or in-house. Hence a 2000 function point transaction system will cost about 80 man-months to build or implement, as a first pass estimate. MIS systems are less demanding technically than transaction systems (as they are generally read only) and better productivity figures can be be achieved here. These industry averages turned to be about right when I was involved in a metrics program at Shell in the mid 1990s. At that time a number of Shell companies counted function points and discovered productivity of around 15 - 30 function points per man month delivered for medium sized transaction systems, irrespective of whether these were in-house systems or packages. Larger projects had lower productivity, smaller projects have higher productivity, so delivering a 20,000 function point system will be a lot worse than a 2,000 function point system in terms of productivity i.e. fewer function points per man month will be delivered on the larger system. Counting function points in full is tedious and indeed is the single factor that has relegated it to something of a geek niche, yet there are short cut estimating techniques that are fairly accurate and are vastly quicker to do that counting in full. By using these short-cut techniques a broadly accurate picture of an application inventory can be pulled together quite quickly, and this should be good enough for a first pass estimate.

There are a host of good books that discuss project metrics and productivity factors which you can read for more detailed guidance. The point here is that by constructing an inventory of the IT applications of both companies involved in a merger you can get a better feel for the likely cost of replacing those systems, and hence make a business case for doing this. In this way you can have a structured approach to deciding which systems to retire, and avoid the two parties on either side of the merger just defending their own systems without regard to functionality or cost of replacement. Knowing the true costs involved of systems integration should be part of the merger due diligence.

Further reading:

Software Engineering Economics
Controlling Software Projects
Function Points

Conferences and clocks

2006-06-23T01:50:00.000-07:00

Those who are getting on a bit (like me) may recall John Cleese's character in the 1986 movie Clockwise, who was obsessed with punctuality. I am less neurotic, but what does distress me is when conference organizers let their schedule slip due to speaker overruns. I speak regulalrly at conferences, and this is a recurring problem. At a conference in Madrid a few weeks ago they managed to be well over an hour behind schedule by the time they resumed the afternoon session, while the otherwise very useful ETRE conferences are famed for their "flexible" schedule. At a large conference this is beyond just irritating, as you scramble to find speaker tracks in different rooms, all of which may be running to varying degrees behind schedule and starting to overlap.

This poor timekeeping is depressingly normal at conferences, which makes it all the nicer when you see how it should be done. I spoke yesterday at the IDC Business Performance Conference in London, which had an ambitious looking 14 speakers and two panels squeezed into a single day. If this was ETRE they would have barely been halfway through by dinner time, yet the IDC line-up ran almost precisely to time throughout the day. It was achieved by the simple device of having a clock in front of speaker podium ticking away a countdown, so making it speakers very visibly aware of the time they had left. I recall a similar device when I spoke at a Citigroup conference in New York a couple of years ago, which also ran like clockwork.

The conference was a case study in competent organization, with good pre-event arrangements, an audio run-through for each speaker on site, and speaker evaluation forms (some conferences don't even bother with this). The attendees actually bore a distinct resemblance to those promised, both in quality and number; recently some conference organizers seem have had all the integrity of estate agents when quoting expected numbers. The day itself featured some interesting case studies (Glaxo, Royal Bank of Scotland, Royal Sun Alliance, Comet) and a line-up of other speakers who mostly managed to avoid shamelessly plugging their own products and services (mostly). Even the lunch time buffet was edible.

In terms of memorable points, it seems that the worlds of structured and unstructured data are as far part as ever based on the case studies, whatever vendor hype says to the contrary. Data volumes in general continue to rise, while the advent of RFID presents new opportunities and challenges for BI vendors. RFID generates an avalanche of raw data, and a presenter working with early projects in this area reckoned that vendors were completely unable to take advantage of RFID so far. Common themes of successful projects were around the need for active business sponsorship and involvement in projects, the need for data governance and stewardship and for iterative approaches giving incremental and early results. Specific technologies were mostly (refreshingly) in the background in most of the speeches, though the gentleman from Lucent seemed not to have got the memo to sponsor speakers about not delivering direct sales pitches. With Steve Gallagher from Accenture reckoning that BI skills were getting hard to find, even in Bangalore, it would suggest that performance management is moving up the business agenda.

Well done to Nick White of IDC for steering the day through so successfully. If only all conferences ran like this.

Wilde Abstraction

2006-06-21T01:40:00.000-07:00

Eric Kavanagh makes some very astute points in an article on TDWI regarding abstraction. As he rightly points out, a computer system that models the real world will have to deal with business hierarchies such as general ledgers, asset hierarchies etc that are complex in several ways. To start with there are multiple valid views. Different business people have a different perspective on "Product" for example: a marketer will be interested in the brand, price and packaging, but from the point of view of someone in distribution the physical dimensions of the product are important, in what size container it comes in, how it should be stacked etc. Moreover, as Eric points out, many hierarchies are "ragged" in nature, something that not all systems are good at dealing with.

The key point he makes, in my view, is that business people should be presented with a level of abstraction that can be put in their own terms. In other words the business model should drive the computer system, not the other way around. Moreover, as the article notes, if you maintain this abstraction layer properly then historical comparison becomes possible e.g. comparing values over time as the hierarchies change. Indeed the ability to reconstruct past hierarchies is something that I believe is increasingly important in these days of greater regulatory compliance, yet it is often neglected in many systems, both packages and custom-built. The key points he makes on the value of an abstraction layer:

- the abstraction layer shields the application from business change
- business-model driven, with the ability to have multiple views on the same underlying data
- time variance built in
- the layer can be a platform for master data management

neatly sum up the key advantages of the Kalido technology, and indeed sums up why I set up Kalido in the first place, since I felt that existing packages and approaches failed in these key areas. It is encouraging to me that these points are starting to gain wider acceptance as genuine issues that the industry needs to better address if it is to give its customers what they really need. To quote Oscar Wilde "There is only one thing in the world worse than being talked about, and that is not being talked about." I hope these key issues, which most designers of computer systems seem not to grasp, get talked about a lot more.

How to eat an elephant

2006-06-19T06:39:00.000-07:00

Robert Farris makes some good observations in his recent article in DM Review. He points out that many companies have ended up with business intelligence being distributed throughout the company e.g. in various subsidiaries and departments, and this makes it very difficult to take a joined up view across the enterprise. As he notes, disjointed initiatives can result in poor investments. Hence it is critical to take an overall view to business intelligence, yet to do so is such a large task that it seems daunting.

In my experience there are a number of things that can at least improve the chances of an enterprise-wide BI initiative succeeding. It sounds like motherhood and apple pie, but the project needs business sponsorship if it is to succeed; IT is rarely in a position to drive such a project. However that statement on its own is of limited help. How can you find this elusive sponsorship?

The place to start is to find the people that actually have the problem, which is usually either the CFO or the head of marketing. The CFO has the job of answering the questions of the executive team about how the company is performing, so knows what a pain it is to get reliable numbers out of all of those bickering departments and subsidiaries. The head of marketing is the one who most needs data to drive his or her business, usually involving looking at trends over time and involving data from outside the corporate systems, and this is usually poorly provided for by internal systems. The CEO might be a sponsor, but often the CFO will be carefully feeding impressive looking charts to the CEO to give the impression that finance is in control of things, so the CEO might not be aware of how difficult data is to get hold of. The head of operations or manufacturing is another candidate, though this person may be too bogged down in operational problems to give you much time. If there is someone responsible for logistics and supply chain then this is often a fruitful area. Sales people usually hate numbers unless it is connected with their commissions (where they demonstrate previously unsuspected numerical ability), and HR usually doesn't have any money or political clout, so marketing and finance are probably your best bet.

So, you have a sponsor. The next step is to begin to sort out the cross-enterprise data that actually causes all the problems in taking a holistic view, which is these days being termed master data. If you have multiple charts of accounts, inconsistent cost allocation rules, multiple sources of product definition or customer segmentation (and almost all companies do) then it this is a barrier in the way of your BI initiative succeeding. There is no quick fix here, but get backing to set up a master data management improvement project, driven by someone keen on the business side. Justifying this is easier than you may think.

In parallel with this you will want a corporate-wide data warehouse. Of course you may already have one, but it is almost certainly filled with out of data data of variable quality, and may be groaning under a backlog of change requests. If it is not, then it is probably not being used much and may be ripe for replacement. To find out, do a health check. There is a bit of a renaissance in data warehouses these days, and these days you can buy a solution rather than having to build everything from scratch.

In truth your company probably has numerous warehouses already, perhaps on a departmental or country basis, so it is probably a matter of linking these up properly rather than having to do everything from the beginning. This will enable you to take an iterative approach, picking off areas that have high business value and fixing these up first. Once you can demonstrate some early success then you will find it much easier to continue getting sponsorship.

In one of the early Shell data warehouse projects I was involved with we had a very successful initial project in one business line and subsidiary, and this success led to a broader roll-out in other countries, and then finally other business lines came willingly into the project because they could see the earlier successes. This may seem like a longer route to take, but as noted by Robert Farris, this is a journey not a project, and if you start with something vast in scope it will most likely sink under its own weight. Much better to have a series of achievable goals, picking off one business area at a time, or one country at a time, delivering incrementally better solutions and so building credibility with the people that count: the business users.

Elephants need to be eaten in small chunks.

The Informatica recovery story

2006-06-19T03:55:00.000-07:00

The data integration market has previously split between the EAI tools like Tibco and Webmethods, and the ETL space with tools like Informatica and Ascential (now part of IBM). The ETL space has seen significant retrenchment over recent years, with many of the early pioneers being bought or disappearing (e.g. ETI Extract still lives on, but is practically invisible now). Mostly this functionality is being folded into the database or other applications e.g. MSFT with SSIS (previously DTS) and Business Objects having bought ACTA. Still in this space are Sunopsis (the only "new" vendor making some progress) and older players like Iway and Pervasive, whose tools are usually sold inside other products. Others like Sagent and Constellar have gone to the wall.

The integration market is surprisingly flat, with Tibco showing 9% growth last year but a 10% shrinkage in license revenues, while Webmethods grew just 4%, with 1% growth in license revenues. Hardly the stuff investor dreams are made of. BEA is doing better, with 13% overall growth last year and 10% license growth, but this is still hardly stellar. Informatica is the odd one out here, having extracted itself from its aberrant venture into the analytics world and now having repositioned itself as a pure play integration vendor. It had excellent 31% license growth and 27% overall growth last year. The logical acquisition of Similarity Systems broadens Informatica's offering into data quality, which makes sense for an integration vendor. When IBM bought Ascential some pundits reckoned the game would be up for Informatica, but so far that is not proving the case at all.

Microsoft builds out its BI offerings

2006-06-16T06:12:00.000-07:00

A week ago Microsoft announced Performance Point Server 2007. This product contains scorecard, planning and analytics software, and complements the functionality in Excel and in its SQL Server Analysis and Reporting Services tools. With Proclarity also within the Microsoft fold now, it is clear that Microsoft is serious about extending its reach in the BI market.

I have argued for some time that rivals Cognos and Business Objects should be a lot more worried about Microsoft than about each other in the long term. Most business users prefer an Excel-centric environment to do their analysis, and as Microsoft adds more and more ways into this it will be increasingly uncomfortable for the pure-play reporting vendors. As ever, Microsoft will go for high volume and low price, so will probably never match BO or Cognos in functionality, but that is not the point. Most users only take advantage of a fraction of the features of a BI tool anyway.

Microsoft is playing a long game here, and the pure-play tools will continue to do well in what is an expanding market. But the ratchet just got tightened another notch.

Uniting Data

2006-06-14T01:52:00.000-07:00

In an article in DM Review Malcolm Chisholm discusses different types of metadata. He sets out a definition which distinguishes between metadata, master data and reference data (separate from “transaction activity” data). I believe that the argument is flawed in several important ways.

Firstly, I believe that the distinction between metadata, master data, enterprise structure data and reference data as made in the article is actually spurious. One point made about master data is the notion that “Customer A is just Customer A” and here is not more to it than that. However, to the account manager looking after the customer there is a complex semantic which needs data to define it. Well, what if that customer is, say: “Unilever”. There is all kind of embedded meaning about the definition of Unilever that is not directly implied by the row itself, but is defined elsewhere e.g. is that the whole Unilever group of companies, or Unilever in the US, a Unilever factory or what? This type of definitional problem occurs to row level entries just as it does to the generic class of things called “customer”. Master data can have semantic meaning at the row level, just as can “reference data” as used in the article. This point is illustrated further if we use the article’s own example of this: the USA having multiple meanings. Both are valid perspectives for the USA but they are different things – they are defined and differentiated by the states that make them up i.e. their composition. This is the semantic of the two objects.

The article seems to want to create ever more classification of data, including “enterprise structure data”. It argues that “Enterprise structure data is often a problem because when it changes it becomes difficult to do historical reporting”. This is really just another type of master data. The problem of change can be dealt with by ensuring that all the data like this (and indeed all master data) has a “valid from” and “valid to” date. Hence if an organisation splits into two, then we want to be able to view data as it was at a point in time: for example before and after the reorganisation. Time stamping the data in this way addresses this problem; having yet another type of master data classification does not help.

The distinction between “reference data” and “master data” made in the article seems to be both false and also misleading. Just because “volumes of reference data are much lower than what is involved in master data and because reference data changes more slowly” in no way means that it needs be treated differently. In fact, it is a very difficult line to draw, since while typically master data may be more volatile, “reference data” also can change, with major effect, and so systems that store and classify it need to be able to expect and to deal with these changes.

In fact, one man’s transaction is another man’s reference data. A transaction like "payment" has Reference data like Payment Delivery, Customer, Product, Payment Type. A transaction
Delivery from the point of view of a driver might consist of Order, Product, Location, Mode of Delivery. Similarly an "order" could be viewed by a clerk as Contract, Product, Customer, Priority. Where is the line between Master and reference data to be drawn??

The article argues that identification is a major difference between master and reference data, that it is better to have meaningful rather than meaningless surrogate keys for things, which he acknowledges is contrary to perceived wisdom. In fact there are very good reasons to not embed the meaning of something in its coding structure. The article states that: “In reality, they are causing more problems because reference data is even more widely shared than master data, and when surrogate keys pass across system boundaries, their values must be changed to whatever identification scheme is used in the receiving system.”

But this is mistaken. Take the very real word example of article numbering. The Standard Industry codes (SIC) European Article Number (EAN) codes, which are attached to products like pharmaceuticals to enable pharmacists to uniquely identify a product. Here a high level part of the key is assigned e.g. to represent the European v. the US v. Australian e.g. GlaxoSmithKlien in Europe, and then the rest of the key is defined as Glaxo wishes. If the article is referred to by another system e.g. a supplier of Glaxo, then it can be identified as one of Glaxo’s products. This is an example of what is called a “global or universal unique identifier” (GUID or UUID), and for which indeed there are emerging standards.

A complication is that when the packaging changes, even because of changed wording on the conditions of use, then a new EAN code has to be assigned. The codes themselves are structured, often considered bad practice in the IT world, but the idea is to ensure global uniqueness and not give meaning to the code. Before Glaxo Welcome and SmithKlienBeacham merged they each had separate identifiers and so the ownership of the codes changed when the merger took place.

Another point I disagree with in the article is “we will be working with a much narrower scope” in the first paragraph. Surely we are trying to integrate information across the company to get a complete perspective. It is only small transactional applets which only need a worms eye view of what they are doing

The article says “Reference data is any kind of data that is used solely to categorize other data in a database, or solely for relating data in a database to information beyond the boundaries of the enterprise”. But someone in the organization does have to manage this data even if it comes from outside the company and that person’s transaction may be the set up of this data and making it available to others.

For example, consider the setting of a customer’s credit rating. Someone in Finance has to review a new customer’s credit rating against a list of externally defined credit ratings say from D&B. Someone in the company spends time lobbying D&B (or parliament/congress) to have additional credit classifications. (the article defines them as Gold, Silver, Bronze etc. But D&B call them AAA, AA etc.). Data is always created through someone carrying out some business function (or transaction) even standards have to be managed somewhere.

A good example of this type of external data where a computer system is used to support the process is the Engineering parts library. It uses the ISO 15926 standard. It is a collaborative process between specialists from multiple engineering companies. It is a high level classification scheme which is used to create a library of spare parts for cars, aircraft, electronics etc. This is a changing world and there are always new and changing classifications. Groups of engineers who are skilled in some engineering domain define the types and groups of parts. One group defines pumps, another piping. Someone proposes a change and others review it to see if it will impact their business, it goes through a review process and ultimately gets authorized as part of the standard.

This example is about reference data, in the terms of the article, but it clearly has the problem the article attributes to master data. There are multiple versions and name changes and a full history of change has to be maintained if you wish to relate things from last year with things for this year.

The artiicle has an example concerning the marketing department’s view of customer v. accounts view of customer. It says this is a master data management issue and is semantic but this doesn’t apply to reference data. It clearly does relate to reference data. (see definition of USA above) and the ISO example above. But what is more important is that the issue can be resolved for both master and reference data by adopting the standards for integration defined in ISO 15926. Instead of trying to define customer in a way that satisfies everyone it is best to find what is common and what is different. Customers in both definitions are Companies – it is just that some of then have done business with us and others have not (yet). Signed up customers are a subset of all potential customers.

At the end of the section on The Problem of Meaning the article says “These diverse challenges require very different solutions” then in the section on Links between Master and Reference data it says “If there is a complete separation of master and reference data management, this can be a nightmare” and then says “we must think carefully about enterprise information as a whole”. I agree with this final statement but it is critical that we do not put up artificial boundaries and try to solve specific problems with some generic rules which differentiate according to some rather arbitrary definition such as Master and Reference data.

The line between master and reference data is really fuzzy in the definition used. Clearly “Product” is master data but I if have a retail gasoline customer which has only three products (Unleaded, Super and Diesel) I guess that means this is reference data. The engineering parts library classification scheme is a complex structure with high volumes (1000’s) of classes so that makes it master data but it is outside the company so does that makes it reference data?

In summary, the article takes a very IT-centric transactional view of the world. By trying to create separate classifications where in fact none exist, the approach suggested, far from simplifying things, will in fact cause serious problems if implemented, as when these artificial dividing lines blur (which they will) then the systems relying on them will break. Instead what is needed is not separation, but unity. Master data is master data is master data, whether it refers to the structure of an enterprise, a class of thing or an instance of a thing. It needs to be time-stamped and treated in a consistent way with other types of master data, not treated arbitrarily differently. Consistency works best here.

I am indebted to Bruce Ottmann, one of the world's leading data modelers, for some of the examples used in this blog.

Through a rule, darkly

2006-06-13T10:15:00.000-07:00

David Stodder makes a good point in an article in intelligent Enterprise. Business rules take many forms in a large corporation but today they are quite opaque. Rules that define even basic tersm like "gross margin" may not only be buried away in complex spreadsheet models or ERP systems, but are in practice usually held in many different places, with no guarantee of consistency. I know of one company where an internal audit revealed twenty different definitions of "gross margin", and that was within just one subsidiary of the company! In these days of stricter compliance such things are no longer merely annoying.

My observation is that business customers need to take ownership of, and be heavily engaged with, any process to try and improve this situation. It cannot be an IT-driven project. It is not critical whether the ultimate repository of this is a data warehouse, a master data repository or some different business rules repository entirely, but it is key that the exercise actually happens. At present the opaquenes and lack of consistency of business rules is not something that most companies care to own up to, yet it is a major controls issue as well as a source of a great deal of rework and difficulty in presenting accurate data in many contexts.

I was amused by the readership poll quoted that said that 61% of respondents say that they have "no standard process or practice" for business rules management. This might imply that 39% actually did, a number I would treat with considerable caution. Personally I
have yet to encounter any that does so on a global basis.

A marketing tale

2006-06-07T06:35:00.000-07:00

Marketing is a tricky thing. One lesson that I have begun to learn over time is that simplicity and consistency always seem to triumph over a more comprehensive, but more complex story. Take the case of Tivo in the UK. A couple of my friends bought Tivo when it first appeared in Britain and started to have that kind of scary, glazed expression normally associated with religious fanatics or users of interesting pharmaceutical products. I then saw a cinema ad for Tivo and it seemed great: it would find TV programs for you without you having to know when they were scheduled - how cool was that?! It would learn what programs that you liked and record them speculatively for you; you then ranked how much you liked or disliked them and it would get better and better at finding things you enjoyed. You could turn the whole TV experience from being a passive broadcast experience into one where you effectively had your own TV channel, just with all your favorite programs. Oh, and it looked like you could skip past adverts, though of course the Tivo commercial politely glossed over that.

Well, I bought one and I was like a kid in some kind of store. I soon acquired the same crazed look in my eyes as my fellow Tivo owners, and waited smug in the knowledge that I was at the crest of a wave that would revolutionize broadcasting. My friend at the BBC confirmed that every single engineer there was a Tivo fanatic. And then: nothing happened. Those BBC engineers, myself and a few others constituted the entire UK Tivo market - just 30,000 boxes were sold in the UK. Eventually Tivo gave up and, although Tivo is still (just about) supported in the UK, you can't even buy Tivo 2, or even a new Tivo 1 except on eBay.

What happened? The message was too complex. Years later Sky caught on to the DVR concept and brought out the vastly functionally inferior Sky+. How did they advertise it? They just showed a few exciting clips with the viewer freezing and then replaying: "you can replay live TV" was all that was said. This was a fairly minor option on a Tivo that the Tivo commercial barely mentioned, yet it was simple to understand. Sky+ sales took off, and myself and some BBC sound engineers are left with our beloved Tivos, praying that they don't go wrong. It is another Betamax v VHS story, but this time the issue was a marketing one. Tivo still limps on in the US, still growing slowly in subscriber numbers through sheer product brilliance (helped by being boosted on "Sex in the City"), but has clearly not fulfilled its potential.

What this little parable should teach us is that a key to successful marketing is simplicity, stripping everything down to the core thing that represents value to the customer, and then shutting up. With a simple message people can describe the product to their friends or colleagues, and so spread the word. With a complex, multi-part message they get bogged down and so cannot clearly articulate what the product does at its heart. It is so tempting to describe the many things that your product does well, but it is probably a mistake to do so. Find the one core thing that matters to customers, explain this as simply as possible, and repeat as often and as loudly as you can.

Size still isn't everything

2006-06-07T05:26:00.000-07:00

Madan Sheina, who is one of the smarter analysts out there, has written an excellent piece in Computer Business Review on an old hobby horse of mine: data warehouses that are unnecessarily large. I won't rehash the arguments that are made in the article here (in which Madan is kind enough to quote me) as you can read it for yourself but you can be sure that bigger is not necessarily better when it comes to making sense of your business peformance: indeed the opposite is usually true.

Giant data warehouses certainly benefit storage vendors, hardware vendors, consultants who build and tune them and DBAs, who love to discuss their largest database as if is was a proxy for their, er, masculinity (apologies to those female DBAs out there, but you know what I mean; it is good for your resume to have worked on very large databases). The trouble is that high volumes of data make it harder to quickly analyse data in a meaninfgul way, and in most cases this sort of data warehouse elephantitis can be avoided by careful consideration of the use cases,probably saving a lot of money to boot. Of course that would involve IT people actually talking to he business users, I won't be holding my breath for this more thoughtful approach to take off as a trend. Well done Madan for another thoughtful article.

Revenue recognition blues

2006-06-07T02:32:00.000-07:00

Cognos shares have slid nearly 20% in recent weeks as an SEC probe into their accounting continues. The questions raised are in the notoriously tricky area of US GAAP rules, specifically on "VSOE" (or vendor specific objective evidence) which determine how much revenue can be credited for a deal in current figures, and what amount should be deferred. The post-Enron climate has ushered in a much harsher review of software industry practices than was normal in the past, and such esoteric sounding accounting rules can seriously impact a company, as Cognos is now seeing.

Word in the market is that the underlying business is actually quite robust at present, so hopefully this will be a blip for the company rather than anything more serious. Cognos 8 means that there is quite a lot of potential for Cognos to gain revenue as customers upgrade to the new software, which features much better integration between ReportNet and Powerplay, and a complete revamp of Metrics Manager, which is retitled Metrics Studio. These improvements should see Cognos customers steadily upgrading, and so having a positive impact on the company's already pretty healthy finances. However perhaps some more conservative interpretation of US GAAP on their part would be wise.

MDM: comply

2006-06-06T02:18:00.000-07:00

James Kobielus makes an important point regarding master data management - its role in compliance. We know that large companies today generally have many different versions of master data scattered around their organizations: 11 different definitions of "product" on average for example, according to one survey from research firm Tower Group. This of course makes any business performance management question hard to answer: "how much of product X was sold last week" is tricky to discover if there are eleven systems that think they are the master source for information of products. However it may be worse than that: if you are having to produce some report for reasons of regulatory compliance, then such ambiguity may have serious consequences.

In the article James says that "without an unimpeachable official system of records, your lawyers will have to work twice as hard to prove your organization is complying with the letter of the law". Of course the lawyers won't be too troubled about that (all those juicy billlable hours) but business executives certainly need to consider the possible compliance implications of poor master data, as well as its consequences elsewhere.

The patter of tiny pitfalls

2006-06-05T06:07:00.000-07:00

There are some sensible tips from Jane Griffin on MDM pitfalls in a recent article. As she points out, improving your master data is a journey, not a destination, so it makes sense to avoid trying to boil the ocean and instead concentrate on a few high priority areas, perhaps in one or two business units. It would make sense to me to start by identifying areas where MDM problems were causing the most operational difficulties e.g. misplaced orders. By starting where there is a real problem you will have less difficulty in getting business buy-in to the initiative. Be clear that there are lost of different types of master data e.g. we are involved with a project at BP which manages 350 different master data types, and clearly some of these will be more pressing an issue than others.

I have seen some articles where people are struggling to justify an MDM initiative, yet really such initiatives should be much easier to justify than many IT projects. For a start IT people can put the issues in business terms. Master data problems cause very real, practical issues that cost money. For example poor or duplicated customer data can increase failed deliveries, and issues with invoicing. Poor product data can result in duplicated marketing costs, and in some cases even cause issues with health and safety. Problems with chart of accounts data can delay the time needed to close the books. These are all things that have a cost, and so can be assigned a dollar value to fix.

Successful MDM projects will be heavily business-led, driven by the need to improve operational performance. IT staff need to educate business people that there are now an emerging set of solutions that can help, and get those business people involved in owning the data. It is the lack of data governance in many companies that contributed to the poor state of master data in the first place.

How healthy is your data warehouse?

2006-06-01T06:35:00.000-07:00

Not all data warehouses are created equal. Indeed both custom-built and some packaged data warehouse products can have surprising limitations in terms of their functionality. Just as I referred recently to Dave Waddington's excellent checklist of things that would indicate a master data management problem, I would like to propose a series of questions that could be used to assess the depth of functionality of your data warehouse, whether it is custom built or packaged. For this list I am indebted to Dr Steve Lerner (until recently IS Director, Global Finance Applications and Integration at pharmaceutical firm Merial), who was kind enough to set out a series of symptoms that he had found indicated a problem with a data warehouse application. What I like about these is that they are all real business problems, and not a series of features defined by a software vendor or database designer. They are as follows.

1. Do you have difficulty conducting what-if analysis for a variety of business or product or geographical hierarchies?

2. Would it be hard for your current system to determine the impact of a business organization change on Operating Income?

3. Would it be hard for your current system to determine the impact of realigning geographical associations on regional profitability estimates?

4. Do you have difficulty restating historical data?

5. Can you view historic data using both a time-of-transaction basis and a current basis?

6. Can you currently restate historical data using new account structures?

7. Do you have difficulty viewing composites of data from sources with different granularities along key dimensions (i.e., comparing daily sales for a month, to forecast sales done monthly, to your annual profit plan, and to a five year long-range projection)?

8. Do you have difficulty with "bad data" getting into your current data warehouse?

9. Do you have difficulty maintaining the accuracy of your reference data?

10. Do you have difficulty with traceability from source to report?

So, how did your data warehouse application score? If it did not do well (i.e. failed on several of these ten points) then you should be concerned, because business users are likely to do exactly these types of things with the data in the warehouses, if not today then at some point. When they struggle, they will come looking for you.

A potential application of this checklist would be identify the best and worst data warehouses in your company. This type of "health check" could be useful in prioritising future investment e.g. it may highlight that some systems are in urgent need of overhaul or replacement. If you work in an IT department then going to business users with this kind of health check could be seen as being very pro-active and enhance the IT department's credibility with the business. If you are a systems integrator then creating a process for measuring the health of a data warehouse along these lines could be a useful tool that could be sold as a consulting engagement for clients.

Casting spells

2006-06-01T05:42:00.000-07:00

I write this column using some software called Blogger, which is fairly simple to use but is rather limited in some ways, so I am probably going to switch to a more flexible blog editor soon. However one cause of constant entertainment is the Blogger spell check function, which almost makes the thing worthwhile. My typing is erratic at best, so I frequently encounter the Blogger spell checker. At first I found its eccentric suggestions annoying, or even inept, but now I find that they have a certain charm of their own. It takes me back to the early days of word processing, when spell checkers were crude, and their alternative suggestions for one's typographical errors were sometimes wildly inappropriate. Blogger's spell checker recalls that era, as it presents sometimes surreal suggestions for what to a human eye is a pretty easy mistake to spot. For example, if you misspell:

"management" as "managemnet"

then you are presented with two alternatives. Its best guess is "mincemeat", which is somehow appropriate in a couple of cases of managers I can recall, but not really a very likely error. Its only other attempt is "mankind". This is not an isolated case. If you write about federated databases then it is endearing to see the typo:

"federaion" have the two alternatives: "bedroom" or "veteran"

proposed by the beastie. "Bedroom"? I would love to understand the algorithm that came up with that one. I was also impressed by:

"peformance"

Instead of the pretty obvious "performance" it rather sweetly suggests "peppermints".

However my favorite is that if you type "Blogger" as a phrase then not only does it not recognize it. The term "blog" also sadly is a complete mystery to it, which might seem an omission given that it is intended as a spell checker for, er, blogs, or perhaps "blocs" as the spell checker so helpfully proposes. For "blogger"then it suggests the wonderfully ironic:

"blocker"

How true, how true. I would be interested to hear of your worst spell check horror, or indeed of a spell checker whose ineptness rival Blogger's. Any offers?

Woods and Trees

2006-05-31T02:12:00.000-07:00

Janet Kersnar writes a well balanced article which highlights some of the ways in which business intelligence has the potential to become more strategic, but sensibly also points out some of the barriers to this occurring. Certainly today most BI solutions deployed are within departmental silos, and it is difficult for most companies to get a true enterprise-wide view. The insight that an up-to-date, cross-departmental view of business performance can give may lead to dramatic benefits, so we see a renewed interest in deploying data warehouse and related technology to address this problem. Companies that are acquisitive have even greater difficulty than most, since each time they buy a company, it may take years for that company's systems and data to be fully integrated into the new corporation. Again, the latest business intelligence technologies and approaches can help here.

However, as the article rightly points out, it is easy to get carried away with the latest tools, and overlook the very real issues getting people to buy in to new systems. One of the case studies acknowledges that it is easy to go crazy with BI technology and end up with an unfathomable morass of data: "Our stores get ranked on well over 150 metrics on a daily basis.", which is clearly a recipee for inaction and confusion. All the modern technology in the world will not deliver benefit unless it is married to useful business metrics, and unless the business users of the information are fully engaged in the process.

Are the media revolting?

2006-05-30T09:45:00.000-07:00

Joshua Greenbaum writes a thoughtful piece on the clash of the "new media" (blogs, wikis etc) with the mainstream media. He correctly concludes that revolutions rarely go in the directions that are originally intended, and he comes down on the side of the mainstream media camp, who he predicts will subsume the newer media. I agree with his analysis. It is exciting to see new content appearing in blogs on many subjects, but if you actually want to know whether something is true you'd be advised to look at the BBC or CNN. It is positive that the barriers to entry to creating content have dropped away, but media brands will be critical in ensuring reliable, truthful content, as distinct from individuals just spouting off on their latest hobbyhorses.

In fact very few industries have been really demolished by the internet. I heard that there are 10% less people working as travel agents than a few years ago, but there aren't too many others that spring to mind. Even that despised breed, realtors (estate agents in the UK) who essentially just control privileged information, are still very much in business. If the internet couldn't displace them, what chance does it have with journalists?

Data warehouse architectures

2006-05-26T01:44:00.000-07:00

Rick Sherman writes an interesting article about how data warehousing, despite being quite venerable in IT terms, is still poorly understood. He makes a good point, discussing various typical implementation approaches and how thee fail to get to the "single version of the truth" dream. Let's consider for a moment a few architectural choices:

(a) direct access (EII)
(b) data marts only - no pesky warehouse
(c) a single warehouse for the enterprise
(d) a federation of linked warehouses.

The first approach is limited to only a small subset of the reporting needs, and is insufficient to meet most enterprise reporting requirements. To have only single subject data marts was still surprisingly commonly advocated as late as the mid 1990s (born mainly out of the frustration of lengthy or failed data warehouse projects) yet pretty clearly is not going to scale for a company of any size. The sheer number of combinations of data sources required to build the marts means that the problem of resolving inconsistency is being done every time a mart is built, rather than being dealt with in the warehouse, so each mart either becomes a major project in itself, or (more likely) people just give up and go with some data source without getting a complete or even accurate picture.

The single giant warehouse certainly has a lot of appeal, as it resolves the semantic differences of source systems just once, allowing dependent data mars to be deployed easily. The trouble is one of practicality: for a large corporation the sheer scale of the task is scary. Large enterprises have hundreds (and usually thousands if they are counting properly) of applications where data is being captured, and these applications are often duplicated by country or major business lines. Hence the sheer scale of getting hold of all these sources and bring them into line is going to be a massive challenge. In the cases of certain industries (retail, Telco, retail banking) the scale of the data itself is also daunting, bring major technical challenges.

Hence for any large corporation it seems to me that a federated warehouse approach is what you will end up with, whether you like it or not. Few companies will have the energy or resources to deliver the single giant warehouse, and even those few that do will, in reality, have a series of skunk works data marts/warehouses dotted around the corporation since such a behemoth warehouse will be a bottleneck, hard to change and inevitably slow to respond to rapidly changing business needs.

The most pragmatic approach would seem to me to acknowledge this reality and architect for a federated approach, rather than staying in denial. It is practical to build a warehouse for either a country-level subsidiary (or groups of countries) or each business line, let that deal with the needs of that particular country or business line, and then link these together to a global warehouse which deals at the summary level. The global warehouse does not need to store every transaction in the enterprise; at that level you need to know what the sales were in Germany yesterday by product, channel and perhaps customer, but not that a particular customer bought a specific item at 14:25 at a store in Rhine-Westphalia. The detailed information like this is the domain of the country-level warehouse. Because the transaction detail is not needed at the enterprise level, you avoid the problems of technical scale that may otherwise occur, and only deal with the data that makes sense to look at across the enterprise as a whole.

Putting a little glitz into data warehousing

2006-05-24T10:03:00.000-07:00

Data warehouse technology is rarely associated with the glamorous world of mergers and acquisitions, usually the domain of sharp-suited investment bankers, late night board meetings, top lawyers and sometimes bizarre behaviour (see Barbarians at the Gate). But once the deal is signed, the party is over and the bankers and lawyers get their modest fees, what happens next? You can be sure that there is a hard-pressed person, possibly the CFO, who is put in charge of delivering all the vast synergy benefits that were promised by the chief executives to their shareholders. Do not envy this person. According to Deloitte: "Between 50-70% of mergers fail to deliver shareholder value after the deal." Moreover, to emphasize just how important quick results are, according to Accenture: "For an acquirer expecting to reap $500 million in yearly cost savings from an M&A transaction, a one-month delay reduces the net present value of the deal by more than $150 million (assuming a 10 percent cost of capital). A seven-month delay costs nearly $1 billion in lost value, or approximately $3.5 million per day."

Given this type of background, it is perhaps understandable that the first thought from the woefully underpaid consultants from the big systems integrators is not "let's build a data warehouse then". Yet understanding the cross-enterprise picture is immediately critical realizing benefits. For example, when HBOS merged, one of the key areas for quick savings was identified as procurement. Yet to just pick one of the existing procurement systems, switch off the other and convert all the data from one to the other was estimated at taking well over a year. Instead, what they did was to implement a packaged data warehouse, map the two sets of data from each bank, and in this way get a single view of the procurement spend across torganizationion without having to convert all the data in the underlying systems. This was achieved in just three months, giving an immediate view of post-merger procurement that allowed huge business savings to achieved. For more on this award-winning project click here.

In this way HBOS cleverly avoided a common trap, neatly summed up by McKinsey: "To succeed, a merger requires the smooth integration of IT systems and services, but the task often plunges the CFO responsible for ensuring the savings into uncharted territory. Confronted by an immediate technical challenge, companies typically choose one of two questionable routes. Some, fearing costs and complexity, never fully integrate their acquisition's systems and thus gain few synergies. Others focus on the promise of synergy gains and improved performance but, in their haste, simply choose one system over another, often alienating both customers and employees."

Other companies that have successfully used a data warehouse in this fashion are Shell, Intelsat, Unilever and Cadbury Schweppes. What is critical in such cases is the need for the warehouse to be able rapidly deployed, so that the business can see quick results. For those toiling away on an unrewarding data warehouse project, remember that next time your company buys another, a data warehouse could be a key part of the solution. Just ask HBOS.

But how do I explain MDM to a business user?

2006-05-24T02:13:00.000-07:00

There is an excellent article today by Ventana analyst Dave Waddington on how to tell if you have a master data management problem in your company. He sets out no fewer than 17 symptoms that would indicate that your master data is not fully under control. The beauty of this article is that it takes a business viewpoint and lists a series of different issues that will resonate with business executives; so many articles on MDM are written by people who have a pure technology problem, but Dave is that rare breed: someone is an expert in technology who worked for many years at Unilever, so has excellent business grasp. Dave also happens to have an unusually sharp mind.

His checklist is an excellent way of engaging with business people to try and put across the concepts of master data management in language that they will understand, rather than discussing hubs and metadata repositories. There has been much written on how difficult it is to justify master data initiatives, and yet if you run through this list of potential issues it should be possible to at least estimate a dollar cost associated with these problems, which is the first step to justifying a project that the business will support. The sorts of issues listed e.g.

"You struggle to determine total product sales to global customers"

is exactly the kind of problem that I recall the business struggling with when I worked at Shell. Shell sold products to Ford motor company, which of course in reality trades under multiple subsidiaries, and moreover different IT systems have different codes to describe "Ford". This is not just an abstract issue: an account can be lost if the customer does not feel that you are able to deal with them consistently on a global basis, yet doing so is a major challenge for most companies. Working out the loss of revenue if a major global account defected to a rival should rapidly justify an MDM initiative.

Complexity conservation

2006-05-23T05:09:00.000-07:00

There is a thoughtful article by Mike Garrett which has the original notion that the amount of complexity in an information system is constant, and that you can only move around the complexity e.g. by putting more effort into the back-end system to shield some complexity from the end user. His article is discussing SAP BW but I like the idea. It is certainly true that in a truly bespoke reporting system everything is shielded from the user, but that requires huge IT resources, while just plonking a complex reporting tool on the user's desktops and hoping they will make sense of the data reduces demand on IT but is pretty much doomed because must users won't be able to navigate the complexity of the database (especially for something as complex as SAP, with 32,000 tables and counting).

However I do believe that a reasonably happy medium can be found if a layer is presented to the end user that uses their own terminology and hides the physical database implementation. This, after all, was what made Business Objects so successful with its "semantic layer". However this certainly passed some complexity back to the IT department, who then had to spend significant effort in building and maintaining the Business Objects "universes". If you go one step further and have a data warehouse that is driven from a business model, then this itself can generate a meaningful environment (such as a Business Objects universe) from the warehouse. This is what Kalido does, for example. Again, it is fair to say that the complexity has not entirely disappeared, but now some effort is needed to build the business model in the data warehouse. However where I think the article's neat idea falls down is to assume that the magnitude of the complexity is always the same. It seems clear to me that there is more effort in customizing every report to an individual user than there is in delivering an environment like a Business Objects universe (or even a carefully built Excel pivot table) which gives the end user a fair degree of freedom of formatting a and exploration. There us still less effort involved if you push the effort back one layer into the warehouse, since the warehouse can then generate multiple Business Objects universes, and not require each one to be customized by hand. Hence the further back in the stack you start the business modeling, the more ripple-through benefits you get by having less separate things to customize by hand. The end user still gets a flexible environment in terms that are meaningful, but there is less effort in total in delivering this environment. Further benefit would occur if all the operational systems were business-model driven, but this is just a pipe-dream today.

Complexity is one thing that should not be conserved if at all possible. Driving things from a business-model as far back in the stack as practical won't make complexity extinct, but may at least make it a little endangered.

The weakest data link

2006-05-19T07:22:00.000-07:00

There is a thoughtful article in McKinsey quarterly on managing supply chains. It highlights the problem that even if you have perfectly consistent and accessible information in your company, in many situations e.g. with mobile phone, there is a web of separate companies between the designer and the customer e.g.

components supplier -> distributor -> ODM -> OEM -> distributor -> customer

Each of these is dependent to some extent on the other, and so if you want to know how your sales are going or how is product quality, you will want to interact with information from other companies further back in the chain. This presents the problem that the systems in other companies will not use the same terminology and coding structures as yours, meaning that you will need to resolve these differences in some way e.g. through a data warehouse project. The article points out that in many cases companies have not built these links and so have no visibility up and down the supply chain. This information is not just nice to have:

"Bridging these gaps pays off. In one case, a leading enterprise-computing company started gathering better data from field services, which gave it information on the incidence of failures and their costs. By feeding that data to design teams, the company developed products that could be serviced and repaired more easily. The result: total costs over the product life cycle fell by 10 to 20 percent."

Clearly such savings are worth having. The article is an excellent illustration that the issues of dealing with multiple semantics are not confined to internal systems, and indeed in such cases standardization is literally unattainable. Instead software solutions are required that can map multiple business structures together and make sense of them. Companies that invest in such data warehouse solutions are, as this article shows, getting very tangible results.

Enter the Dragon

2006-05-19T01:55:00.000-07:00

I spent the last two weeks on holiday in China. Apart from the awesome sense of history (the Great Wall is 3,000 miles long and was completed n 220 BC) it was intriguing to get a sense of one of the world's two great emerging economies. Shanghai was striking in this regard. In just 20 years since Deng Xiaoping's reforms Shanghai has been transformed into the most dynamic of ultra-modern cities. There is a striking symbolism in standing on the Bund (one side of the city's Huangpu river) amongst the fine 1920s and 1930s building built mainly by the British, and looking out across the river at the future. On the opposite bank is Pudong, a sort of Canary Wharf on steroids, a city of gleaming steel and glass. The sheer scale of Pudong is best appreciated from the Grand Hyatt hotel, the tallest hotel in the world at 1,380 feet. From either the 54th floor lobby or the 88th floor (8 is a lucky number in Chinese culture) bar you look out across at the old Shanghai, but also at the forest of skyscrapers that is Pudong. A quarter of the world's cranes are at work here, to give some sense of scale. The desire to create an image of progress is epitomized by the Maglev train, which whisks you from the town to the international airport at a top speed of 266 mph (431 km/h). It can go at 311 mph (501 km/h), but at its slower cruising speed still does the 30 km journey in well under eight minutes. Symbols are important, and the Maglev stands in striking contrast to the shambolic infrastructure of India's airports and trains. India does have the key advantage of widely spoken English, but China's modern infrastructure wins hands down. One danger to Western companies is also apparent in the Maglev. Built on German technology, China now intends to build a far longer Maglev track to Hangzhou, but will build it on Chinese technology: quick learners, or intellectual property theft? Conversations I had when in China suggested that intellectual property rights are an alien notion in China, at least for now; our guide in Beijing ran a web site selling fake Rolex watch mechanisms which can be made up into expensive replica watches. He was simply bewildered at the notion that there could be anything wrong with this.

However, despite this, China is now the world's largest exporter of hi-tech products. When you are there you can sense the sheer dynamism of the place in the air. As an example, just today Teradata announced that their new R&D centre was to be based in Beijing.

A short intermission

2006-05-03T10:27:00.000-07:00

I am just off on vacation for a couple of weeks, so the blog will be quiet for a while. Normal service will be resumed on my return.

Searching for an MDM strategy

2006-05-03T09:00:00.000-07:00

I saw a curious article called "Informatica addresses master data management" in which I expected some sort of product announcement or acquisition that would launch Informatica into the MDM space. Yet you can scour the article for as long as you like in search of anything resembling a product announcement. It seems that Informatica "supports" MDM, which is fair enough in that they of course provide one of the main data integration technologies out there, and so indeed can move master data (amongst other data) around. However they had the exact same technology yesterday, so exactly what had changed?

It seems to me that Informatica is crying out for an MDM strategy of some kind, perhaps via a partnership or even an acquisition (though most of the juicy MDM titbits like Razza have already been gobbled up). Given that Informatica has data quality capability via the Similarity Systems acqusition, and its focus on data integration, it would be a natural extension to move into MDM proper. So, when will the other shoe drop?

CDI compared to other master data

2006-05-02T09:36:00.000-07:00

There is a good article on CDI by Jill Dyche, a co-founder of Baseline Consulting and someone who has clearly seen a lot of real-world CDI projects. She does a good job of explaining how CDI projects have traditionally been quite transaction-oriented, with hubs serving up customer data via middleware to other applications. CDI hubs are at one end of the MDM spectrum, firmly at the "operational" level. At the other end are "analytic" MDM applications, which enable companies to take a cross-enterprise view of key information like assets, people, products, channels etc. Getting to understand the differences between the multiple, conflicting definitions embedded in the source systems is a major job in itself, and will usually result in a master data repository. This in turn can be a feed into a corporate warehouse. A few pioneering companies have taken the final logical step and hooked up their master data repositories, via middleware like Tibco or IBM Websphere, to their operational systems, so that the master data repository becomes the true master source, driving changes as required back down into the operational systems like ERP and CRM.

CDI hubs have started at the other end, linking up to systems providing customer data, often in real-time. Customer data represents a high-value area of MDM, as in the case of consumers the customer data is often quite simple, but is in high volume, and requires fairly simple processing to match a customer record in one system to one in another (e.g. matching "A. Hayler" v "Andy Hayler"). However, this is only part of the answer, as even in the case of "customer" things can get more complex. Suppose you are a company like Shell and you want to treat Unilever as a key global account. Finding out all the information about Unilever is not just a simple keyword matching exercise, since Unilever trades under many different subsidiary names and brands around the world e.g. its main Indian subsidiary is not called Unilever but Hindustan Lever; it also owns a company called Algida, and I defy even the cleverest fuzzy logic algorithm to associate "Algida" with "Unilever" (such examples are why you should always be sceptical about vendors selling matching algorithms) It can be seen that, for more complex situations like this, human intervention is required in order to correctly add up all the element of Unilever's business.

This issue can become considerably more complex with things like "asset" or "product", which can have a whole hierarchy of sub-types. This is why CDI hub technology tends to be used specifically for consumer information. Other types of MDM technology are required to manage more complex data and the workflows that surround the updating this e.g. no automated system is going to just create a new brand; this requires numerous approvals and has various knock-on effects to other master data.

I would argue that, at least at present, you are likely to require one kind of technology to handle general purpose MDM data, whether customer or asset or whatever, from an analytical viewpoint, and potentially a separate technology to handle real-time updates, perhaps real-time. Of course it would be nice if a single product did everything, but at present nobody can truly claim this. What does seem a missed opportunity is the way that vendors have made their technology so very specific to particular types of master data e.g. PIM and CDI. While operational and analytic needs are inherently different, there is no reason at all not to take a generic approach to all types of master data. Customers can hardly be expected to buy a separate hub for every type of master data.

One more TLA to remember

2006-05-02T02:40:00.000-07:00

EIM is a recent Gartner market positioning which is an umbrella term for business intelligence, master data management and content management. While there is a certain inevitable "not another acronym" reaction, this particular one makes quite a lot of sense to me. Gartner have sensibly made the term explicitly cover business processes rather than just technology, so that data governance and stewardship would be part of this broad area. As the Gartner notes say, data integration is at the heart of this.

I think this is positive because the industry has taken an overly technology-centric perspective this so far. Technologies such as ETL are necessary but not sufficient to deliver a broad-based understanding of corporate information. I have observed some forward-looking companies setting up new organizations to manage information, staffed with mainly business rather than IT staff. The groups have the remit to cover the provision of data as a service to the rest of the enterprise, and so they have to worry about data quality, data warehouses, master data, integration middleware and all the processes that go along with it: indeed, this is pretty much a definition of what EIM is all about. Taking a holistic, business-led approach is the right thing to do, since providing high quality, timely data requires a level of business ownership that cannot just be delegated to the internal IT department, or out-sourced to India. The various supporting technologies need to do just that: support business rather than being ends to themselves.

It will be interesting to see how this new terminology catches on, but I think it has legs since it seems to me to incorporate a lot of common sense.

Are you an MDM sinner?

2006-04-28T09:29:00.000-07:00

There is a lot of common sense in the article about the "seven deadly sins" of MDM according to Knightsbridge. In particular:

"Believing that complete enterprise consensus is possible. Master data will exist in a constant state of evolution. It must be accepted that there will never be a point in time at which all business units are in agreement with and have completely implemented master data for all key subject areas."

This may seem obvious but it is a critical point which businesses time and again fail to grasp, waiting for some mega central project to deliver this nirvana, which will never happen. Master data will NEVER be standardized across an entire corporation. You can improve the degree to which different versions are around, but the actual goal of harmonized master data is always an over-the-horizon goal. For a start, business change too rapidly to expect anything else. Even if by magic you could unify all your master data tomorrow, within a short time there would be a major change e.g. an acquisition of another company, which will upset that pretty picture; the new company's master data will clearly not be the same as yours, and the operational systems at least cannot just be switched over without a protracted systems integration project. Hence you must always have to deal with the situation where master data is in multiple places. What you need to do is to be aware of that and to put in place processes and workflow that manage the situation, rather than assuming it will one day go away.

Knightsbridge are a very experienced firm that have more experience of MDM than most and Faisal Shah is a smart guy: their whitepaper is well worth reading.

MDM in the real world

2006-04-26T06:36:00.000-07:00

Master data management has become remarkably trendy now, with software vendors lining up to re-badge their offerings as relating to master data in some way. Some of these offerings run best on the overhead projector, and it is relatively hard to find customers who have actually finished an MDM project and been live for some time, rather than those have completed pilot projects or are "investigating" the issue (i.e. attending conferences in nice sunny locations)

Irrespective of your technology choice, if you want to look at MDM at the sharp end then you could do worse than read a Ventana report which documents several KALIDO MDM customer case studies bsed on in-depth interviews of these customers by Ventana analysts. The case studies show a number if important points whatever technology you are using e.g. the sheer breadth of master data that is out there just waiting to be managed. One bank implemented an MDM system to manage the various versions of statutory accounts that are submitted by its numerous subsidiaries, and which have to be carefully consolidated from a compliance viewpoint. As you can imagine, there is a lot of complexity in the master data associated with a chart of accounts for a massive international bank. This is a long way from the customer hub stereotype that many peopel have of MDM projects.

Another customer manages 350 different types of master data, showing that "customer" and "product" are just a small part of the problem. An issue discussed in the report is the need to set the MDM project in the context of a data governance initiative, for example defining roles for data "customers", "administrators" and "stewards", each of whom have different responsibilities. Any MDM project will have to address the organizational issues and roles of this type, and the issues encountered will be similar, even if the Kalido-specific elements of the report is not of interest to you.

And you thought Sarbanes Oxley was bad

2006-04-24T08:32:00.000-07:00

John McCormick raises a spectre more suited to Halloween than spring: the possibility that CIOs in public companies could be required by the US government to certify that the information they supply to governemnt agencies is correct. As the article points out, given that much of the information stored in corporate systems is wrong or incomplete (the article quotes a Gartner guesstimate of 25% being in this category) this could result in a few worry lines on CIO foreheads.

I think that, at least for now, this is scaremongering. The US government is aware of the considerable backlash against Sarbanes Oxley from business, and indeed some minor softening may occur in due course. The notion that they would compound this massively by asking for certification of all data in a company, something that is manifestly impossible anyway, seems far fetched. Even if the governemnt were "selective" as the article suggests, then presumably areas of interest would tend to be things which are currently carefully regulated anyway e.g. FDA documents in the pharma industry, or information in defence companies. Even the current administration would hardly try to put in regulation that would allow government agencies to go on fishing trips through corporate data, and then demand certificatoin that what they found was right.

Or would they?

Hyperion goes shopping

2006-04-24T05:53:00.000-07:00

In Hollywood there is something of a herd mentality. If one studio brings out a gladiator film (say) then the others feel obliged to do likewise. Software is something of a fashion business, and so some similar behavior occurs. It seems like the indispensable fashion accessory at the moment is a data quality vendors: one simply can't be seen out without one, darling. Hyperion showed off its latest purchase this week: Upstream software, a small data quality vendor from Michigan. As in fashion, the price is so much more tantalizing if it is only available to those in the know, and so the terms of the deal were not announced. I believe that Upstream's revenues were about USD 8M and had about 30 employees. At some point Hyperion will presumably have to come clean, since they are a public company. Recently data quality vendors have been snapped up at bargain-basement prices, and certainly Hyperion has the sort of bank balance that it could pay fro Upstream out of loose change even if it paid a full price. However its most recent results show a certain amount of stumbling. Even though profit margins were still strong at 16%, but license revenues (the key to long term health for a software vendor ) actually fell by 6% on a year over year basis. Nonetheless, Hyperion has an enviable franchise as the undisputed king of financial consolidation, and is highly profitable.

The deal actually makes good sense to me: financial consolidation is Hyperion's core business, and those of us who work in the industry know just how flaky the quality of data can be in those superficially shiny ERP finance systems. Hence a data quality spin makes good sense for Hyperion's message to nervous CFOs. Unfortunately data quality is a very intractable problem, involving as it does human nature, and even the cleverest software can only assist with fixing issues of this nature. The data quality software vendor market may be shrinking, but underlying the problem of data quality itself is not.

Psst, wanna buy a BI vendor?

2006-04-20T02:48:00.000-07:00

In Enterprise Systems, Stephen Swoyer concurs with my view that Oracle's recent BI announcement was a big yawn. Moreover, he raises the interesting possibility of IBM buying Cognos, a rumor that has been circulating in the industry for several months. IBM has been strictly agnostic with regards to getting into applications, but that would not stop it buying a pure-play BI vendor. Clearly it has the financial resources to do what it wants, and I feel that Cognos would be a slightly more natural fit for IBM than Business Objects, as Cognos plays a little higher up the corporate food chain e.g. with its Adaytum budgeting acquisition. Of course IBM did buy Alphablox, but this was a small vendor that had some technology useful to IBM, so that did not really alter the big picture. Cognos is getting back on its feet after a stumble late in 2005, and has generally strong technology.

If you consider the broad landscape, the industry has its "stack wars", with IBM, SAP (Netweaver), Oracle (Fusion) and Microsoft. IBM is, in my view, the best placed here since it is the only one of these that does not have application turf to defend (even Microsoft has its applications to worry about, and like errant teenagers to its parents, a big worry they are). IBM has planted its feet firmly in the master data management landscape, and to me it would seem entirely complementary and consistent for it to make a big play into the business intelligence world. Acquiring a company the scale of Cognos would certainly be the kind of bold step that would demonstrate that IBM is very serious about this area. It would firm up its "stack"offering in a quite natural way. Of course I am not privy to the internal musings of IBM, but this rumor makes more sense than many.

Lies, Damned Lies and Surveys

2006-04-10T04:26:00.000-07:00

A survey sponsored by Oracle (http://www.computing.co.uk/itweek/news/2153695/surveys-show-bi-failing-s) hits a new low in terms of insight. The classic line is: "In Oracle's survey of 200 UK and Irish IT managers, over half of organisations said they did not have any BI systems, though 69 percent of respondents said BI was important to help senior managers run their business." Apart from the apparent conclusion that over 20% of the respondents seem to struggle to keep two ideas in their head for more than five minutes, the notion that half of the UK's companies lack a single BI tool is pretty absurd. We have had well over a decade of Business Objects, Cognos and others jamming BI tools, and even before that there were tools like Focus and Nomad. You would have to be recently returned from the moon not to have encountered a BI software salesman as a UK IT manager.

I do wonder sometimes about the accuracy of some of these surveys. I recall years ago at a Gartner conference being handed a thick survey, which demanded all kinds of detail in terms of IT budget breakdown, future spending trends by area etc. You needed to return the completed survey in order to get a chance of winning a prize, and I remember saying to a guy next to me who had just finished his "how on earth do you remember all of that budget info for your organisation?" The reply was "are you kidding, I just made it up, but I really want that prize". Many surveys do make use of incentives to get people to fill them in, and I wonder just how accurate the data really is in many of them as a result.

Separately, a more plausible insight in a different survey is: "Meanwhile, a survey of 1,000 UK business managers at companies with over 250 staff, published by ICS, indicates a widespread need for better BI systems. The study found that over three quarters of respondents were forced to make decisions "blind" due to late or insufficient business information". By contrast, this is entirely believable, though not for the reason that the article gave. The critical issue is that you can have as many pretty reporting tools and dashboards as you like, but you need accurate and timely information to feed those systems coming from a data warehouse (unless you are one of the few brave souls using EII). The problem is that most data warehouses are entirely unable to keep up with the pace of business change (reorganisations, acquisitions etc) and so are constantly out of date. Consider a data warehouse with just ten source systems. A major change in one of its sources will impact the warehouse schema, and may take three months to fix the schema, the load routines and the reports that are impacted by the change (this is a pretty typical figure in my experience at Shell).

A major change of this type does not happen every day, but is almost certain to happen once a year to each of these source systems, maybe twice. There are then ten sets of separate changes, each taking three months worth of changes needed to the warehouse every year. Even assuming that the changes are neatly spread over the year and that you have plenty of programming resources to fix the changes, so you can do these in parallel, you still have 15 months of change to fit into 12 months; basically the warehouse can never catch up. You may well have more than 10 sources for your data warehouse, so the problem could be even worse than this. This is indeed what happens in reality: the data warehouse is usually out of date, so armies of Excel jockeys in finance get the answers via email and have to manually number-crunch for anything really critical while the warehouse lumbers on with out of date information. This situation is not the fault of the BI tools - it is the fault of the data warehouses that feed the BI tools. Until companies admit that the status quo is failing and start abandoning custom-build warehouses this problem will persist. It is like with treating alcoholism: the first step is admitting that there is a problem.

Honey I shrunk the attendees

2006-04-05T20:37:00.000-07:00

I was in Silicon Valley this week speaking at Software 2006. This was pitched as having 2,500 attendees, and the organizers claimed 1,700 on the day itself, yet at any plenary session I could only count about 400 or so. Indeed on the first morning the main hall was so awkwardly empty that the number of chairs was dramatically reduced for future sessions, presumably to make it seem fuller. This is getting silly, rather like the perennial numbers game between police ("10,000 protestors") and demonstrators ("100,000 protestors") played out in countries the world over. As noted previously the trade show seems to be in secular decline, even here in the heart of hi-tech country. The show itself had good speakers and excellent conference admin, yet the partly deserted exhibit hall spoke volumes. Even the bikini-clad girl handing out free gifts (note to the marketing manager at Aztec who hired the model: you may want to consider trying a gimmick that does not look quite so tacky; even in the 1980s this seemed a bit dubious) was unable to lift the atmosphere. The exhibit sponsors did not seem best pleased (sample comment: "four of out of five people who came to the booth were trying to sell us something rather than the other way around") and the supposed legion of CIOs attending were either cunningly disguised or further optimism on the organiser's behalf.

Ironically the conference hotel perhaps held the clue as to why attendance at trade shows seems to be so hard to drum up these days. If you go to the Hyatt Regency Santa Clara concierge desk you are greeted not by a person but by a video screen. The concierge herself (the helpful Anna) sits 80 miles away and chats to you, even able to print out directions on the printer at the concierge desk. If even the hotel concierge can't be bothered to travel to work any more and can do her job quite adequately by video link, is it any wonder that busy executives spend less time at trade shows and more time on webinars?

Integration or management?

2006-04-05T14:34:00.000-07:00

I enjoyed a thoughtful article by Colin White, long-time database expert and general guru, about data integration and MDM. In it he defines an architecture for data integration that splits out the technologies from the different techniques and supporting applications. He also points out a couple of important things: one is that CDI is not a very useful term, since it implies that it is only about integrating customer data. This is very true, and applies to MDM in general. It is not the case that CDI just means taking half a dozen separate customer data sources and somehow banging them together. In reality it will not be practical to end up with one master source for customer (or pretty much any master) data, so what is important is to be able to to catalog what is out there, map the differences in definitions so that sense can be made of these differences, and a process defined so that changes can be propagated in a controlled way to the various sources. This is much more than just synchronizing updates between SAP and Siebel. Beyond simple things like names and addresses, you also need to consider how customer data is to be classified e.g. into different a market segments or demographic groups. Changing this classification is a non-trivial business process that will require input from various people within (and possibly beyond) marketing, and will likely involve multiple versions that need to be discussed, tested and modified before being published, and then propagated into the various operational systems. As Colin says, "management" is a much more appropriate term here than "integration": this may seem an esoteric point, but names matter (if you doubt this, ask Vauxhall, whose "Nova" car means "no go" in Spanish)

Another point well made by Colin is how the term "real time" is regularly abused. Since most business intelligence requires some form of analysis, it is rare indeed that it needs to be truly "real time" e.g. looking at the buying patterns in a retail store by branch may usefully show all sorts of things (which items are moving, which promotions are working etc) yet this information has no more meaning of you get it at 14:15 than if you get it at 14:05, or indeed at 11:32. Having it a few minutes more "real time" adds no meaning, yet will cost dramatically more in terms of IT complexity and cost. I would argue that there are very few things indeed in business intelligence that truly require real-time data feeds. Certain operational queries may need this e.g. checking a customer's credit rating, or looking at overall trading exposure before placing a trade, but these are a small subset of what is usually termed business intelligence.

True love

2006-04-04T07:06:00.000-07:00

After a long courtship Microsoft today announced its engagement to its long-time lover, ProClarity. ProClarity had long been in a monogamous relationship with Microsoft as a partner, and had clearly been trying to win Microsoft's heart ever since they first met. This s a excellent example of power dating in the software industry, as Proclarity had seemingly tied its fate to Microsoft as a conscious strategy for a long time. Proclarity had revenues of USD 15.1M in 2005, but had barely grown in 2005 over 2004 (just 2% growth) and at 135 employees was probably not profitable. Growth of 2% and losing money is not a healthy position for Proclarity or any other software vendor, so its new position as a blushing bride is a sound move for the company. Financial terms of the Microsoft dowry were not obvious from the announcement.

Microsoft gains a strong reporting technology and moves ever further into direct competition with the mainstream BI vendors Business Objects and Cognos, a trend noted previously in this blog.

A bit poor

2006-03-31T07:25:00.000-08:00

You may recall my blog on SAP's farcical claims about its software's impact on company profitability. It looks like someone with more time on their hands than me actually checked up on the figures and found these lacking, in addition to the lack of logic in the original claim. Nucleus Research, who are noted for their rigor with numbers, found that in fact that SAP customers (identified by being listed on SAP's web site) were 20% less profitable than their peers, rather than 32% more profitable. Of course this is not quite the same thing, but it is amusing: it suggests that only SAP's identified reference customers are relatively unprofitable. Perhaps the ones who keep quiet are doing OK? As I noted earlier, the SAP claim was deliberately skewed to exclude all financial institutions (which share the twin characteristics of being highly profitable and rarely using SAP) while anyhow the notion that the choice of your ERP systems provider is a cause of either good or bad profits is both logically flawed and also deeply amusing to those of us who have watched companies spend billions implementing SAP to little obvious effect in terms of hard business benefits.

Good on Nucleus for poking further holes in this especially egregious piece of over-marketing. Bruce Brien, CEO of Stratascope, the company that did the market research for SAP, reacted by sayng:“They’re making an implication that my numbers can’t prove, but it’s a marketing message. Companies do that all the time,” he says. Oh well, that's all right then.

Cognos recovers somewhat

2006-03-31T06:44:00.000-08:00

Cognos announced its full year results, notably seeing a recovery in license revenues to USD 118M in their fourth quarter (i.e. Q1 2006) after the disappointing Q4 2005 results. It was also important to note that the company closed 18 deals over a million dollars in size, which was another marked improvement on the previous quarter. Profit margins were a healthy 18%. Still, license revenue was actually down compared to the same quarter a year ago (USD 130M) while overall revenues at USD 253M for the quarter was slightly down on the same period last year. Actually shrinking is not generally a cause for celebration in a software company, so it is a measure of just how bad Cognos' previous quarter was that these results were generally greeted with relief.

This (relative) recovery all bodes well for the broader sector, and indicates that Cognos' stumble at the end of 2005 was to do more with company-specific issues (limited deployment of its new product line) than with any general slow-down in the business intelligence market (which just about every analyst predicts will grow at a healthy clip in 2006). In the medium term, Cognos faces the same issues as other BI suppliers: the relative saturation of the market, and the ever-growing threat from Microsoft.

The ratchet goes up a notch

2006-03-31T01:21:00.000-08:00

Back last year I wrote about the creeping progress of Microsoft into the business intelligence arena. In CBR Madan Sheina, (one of the smartest analysts in the industry by the way), examines the latest move in this direction, the SQL Server 2005 suite's enhanced business intelligence offerings. The new ETL offering SSIS (previously DTS) will be of interest, although its SQL Server ties may limit the take-up of this relative to database-neutral offerings. However the new Analysis Services and Reporting Services promise to ratchet up the pressure on the pure-play BI players, Business Objects, Cognos and the rest. I have long argued that the most ubiquitous BI tool is actually Excel, and that given that people already know this, an ideal BI tool for many users would be one which magically got the data they wanted out of a data warehouse directly into an Excel pivot table. Yes, there will always be a subset of power users for who this is not enough, but in the vast majority of cases this will actually do the trick. Other tools (visualization, data mining etc) would be relegated to niches if this were to happen significant niches perhaps, but niches nonetheless.

Business Objects has done well because of its semantic layer, the "universe", which overlays something closer to a business view on top of data marts and warehouses; this imposes some maintenance overhead but this is acceptable to users since it represents the data in a more business-like form. However Business Objects has always struggled with its OLAP capability relative to competitors. Cognos by contrast, had the best OLAP tool out there in Powerplay, but a rather ordinary reporting offering. These two vendors pretty much carved up the market between them, though in a growing market there was enough room for other tools like Microstrategy, Actuate etc as well. Microsoft's new suite poses a potent threat to most of these BI vendors, since most users do not use more than a tiny fraction of the features of a BI tool, so adding more features just to stay ahead of Microsoft is ineffective; the end users simply don't need more features. With its low price point and "good enough" features, the Microsoft tools are likely to gradually eat into the market share of the independent vendors. Nothing dramatic will happen overnight, and the curious restraint of Microsoft from serious marketing of its tools to the enterprise will also slow progress. What was the last time you saw a webinar or advert for Analysis Services? Compare and contrast with Business Objects, which is a marketing machine.

However, just like a pack of hunting dogs wearing down a large prey animal, the Microsoft tools can just edge up on the BI vendors in reach with each release, secure in their Office base that they control what users really want: Excel.

Iteration is the key

2006-03-30T09:05:00.000-08:00

Ken Pohl writes a thoughtful article on the issues of project management of a data warehouse project, and how this can differ from other IT projects. As he points out, a data warehouse project is unusual in that it is essentially never finished - there are always new sources to add, new types of analysis the customers want etc (at least there are if the project is a success: if it failed then at least you won't have too many of those pesky customer enhancement requests).

As the article points out, a data warehouse project is ideal for an iterative approach to development. The traditional "waterfall" approach whereby the requirements are documented at ever greater levels of detail, from feasibility through to requirements through to functional specification etc is an awkward approach. I have observed that in some companies the IT departments have a rigid approach to project management, demanding that all types of projects follow a waterfall structure. This is unfortunate in the case of data warehouse projects, where end-users are often hazy on requirements until they see the data, and where changing requirements will inevitable derail the neatest functional specification document (see diagram).
Given a 16 month average elapsed time for a data warehouse project (TDWI) it is almost certain that at least one, and possibly several, major changes will come along that have significant impact on the project, which in a waterfall approach will at the very least cause delays and may put the entire project at risk.

By contrast a data warehouse project that bites off scope in limited chunks, while retaining a broad and robust enterprise business model, can deliver incremental value to its customers, fixing things as needed before the end users become cynical, and gradually building political credibility for the warehouse project. Of course the more responsive to change your data warehouse is the better, but even for a traditional custom build it should be possible to segment the project delivery into manageable chunks and deliver incrementally. The data warehouse projects which I have seen go wrong are very often those which have stuck to a rigid waterfall approach, which makes perfect sense for a transaction processing system (where requirements are much more stable) but is asking for trouble in a data warehouse project. Ken Pohl's article contains some useful tips, and is well worth reading.

Unifying data

2006-03-30T00:51:00.000-08:00

I can recall back in the early 1990s hearing that the worlds of structured and unstructured data were about to converge. A decade on, and despite the advent of XML, and that prospect still looks a long way off. It is like watching two people who have known each either for years and are attracted to each other, yet never seem to find a way of getting together. Some have argued that the data warehouse should simply open up to store unstructured data, but does this really make sense? When DBMS vendors brought out features allowing them to store BLOBS (binary large objects) the question should have been asked: why is this useful? Can I query this and combine it usefully with other data? Data warehouses deal with numbers (usually business transactions) that can be added up in a variety of ways, according to various sets of business rules (such as cost allocation rules, or the sequence of a hierarchy), which these days can be termed master data. The master data gives the transaction data "structure". A Powerpoint slide or a word document or an audio clip tends not to have much in the way of structure, which is why document management systems place emphasis on attaching keywords or tags to such files in order to give them structure (just as web pages are given similar tags, or at least they are if you want them to appear high up in the search engines).

You could store files of this type in a data warehouse, but given that these things cannot be added up there is little point in treating them as transactions. Instead we can consider them to be master data of a sort. Hence it is reasonable to want to manage them from a master data repository, though this may or may not be relevant to a data warehouse application.

I am grateful to Chris Angus for pointing out that there is a problem with the terms 'structured data' and 'unstructured data'. Historically the terms came into being to differentiate between data that could at that time be stuffed in a database and data that could not. That distinction is nothing like as important now and the semantics have shifted. The distinction is now more between data constrained by some form of fixed schema and whose structure is dictated by a computer application v data/documents not constrained in the same way. An interesting example of "unstructured data" that is a subject in its own right and needs managing is a health and safety notice. This is certainly not just a set of numbers, but it does have structure, and may well be related to other structured data e.g. HSE statistics. Hence this type of data may well need to be managed in master data management application. Another example is the technical data sheets than go with some products, such as lubricants; again, these have structure and are clearly related to a traditional type of master data, in this case "product", which will have transactions associated with it. Yet another would be a pharmaceutical regulatory document. Hence "structure" is more of a continuum than a "yes/no" state.

So, while the lines are blurring the place to reconcile these two worlds may not be in the data warehouse, but in the master data repository. Just as in the case of other master data, for practical purposes you may want to store the data itself elsewhere and maintain links to it e.g. a DMBS might not be an efficient place to store a video clip, but you would want to keep track of it from within your master data repository.

Microsoft MDM? Don't hold your breath

2006-03-29T00:54:00.000-08:00

At a conference this week at which Microsoft explained how it intends to unify its rambling applications offerings, Mike Ehrenberg (architect for Microsoft's MBS products) mentioned that Microsoft was "investigating"an MDM product offering. It should be said that Microsoft should be in an excellent position to understand the problem of inconsistent master data, at least within their own portfolio of business software products. Through a series of acquisitions they have assembled no less than five distinctly overlapping products for SMEs, and have manifestly failed to explain how any of this resembles a strategy. This mess has enabled innovative newcomers like Ataio make steady progress in what should really be Microsoft's natural turf, as customers have been bemused by Microsoft's seeming inability to articulate which technologies they were really intending to invest in. The answer, it seems, is all of them - MSFT will "converge" their five products "no sooner than 2009" (unofficially, 2011 is a target date I have heard from an insider). The most amusing line in the article was: "The MBS products, Gates said, "have more head room for growth than just about any business we're in." This is about as backhanded a compliment as one can think of: I have heard that Microsoft management is very unhappy about the lack of progress in this division, so this comment is like saying to a sports team that just came bottom of the league "we now have more room to improve than anyone".

Microsoft seems perennially to struggle in the enterprise software market, despite its vast resources, huge brand and marketing clout. It essentially stumbled into the DBMS marketplace; I have it on good authority that Gates originally approached Larry Ellison with a view to bundling Oracle as the DBMS on Windows NT, and it was only after being spurned that Microsoft decided to launch SQL Server out of the ashes of the Sybase code-base it had purchased (this is a piece of hubris that Oracle may live to regret). In Excel and Analysis Services Microsoft has the most ubiquitous business intelligence software out there, yet has hardly any mind-share in this market. Perhaps it is just not in Microsoft's DNA to really relish the enterprise software market, when its business model is above all about high volume, and large enterprises demand endless tinkering and specialization of software to their specific needs.
Based on the train-wreck that is Microsoft's enterprise applications strategy, I wouldn't count on a strong MDM product entry any time soon.

Data warehouse v master data repository

2006-03-28T01:29:00.000-08:00

Bill Inmon notes that "Second-generation data warehouses recognize the need for tying metadata closely and intimately with the actual data in the data warehouse". This is indeed a critical point, and is at the heart of why all those enterprise data dictionary projects in the 1990s (and even 1980s; sad to say I am old enough to have been involved with one in the 1980s) failed. Because the dictionaries were just passive catalogs, they were of some use to data modelers but otherwise there was little incentive to keep them up to date. In particular, the business people could not see any direct benefit to them, so after the initial project went live the things quietly got out of date. In order for such initiatives to succeed it is critical that the business metadata (more important than the technical metadata) is tied into the actual instances of master data, so that the repository does not just list the product hierarchy structure (say) but also lists the product codes that reside within this structure. Ideally, the repository would act as the primary master source of master data for the enterprise, and serve up this data to the various applications that need it, probably via an automated link using middleware such as Tibco or IBM Websphere. Not many companies have taken it to this stage, but there are applications at BP and Unilever that do, for example.

However one important architectural point is that you may not want the data warehouse to actually manage all the master data directly; instead it may be better to have a separate master data repository. The reason for this apparently odd approach is that in a data warehouse you want the data to be "clean" i.e. validated, conforming to the company business model etc. On the other hand master data may have separate versions, drafts (e.g. draft three of the planned new product catalog) that need to be managed, and potentially "dirty" master data that is in the process of being improved or cleaned up. Such data has no place in a data warehouse, where you are relying on the integrity of the numbers.

Hence a broader picture may see an enterprise data warehouse alongside a master data repository, the latter feeding a "golden copy" of master data to the warehouse, just as it will feed the same golden copy to other applications that need it. With such an approach, and current technology, those old enterprise modeling skills might just come in handy.

Incidentally, spring is definitely in the air in Europe. The sun is out in London, there is a spring in people's step, and the French have called a general strike.

Putting lipstick on a caterpillar does not make a butterfly

2006-03-27T01:30:00.000-08:00

Oracles recent repackaging of its BI offerings appears to be just that: a repackaging of existing technologies, of which of course they now have a lot. Peoplesoft had EPM, which had a mediocre reputation, but they did better with Siebel, who had astutely acquired nQuire, a good product that was relabeled Siebel Analytics. Oracle also has Discoverer, a fairly blatant rip-off of Business Objects, a series of pre-built data marts for Oracle apps as well as assorted older reporting tools developed along the way, like Oracle ReportBuilder, which seems to me strictly for those who secretly dislike graphical user interfaces and yearn for a return to a command prompt and "proper" programming. This assortment of technologies has been placed into three "editions", but you can scour the Oracle website in vain for anything which talks about the actual integration of these technologies at anything below the marketing/pricing level. Hence it would seem that customers will still essentially be presented with a mish-mash of tools of varying quality. Perhaps more R&D is in the works to integrate the various BI offerings properly, but it seems that for now Oracle still has some work to do in presenting a coherent BI picture. Business Objects and Cognos will not be quaking in their boots.

When did "tactical" become a dirty word?

2006-03-21T01:57:00.000-08:00

A new report from Butler Group bemoans the "tactical" use of business intelligence tools on a piecemeal, departmental basis, calling for an enterprise-wide approach. However it rather misses the point about why this state of affairs exists. The author reckons "Business will only be able to improve its information services, and obtain real value from the ever-increasing data silos that it continues to generate, when it accepts the significant advantages to be gained from integrating and standardizing its approach to the management of its BI technology services." Or, to paraphrase: why on earth are you deploying separate departmental solutions, you bunch of dimwits?"

As I have discussed previously on this blog. There are actually several reasons why most BI initiatives are departmental, often using different tools. It is not that the business people are a crowd of masochists. The first reason is that a lot of BI needs are legitimately local in nature, specific to a department or operating unit. It is dramatically easier for a department to set up a data mart that has just its own data, and stick on top of that a reporting tool like Business Objects or Cognos, than it is to wait for the IT department to build an enterprise warehouse, which takes 16 months on average to do, costs 72% of its build costs every year to support, and then usually struggles to keep up with changing business requirements.

So it is not a matter of "accepting the significant advantages" of an enterprise approach. Everyone accepts that such an approach would be preferable, but the IT industry has made it very, very difficult to actually deliver in this promise, and people naturally fall back on "tactical" (i.e. working) solutions when grandiose ones fail. Ideally you would like an enterprise data warehouse, deployed in a few months, that can deal with business change instantly, and can at the same time both take an enterprise view and respect local departmental business definitions and needs, which will differ from those of central office. The trouble is, most companies are not deploying data warehouses like this, but are still stuck in a "build" timewarp, despite the existence of multiple packaged data warehouses which can be deployed more rapidly, and in at least one case can deal with change properly. Until this mindset changes, get used to a world with plenty of "tactical" solutions.

The data warehouse market breaks into a trot

2006-03-20T02:16:00.000-08:00

The latest figures from IDC (who, by the way, are by far the most reliable of the analyst forms when it comes to quantitative estimates) is that the data warehouse market will grow at a 9% compound rate from now through to 2009, reaching USD 13.5 billion in size (up from USD 10 billion today), as reported in an article on the 17th of March. Gartner also reckon that this market is growing at twice the pace of the overall IT market (their estimates are slightly lower, but would trust IDC' more when it comes to figures). It would be interesting to see the proportion of this that is packaged data warehouse software (see the recent report by Bloor) but unfortunately they don't split out the data in this way. This figures does not include services, but based on other analyst estimates this market is at least three times this size; there never seems to be any shortage of need for systems integrators.

Given all the billions spent on ERP systems in the last ten years or so, it is about time that more attention was paid to actually trying to make sense of the data captured in these and other transaction processing systems, which for a long time have consumed the lion's share of IT development budgets. After all, there is likely to be more value in spotting trends and anomalies in the business than in merely automating processes that were previously manual, or in just shifting from one transaction processing system to another.

Should ETL really be ELT?

2006-03-16T08:54:00.000-08:00

Traditionally ETL (extract/transform/load) products such as Informatica, Ascential and others have fulfilled the role of getting data out of source systems, dealing with inconsistencies between these source systems (transform) and then loading the resultant transformed data into a set of database tables (perhaps an operational data store, data marts or directly to a data warehouse).
However in the process of doing the "transform" a number of issues crop up. Firstly, you are embedding essentially what is a set of business rules (how different business hierarchies like product classifications actually relate) directly into the transformation rules. This is a dark place should you want to make sense of them in other contexts. If the rules are complex, which they may well be, then you can create a Frankenstein's monster of transform rules that become difficult to maintain, in a set of metadata that may be hard to share with other applications.

Moreover this is a one-way process. Once you have taken your various product hierarchies (say) and reduced them to a lowest common denominator form, then you can certainly start to analyze the data in this new form, but you have lost the component elements to all intents and purposes. These different product hierarchies did not end up different without some reason; they may reflect genuine market differences in different countries, for example. Moreover they may contain a level of richness that is lost when you strip everything down to a simpler form.

Ideally in a data warehouse you would like to be able to take an enterprise view, but also retain the individual perspectives of different business units or countries. For example it may be interesting to see the overall figures in the format of a particular business line or country. Now of course there are limitations here, since data from other businesses have not have sufficient granularity to support the views required, but in some cases this can be fixed (for example by providing additional allocation rules) and at least you have a sporting chance of doing something useful with the data if you have retained its original richness. You have no chance if it is gone.

Hence there is a strong argument to be made for an "ELT" approach, whereby data is copied from source systems pretty much untouched into a staging area, and then only from there is transformation work done on it to produce cross-enterprise views. If this staging area is controlled by the data warehouse then it is possible to provide other, alternate views and perspectives, possibly involving additional business metadata at this stage. The only real cost in this approach is some extra storage, which is hardly a major issue these days. Crucially, the transformation logic is held within the data warehouse, which is open to interrogation by other applications, and not buried away in the depths of a proprietary ETL format. Moreover, the DBMS vendors themselves have added more capability over the last few years to deal with certain transformations; let's face it, a SQL SELECT statement can do a lot of things. Since the DBMS processing is likely to be pretty efficient compared to a transformation engine, there may be performance benefits also.

This approach has been taken by more modern ETL tools like Sunopsis, which is explicitly ELT in nature. Intriguingly, Informatica added an ELT option in Powercenter 8 (called the "PowerCenter 8 pushdown optimization"), which suggests that this approach indeed is gaining traction. So far, good on Sunopsis for taking the ELT approach, which I believe is inherently superior to ETL in most cases. It will be interesting to see whether Ascential also respond in a future release.

The unbearable brittleness of data models

2006-03-14T08:37:00.000-08:00

An article in CRM Buyer makes an important point. It highlights that a key reason why customer data integration projects fail is the inflexibility of data model that is often implemented. Although the article turns out to be a thinly disguised advert for Siperian, the point is very valid. Traditional entity-relationship modeling typically is at too low level of abstraction. For example courses on data modeling frequently give examples like "customer" and "supplier" as separate logical entities. If your design is based on such an assumption, then applications based on this will struggle if one day a customer becomes a supplier, or vice versa. Better to have a higher level entity called "organization", which can have varying roles, such as customer or supplier, or indeed others than you may not have thought of at the time of the modeling. Similarly, rather than having an entity called "employee" it is better to have one called "person", which itself can have a role of "employee" but also other roles, perhaps "customer"for example.

This higher level of data modeling is critical to retaining flexibility in systems, removing the "brittleness" that so often causes problems in reality. If you have not seen it, I highly recommend a paper on business modeling produced by Bruce Ottmann, one of the world's leading data modelers and whose work has found its away into a number of ISO standards. Although Bruce works for Kalido, this whitepaper is not specific to Kalido but rather discusses the implications of a more generic approach to data models.

I very much hope that the so-called "generic modeling" approach that Bruce recommends will find its way into more software technologies. Examples where it does are Kalido and Lazy Software, and, although in idea rather than product form, in the ISO standard 10303-11, which covers a modeling language called Express that can be used to represent generic data models. It came about through work originated at Shell and then extended to a broader community of data modelers, including various academics, and was particularly aimed at addressing the problem of exchanging product models; it is known as STEP. However the generic modeling ideas developed with this have much broader application than product data. Given the very real advantages that generic modeling offers, it is to be hoped that more software vendors pick up on these notions, which make a real difference to the flexibility of data models, and hence improve the chances of projects, such as CDI projects, actually working in practice.

ETL moves into the database

2006-03-13T03:05:00.000-08:00

With SQL Server 2005 Microsoft has replaced its somewhat limited DTS ETL offering with SQL Server Information Services (SSIS), which is compared with IBM's offerings (based on the Ascential acquisition). I have written previously about the shrinking of the ETL vendor space, and the enhanced Microsoft offering will merely accelerate this. Oracle has its Warehouse Builder technology (despite its name this is really an ETL tool) as well as Microsoft and IBM, and as these tools improve it will be tough for the remaining ETL vendors. Informatica has broadened into the general data integration space, and seems to be doing quite well, but there are not many others.

Sunopsis is innovative with its "ELT" approach which sensibly relies on, rather than competes
with, the native DBMS capabilities, but it remains to be seen how long it can flourish, given that the DBMS ETL capabilities will just keep getting better and eat away at its value. The surreal Ab Initio is reportedly doing well at the high volume end, but given its secretive nature it is hard to say anything with certainty about this company other than its business practices and CEO are truly eccentric (a fascinating account of its predecessor Thinking Machines can be found at the following link). Data Junction has a strong reputation and is OEMed by many companies (it is now part of Pervasive Software). There are a few other survivors, like ETI, who have just recapitalised their company after struggling for some years , but it is hard to see how ETL can remain a sustainable separate market in the long term. Indeed Gartner has recently stated that they will are to drop their "magic quadrant" for ETL entirely.

The future of ETL would appear to be in broader offerings, either as part of wider integration software or as just a feature of the DBMS.

The hollowing out of ERP

2006-03-09T09:33:00.000-08:00

Now that there are effectively two enterprise ERP vendors bestriding the world, it may seem that they can just sit back and count the spoils. Both have huge net profit margins derived from their market leadership, so it may seem churlish to contemplate their eventual demise, yet a number of factors are combining that should cause a few flutters in Redwood City and Walldorf. Consider for a moment what a transaction system application such as ERP actually does, or used to do:

business rules/workflow
master data store
transaction data store
transaction processing
user interface
(and perhaps some business content e.g. pre-built reports)

This edifice is under attack, like a house being undermined by termites. Transaction processing itself has long been mostly taken care of elsewhere, by old-fashioned TP monitors like IBM CICS or by new-fashioned TP monitors like BEA's Weblogic or IBM Websphere. These days there are alternate workflow engines popping up, like Biztalk from Microsoft, or even a slew of open source ones. Moreover, more than half of the ERP functionality purchased is unused. The storage of data itself is of course done in the DBMS these days (though SAP tries hard to blur this line with its clustered table concept). As the idea of separate master data hubs catches on e.g. customer data hubs like Siperian's, or product data hubs, or more general ones, and the serving up of such data is possible through EAI technology, then this element too is starting to slip away from the ERP vendors. The user interface for update screens should hardly be that complicated (though you'd never guess it if you have ever had the joy of using SAP as an end user), and these days can be generated from applications e.g. from a workflow engine or a master data application. This does not leave a great deal.

If, and it is a big if, SOA architecture takes off, then you will also be able to plug in your favorite cost allocation module (say) from a best of breed vendor, rather than relying on the probably mediocre one of your ERP supplier. Combine this with the emergence of "on demand" hosted ERP services from emerging companies like Ataio and Intacct as alternatives, and the vast ERP behemoth looks a lot less secure up close than it may do from a distance. If the master data hubs and business workflow engines continue to grow in acceptance and chip away further at key control points of ERP vendors, then at some point might it be reasonable to ask: exactly what is it that I am paying all those dollars to ERP vendors for?

This line of reasoning, even if it is very early days, explains why SAP and Oracle have been so anxious to extend their product offerings into the middleware space, with Netweaver and Fusion respectively. This is also what SAP has been trying to falteringly launch an MDM application (the rumor is that after the botched initial SAP MDM, the buy-in of A2i isn't going that well either; maybe a third attempt is in the works?) and Oracle has been keen to promote its customer hub.

Of course it is too soon to be writing the obituaries of ERP yet, but a combination of evolving technologies is starting to illuminate a path for how you would eventually migrate away from dependence on the giant ERP vendors, rather than endlessly trying to consolidate on fewer vendors, and fewer instances of each. Now that would be radical thinking.

Information as a service?

2006-03-08T02:31:00.000-08:00

I see in our customer base the stirrings of a movement to take a more strategic view of corporate information. At present there is rarely a central point of responsibility for a company's information assets; perhaps finance have a team that owns "the numbers" in terms of high level corporate performance, but information needed in marketing and manufacturing will typically be devolved to analysts in those organizations. Internal IT groups may have a database team that looks after the physical storage of corporate data, but this group rarely have responsibility for even the logical data models used within business applications, let alone how those data models are supposed to interact with one another. Of course things are complicated by the fact that application packages will have their own version of key data, and may be the system of record for some of it. Yet how to take a view across the whole enterprise?

organizationally, what is needed is a business-led (and not IT-led) group with enough clout to be able to start to get a grip on key corporate data. This team would be responsible for the core definitions of corporate data, its quality, and being the place that people come to when corporate information is needed. In practice, if this is not to become another incarnation of a 1980s data dictionary team, then this group should also have responsibility for applications that serve up information to multiple applications, and this last point will be an interesting political battle. The reason that such a team may actually succeed this time around is that the technologies now exist to avoid the "repository" (or whatever you want to call it, of master data being a passive copy. Now the advent of EAI tools, enterprise buses, and the more recent master data technologies (from Oracle, Kalido, Siperian, IBM etc) mean that master data can become "live", and synchronized back to the underlying transaction systems. Pioneers in this area were Shell Lubricants and Unilever, for example.

However technology is necessary, but not sufficient. The team needs to be granted ownership of the data, this notion sometimes being called "data stewardship". Even if this ownership is virtual, it is key that someone can arbitrate disputes over whose definition of gross margin is the "correct" one, and who can drive the implementation of a new product hierarchy (say) despite that fact that such a hierarchy touches a number of different business applications. It is logical that such a group would also own the enterprise data warehouse, since that (if it exists) is the place where much corporate-wide data ends up right now. This combination of owning the data warehouse and the master data hub(s) would allow infrastructure applications to be developed that can serve up the "golden copy" data back to applications that need it. The messaging infrastructure already exists to allow this to happen.

A few companies are establishing such groups now, and I feel it is a very positive thing. It is time that information came out if its back-room closet and moves to centre stage. Given the political hurdles that exist in large companies, the ride will not be smooth, but the goal is a noble one.

Broaden your horizons

2006-03-06T06:00:00.000-08:00

At a talk at the recent TDWI show, consultant Joshua Greenbaum, an analyst with Enterprise Applications Consulting (who?) managed to bemoan the cost of data warehouses, but then demonstrates a seeming lack of knowing exactly what one is by claiming that the alternative is to do "simple analyses of transactional data". Well Joshua, that is called an operational data store, and indeed it has a perfectly respectable role if all you want to do is to look at a single operational system for operational purposes. However a data warehouse fulfils quite a different role: it takes data from many different sources, allows analysis across these inconsistent sources and also should provide historical context e.g. allowing comparisons of trends over time. You can't do these things with an operational data store.

Hence it is not a case of "ODS good, data warehouse bad" - instead both structures have their uses. Of course Joshua is right in saying that data warehouse success rate is not great, but as I have written elsewhere, it is not clear whether data warehouse projects are really any worse than IT projects in general (admittedly, that is not setting the bar real high). Perhaps Joshua was misquoted, but I would have expected something more thoughtful from someone who was an analyst at Hurwitz. Admittedly he was an ERP (specifically SAP) analyst, so perhaps has a tendency to think of operational things rather than things wider than ERP. Perhaps he is suffering from the same disease that seems to affect people who spend too much time on SAP.

Interest in MDM grows

2006-03-06T05:32:00.000-08:00

Last week I was a speaker at the first CDI (customer data integration) conference, held in San Francisco. Although the CDI institute (set up by Aaron Zornes, ex META group) started off with customer data integration, looking at products like Siperian and DWL, the general movement towards MDM as a more generic subject has overtaken it, and indeed Aaron mused in his introductory speech whether they may change the title to the MDM Institute. For a first conference it was well attended, with 400 people there and supposedly 80 being turned away due to unexpectedly high demand. There were the usual crowd of consultants happy to advise expertly on a topic they had never heard of a year ago. Most of the main MDM vendors put in an appearance e.g. IBM/Oracle/I2 (but no SAP) as well as specialists like Siperian and Purisma, plus those like HP who just have too big a marketing budget and so have a booth everywhere, whether or not they have a product (those printer cartridges generate an awful lot of profit).

The conference had a rather coin-operated feel, as sponsoring vendors duly got speaker slots in proportion to the money they put in - with IBM getting two plenary slots, but there were at least a few customer case studies tucked away amongst the six concurrent conference tracks. My overall impression was that MDM is a bit like teenage sex: everyone is talking about it, people are eager to know all about it but not that many are actually doing it. As time passes and MDM moves into adolescence there will presumably less foreplay and more consummation.
Further conferences are planned in London, Sydney and Amsterdam, demonstrating if nothing else that plenty of vendors are willing to pay Aaron to speak at the shows.

It's all in the timing

2006-02-28T04:59:00.000-08:00

Database guru Colin White raises a point in a BI network article that is often overlooked by commentators. To quote:

"...another difficulty is that customer reference data and relationships vary over time. This issue has important implications for business intelligence applications that may analyze customer data across various time periods, comparing revenue this month to this time last year, for example. If, during the last 12 months, the customer hierarchies have changed or the sales organization has been restructured, then this will affect the validity of the comparison. This means that metadata and metamodel changes may have to be tracked and recorded in MDM applications."

Spot on Colin! Except that "may have to be tracked" should be "must". Organizations do not make significant changes to their business stcuctures every day, but these changes do happen every few weeks or months e.g. reorganizations occur, marketing reclassify their product hierarchy or customer segmentations, finance changes allocation rules. Yet most MDM products concentrate on point-in-time synchronisation e.g. of customer data, and frequently retain no history whatsoever. Hence when you want to make comparisons over time, or go back and reconstruct something in the pass to deal with a compliance request, the task is difficult to impossible since the old transactions may be archived, but the master data associated with them is typically not.

A well-designed system to handle master data should be able to reconstruct past hierarchies for comparison. Just as within a data warehouse, where you often want to go back and look at data "as is" and "as was" and even "as it would have been", the need to go back in time and understand the changes in master data is very important. However the big vendors don't understadn this issue properly, and most customers have just started dabbling in MDM so haven't thought yet about this thorny issue, which will come back to bite them in due course.

Metadata repositories are not enough for MDM

2006-02-27T09:17:00.000-08:00

An article in CIO today has one good nugget and then sets out to miss the main point entirely. The author correctly says that "Working with data across an enterprise -- especially in an SOA environment -- requires understanding its context and semantics, not just its format and field attributes". Spot on. However he then drifts off to bemoan the general state of the metadata repository market. What is missed is that the solution to master data management can never be a passive repository of the type that organizations put in during the 1990s e.g. the Reltech and Brownstone products later bought by CA, or even the IBM attempts prior to this e.g. the IBM Data Dictionary, which I had fun using in the 1980s. Such initiatives fail for several reasons:

The metadata that is captured tends to be technical metadata e.g. "CUST is VARCHAR(8)" rather than business metadata: "Coca cola is a product within the product class carbonated drink, which is in turn within the product group beverage". Technical matadata is of limited use.
If the metadata is not linked back into real operational systems, it will become as out of date as the repositories of a decade or more ago
The tools tend to use arcane conventions which business users have trouble relating to, do this job is left to IT folks, who aren't really in the best position to know what the data rules really are.
There is no one single definition of almost any master data in a company; instead there are many, and they need to be managed.

It is critical for the success of any master data (or metadata) project that the people who actually understand the master data like general ledger structures, customer segmentation, product hierarchies i.e. the business people who set them up, are put in charge of owning the core definitions, including the process to update this data. With this is mind there is a data governance process required in addition to any software tool. Such software should ideally be able to deal with business modeling in a way that is more intuitive than E/R diagramming, which most business users have trouble with.

Next, there needs to be automated workflow to support this update process, which may be complex e.g. there may be several draft versions of a new product hierarchy needed, with different groups of people who have to review and approve before final publication. If this just happens by email then errors will occur.

The master data repository then needs the capability of "semantic integration" i.e. enabling the storage of multiple versions of the various types of master data that actually exist today, so that it can be at the hub of any project to improve the quality of this data, which may involve some rationalization.

Having understood what is out there, modeled it, mapped together the different versions and defined the workflow needed to deal with update, the master data project then needs the ability to hook up to messaging technology to actually drive changes made to the master data repository back out into the other operational systems like ERP and CRM. Without such integration it will only be a partial solution.

Not many, perhaps any, vendors today offer a complete solution to the above set of issues, but some come close, and not one of them is a traditional repository vendor or an ETL tool.

Tackle master data to achieve SOA dream

2006-02-23T04:38:00.000-08:00

Dana Gardner makes a sensible point regarding SOA, which I would amplify. In order for the promise of SOA to come to fruition, the discussion needs to move beyond the user interface and all those sexy Web 2.0 applications, and towards dealing with the barriers to applying SOA to general business applications. The thorny issues of dealing with inconsistent data models, and of addressing data quality seem rarely to come up in discussion, yet these are critical to the broader success of the SOA dream. It is tricky enough to get an enterprise application up and running, but if you want to be able to have different applications from different vendors interacting happily with one other then it is not just a matter of application layer standards; for business applications these need to be able to share data as well. Unfortunately the reality in large companies is that there are multiple, incompatible, versions of key business definitions such as "product", "customer", "asset", "supplier" and even seemingly unambiguous terms like "gross margin" (the average large company has 11 systems that each think they are the master source of "product", for example, according to a survey a couple of years ago. All the elegant user interface design in the world is of limited help if two applications cannot agree on what to display when they are supposed to be showing a product hierarchy, or where the definitions of cost allocations differ.

This is why projects and tools that can enable the improvement of the management of key business terms, such as "master" data like "product", "customer" etc are a necessary precursor to broader SOA deployment in mainstream business applications, as are improvements in data quality, another area that is far from sexy but in practice is a major issue whenever applications have to bring together data from multiple sources, as any data warehouse practitioner can testify. Dealing with the "semantic integration" of the varying business models that exist out there in multiple ERP, CRM and supply chain systems is a major barrier to broad SOA adoption, yet it is scarcely mentioned in the technology media. When those first SOA prototypes start showing data to business users, and it is realized that the data is "wrong", this topic will become much higher profile.

Vendor due diligence

2006-02-21T05:57:00.000-08:00

In a previous blog I gave some general thoughts about vendor evaluation, and expanded on this to give an outline framework for such evaluations. One thing that should be considered in any evaluation is "his stable is the vendor" i.e. will they still be around in a few years? This question can be surprisingly hard to answer, and in fact the question itself has a flaw. The question should be: "will this product still be further developed in a few years?". The reason for the distinction is that even the largest vendors sometimes discard technologies due to changes in their product roadmap, internal political issues or because the thing isn't selling very well.

So, if buying from a very large vendor, it will be easy enough to look at their general health because their finances are public (OK, there are sometimes accounting frauds but you can't do much about these). The usual ways of analyzing company's health can be used here. I highly recommend the "Guide to analyzing Companies" by the Economist, which is clearly written and gives excellent examples of the indicators of company health and also of early warning signs. You should not assume that just because a company is publicly listed then it must be OK - ask the customers of Commerce One about that approach, for example.

In such cases you will probably find that the company is fine, so the bulk of the diligence efforts should instead be directed to how important this particular product is to the company, and so assess how likely is to to get ongoing development. Of course the vendor is hardly a reliable source here, but you can seek advice from analysts, and also it is fair to ask the vendor
just how many customers of this particular product there are (you should be able to talk to some of them). For example, SAP's MDM product had seemingly shifted just twenty copies into customer use throughout its 18,000 strong customer base in two years. Given this dismal penetration rate it was perhaps not a shock that they dropped it (their replacement MDME offering is based on an acquisition of PIM vendor A2i). Anything which is not contributing to a vendor's sales figures on any scale should be considered suspect.

In the case of small vendors you have different problems. You can be pretty sure that the product is important to the vendor, since it is probably the only one that they have. The question is whether the vendor will survive. This is trickier, since the company is probably privately held, and so is not obliged to publish its accounts, at least in the US. In the UK you can get around this by paying a few pounds to Companies House and look up their acocunts.
If you are making a large purchase then it is fair game for you to ask for information on the company financials, and you should get nervous if they refuse. One thing that will achieve little is to ask for a letter from the VCs backing the company. They will inevitably sing its praises; they are hardly going to say "ah, this one's on a bit of a knife-edge, I'd watch out if I were you". Indeed I knew of one case where a major deal was in progress at a BI vendor, and through a contact I became aware that the entire future financing of the (cash strapped) company was dependent on this deal going ahead; in such cases you cannot expect objectivity from investors.

So, what can you do? Well, profits are an opinion but cash is not. Hence, assuming you can see some figures, you can get a sense of how much cash the company has, and attempt to work out the "burn rate" i.e. how fast are they burning through this cash (most VC backed start-ups are unprofitable; if they were profitable then they probably wouldn't need expensive VC money). However this on its own may give false signals. Due to their IRR-driven instincts, VCs don't dole out more cash than they need to start-ups; they like to always have the option of pulling out if they need to, so it is rare for a start-up to actually have more than about one year's cash needs in hand. The question is: will they be able to raise more cash if they need it? This is a complex subject, but essentially you should be able to get a sense of this by talking to analyst familiar with the VC community. For example, companies growing at 50% or more are very likely to be able to raise cash, even if they very unprofitable. The gross margins in software are commonly 90%, so profits will come eventually if the company can just grow large enough; this is why VCs invest in software companies more than, say, restaurants. So if you cannot find someone knowledgeable to look the figures over for you and make an assessment, then a decent proxy for security is the revenue growth rate. If the company's growth is stalling (say 10% a year growth for a small-medium software company) then things could be sticky in a future financing round. This is a generalization (and companies with a subscription model, for example, have a much more predictable life than ones selling traditional licenses) but it may be the only real set of figures that you can dig out.

Another source of due diligence is other customers, who may well have done exactly the same due diligence exercise as you fairly recently. Of course you have to be careful it was not out of date, and you should check how thorough they really were, but you may be able to save yourself a lot of work. If three Fortune 100 companies recently did detailed due diligence on a vendor and bought its software anyway, this may help you at least feel better.

Remember: the company or product does not have to be around in ten years if your payback case is 13 months. The faster the payback period for the product, the less concerned that you need to be about agonizing over the long term future of the company, or of the product within the vendor. You did do a proper business case for the purchase, right? Of course you did.

Evaluating Software Vendors - a framework

2006-02-21T02:46:00.000-08:00

I have written previously in general about vendor evaluation processes. Over time I will flesh this topic out further, as I feel it is a much-neglected area. As I have said, it is important to define up front your main functional and technical requirements from the software package that you want to buy. It is then important to have a process to take the "long list" of candidates down to a manageable number of two to four vendors to evaluate properly.

So, you have done this, and are down to three vendors. What are the mechanics of the evaluation criteria? I show in the diagram a simplified example to illustrate. It is critical that you decide on what is most important to you by selecting weightings for each criteria before you see any products. Ensure that you group the broad criteria into at least two and perhaps three categories: functional should list all the things you actually want the product to do, and you may choose to separate "technical", which may include things like support for you particular company's recommended platforms e.g. "Must run on DB2", or whatever. What is sometimes forgotten is the commercial criteria, which are also important. Here you want things like the market share and financial stability of the company, how comprehensive its support is, how good is its training etc. I would recommend that you exclude price from these criteria. Price can be such a major factor that it can swamp all others, so you may want to consider it as a separate major criteria once you have done the others. I would recommend that the "functional" weightings total not less than 50%, It is no good buying something from a stable vendor if the thing doesn't do what you need it to.

An important thing about using a weighting system like this one is that the weights must add up to 100. The point here is that it forces you to make trade-offs: you can have an extra functional criteria, but you must reduce the existing weights to make sure that the weights still add to 100. This gives the discipline to stop everything being" essential". You assign all the weights before the evaluation begins. You can share this with the vendors if you like. Coveniently, however you assign the weights, the scores will come in out of 1000, so can be easily expressed as a percentage e.g. vendor B is a 74% match to the criteria in the example, while vendor C is 67%.

The final stage is that you need to score the various criteria that you have laid out. You want this to be as objective as possible, which is why you do not want too many - you want to see evidence for the functional criteria. Just because the salesman says that it does something is not sufficient to credit a score - you need to see the feature yourself, preferably working against some of your own data rather than faked up demo. I recall doing an evaluation of BI tools in 1992 at Shell and having one vendor with quite a new product that due to a stellar analyst recommendation made it to the short-list. When the pre-sales guy turned up and was presented with a file of our data for him to do the trial on he went white; their whole product was virtually hard-coded around their demo dataset, and it quickly became clear that even the slightest deviation from the data they were expecting caused the product to break.

Score each criteria out of 10. Commercial criteria can be done off-line and in advance; analyst firms can help you with this, as they tend to be up on things like market shard (IDC have the most reliable quantified figures, but rough estimates are probably good enough). Financial stability is a subject all in itself, and I will cover this in another blog.

The evaluation then becomes quite mechanical, as you crank out the scores. You see that in this simplified example vendor B has won, though not by a huge margin. If it turns out that vendor B's price is twice that of the others then you may decide this difference is not big enough to justify the slightly better scores (we will retunr to this shortly). Again, you could weight price as a factor if you prefer.

However, don't get too hung up on price; as someone who used to do technology procurement it may seem like the be all and end all, but it is not. The total cost of a new software package to your company is far greater than the initial license cost. There is maintenance and training over several years, and also the people time and cost in actually implementing the technology, which will usually be several times the cost of the software package. Hence getting a package that is 20% more productive than the next best is worth a lot more than 20% extra in the license price, as the associated costs of people will be multiples of the software cost (people costs being five times package software costs in a project is common, ten times is not unusual). It is sensible for you to try and consider the likely full lifetime costs of the software in this way (assume, say five years) since you will then have an idea as to how important the license cost really is. For example if you are about to do a 30 country roll-out budgeted at USD 50 million, making sure that the product you select is the most productive one is a lot more important than if you are doing a single project for USD 500k. Here a product that is 10% more productive than the next one to implement may save you USD 5 million, so haggling to the death over that last USD 10k of license discount may not be so critical. This will give you a true bottom line case for the level of spend you can afford to make.

Taking a structured evaluation approach like this has a number of benefits. Firstly, it reduces the amount of gut feel and "did I like the salesman" mentality that too often creeps in. You'll probably never see the salesman again unless you want to buy more software, but you will be stuck with the product that you select for years. Secondly, it gives you a documented case for selection that can, if necessary, be used to back up things internally e.g. in the case of an audit, or just to give comfort to senior management that a sound process has been used.

Moreover, given that salesmen get paid on how much they sell you, you'd be surprised at the tactics they can adopt; they will try and go over your head if they think they are going to lose, and make all sorts of accusations about how the process wasn't fair and how you are about to make a terrible mistake, so having a solid, documented case will make it much easier for your manager to do the right thing and tell the salesmen to take a running jump. I am amazed at how often this tactic was tried when I was procuring software, but I never once had a decision overturned. If you ever find yourself in this situation, remember that revenge is a dish best served cold. After a particularly acrimonious session with one vendor salesman when I was working at Exxon, I was amused to find, a few years later, the same salesman turning up a few years later when I transferred to Shell. He walked in the room, his face fell when he saw me and he walked back out again. Good professional sales staff know that the world is a small place and that it does not pay to aggravate the customer, but all too few remember this.

In another blog I will return to the subject of assessing the financial stability of vendors.

Packaged Data Warehouse Market

2006-02-20T03:02:00.000-08:00

I was impressed with the depth of a recent Bloor report "Packaged Data Warehouses", which examines in depth Decision Point, Kalido, IBM DM", SAS, SAP BI, Showcase and Teradata. In these days when some analyst firms' presentations barely mention vendors at all it is great to see an old-fashioned, detailed set of product evaluations. Also, unlike many, this report was not paid for by a particular vendor, so the companies covered within it have had no opportunity to influence the results.

The report evaluates and scores the products by:

stability and risk
support
performance
ease of use
fit for purpose i.e. product quality
architecture
value for money

Having looked carefully through the Kalido evaluations, it seems fair and accurate, and clearly is the product of a great deal of work. I commend Philip Howard, the analyst who led this, for producing such a comprehensive piece of analysis in an age of lightweight analyst sound-bites.

The report is not cheap, but it does have genuinely good in-depth analysis, and anyone considering a data warehouse project should buy this before they contemplate building one themselves.

Blue sky thinking?

2006-02-17T08:02:00.000-08:00

IBM continues to invest in "information management", creating a major practice in this area. It has been active for some time in its software division in acquiring technologies related to data integration and various aspects of master data management. Now this latest reorganization puts in place the services side that accompany the software products. Clearly information management goes well beyond just technology, involving ownership of data (business or IT?) governance issues, project management, change management, perhaps setting up competence centers, etc. Hence there is clearly a large services component. Estimates of the BI/data warehouse market vary, but the total market is perhaps ten times larger than that purely for the software technologies. This reorganization makes clear that IBM sees this is a growing opportunity.

Someone once described IBM as the "beige" of the computer world i.e. they go with anything, meaning that IBM raises less hackles in customer organizations than some other high profile organizations e.g. Accenture is loved by some, but loathed by others. Hence in some ways IBM ought to be quite well placed to provide advice on projects that tend by definition to span wide areas of the business, and touch many other technologies. At some point systems integrators need to stop sucking on the comforting teat of ERP implementations, ERP re-implementations and ERP consolidation projects (surely at some point customers are going to stop paying for re-implementing these huge projects for the nth time?), so it would seem timely for enterprise software consulting teams to have a major alternative to draw on.

It must be generally good for the industry that IBM is addressing information management in a serious way, as if nothing else it will raise its profile. Delivering better information to decision makers is something that has always seemed to me obviously higher value than just automating operational processes, yet the bulk of the IT spend has always been on such projects. Perhaps this latest IBM move is another straw in the wind indicating that information management is finally moving up the agenda?

Not quite dead yet

2006-02-16T08:13:00.000-08:00

Infoworld has a piece intriguingly titled "The Death of the Software salesman". Those of us who have been on the receiving end of high pressure negotiating tactics from software companies may simply want to ask "when please?". However rather than being a glorious article about lynch mobs of customers, this article is more mundanely concerned with open source, and how this model may be an alternative to traditional software licensing. It observes that 40% of software company budgets go on sales and marketing, and indeed this estimate is a little on the low side. Enterprise software companies will typically spend 45-55% of their budgets on sales and marketing, with the lion's share of this going to sales, depending on the stage of the company (obviously this may be lower in very early stage companies which are still mostly in R&D).

It is all a bit ironic. The incremental cost of printing one more CD with a software product on it is less than a dollar, which is why venture capital firms like software companies. However to actually convince anyone to shell out (say) half a million dollars on this CD requires a great deal of expensive sales and marketing effort. It is rather naive of the panelists at the Open Source Business Conference to believe that this is going to change any time soon outside of a few niches. Sure, Linux has done very well, but some of this success is because IBM has put a lot of muscle into it (to avoid Microsoft eating further into the server operating system market). However if you move up a layer or two in the stack, open source is still in the special interest category. MySQL gradually improves, but it is still a long way off being a heavy duty DBMS; Oracle, DB2 and Microsoft are far from quaking in their boots. Higher still, there are very early initiatives in business intelligence like Pentaho and good luck to them, but not even the wildest-eyed open source advocate could accuse them of having made even a dent in the enterprise BI market yet.

Hence, while the idea of running a mostly R&D company and letting the customers come to you may sound appealing to some engineering types, the sad reality is that this is not going to happen. Customers buy through a series of stages. One model of this is called AIDA "awareness" -> "interest" -> desire-> action. Unless people are made aware of your product then they cannot buy it, and so you have to spend money on marketing. Once they have become aware and show interest in the value it offers them, they need to be nudged along, tantalised and have their many objections overcome ("does it really work","how many customers are using it", "will it work with my existing technology" etc). If you make it to this stage then the reality for any enterprise software is that you have what is called the "complex sale" i.e there are multiple people who influence the decision, each of whom have to be convinced or at least be persuaded to not object. Miller Heiman and others make a living by selling training in sales methodologies that go through this. It is very rare for a six or seven figure software purchase to involve just one person, and that's where sales come in. The salesman needs to get the proposition in front of the prospect, find if there is a project that fits, identify the key influencers in the account, see whether they really have a budget or just like talking to salesmen, navigate the client organization and unravel its purchasing process, perhaps deliver a trial or proof of concept, and all this before you get anywhere near negotiating a deal.

I just can't see all this happening without a lot of marketing and sales effort, except in very specific situations or niches that can suit open source, and those are too rare at present to put enough money on the table to pay for all those creative software engineers. I fear that, like Mart Twain's demise that was mistakenly reported during his lifetime, the death of the software salesman is being much exaggerated.

Customer satisfaction pays

2006-02-15T02:56:00.000-08:00

The software industry is full of awards for the fastest growing companies e.g. the "Inc 500" and numerous others, but it is perhaps revealing that the industry is almost silent on a critical measure: customer satisfaction. There seem to be two objective ways of measuring this: renewal rates i.e. do the customers actually pay maintenance, and surveys. The former has the advantage of being wholly objective i.e. they either pay up or they drop the software, with no wiggle room for spin, though of course it is a somewhat crude measure. A McKinsey report I saw recently reckoned that best practice in the software industry has renewal rates of 85%-95%, and indeed SAS Institute has made considerable (and justified) play over renewal rates of over 90% year after year (I am pleased to say that Kalido's renewal rate in 97%).

The other measure is surveys, which are a richer measure but of course are always subject to the perversion of framing the question e.g. "are you happy with our software?" is not the same question as "are you delighted with our software?". One company that has had a lot of fun at some companies' expense in Nucleus research, who sometimes call up the reference customers of software companies (as featured on their web sites) and ask them how happy they are. An amusing one a few years back was one they did on Siebel, where more than half of the Siebel reference customers they spoke to reported that their projects had cost more than the benefits. It was a similar story when they surveyed I2 reference customers, and an ERP report they produced showed that no ERP vendor could must even a 50% score in response to the question: "would you recommend this software to others", which is pretty shocking given that these are the reference customers of these vendors (highest scoring was Peoplesoft at a hardly dazzling 47%).

Given that endless studies have shown how more expensive it is to sell to new customers that to sell more to existing ones (four times more is one figure often quoted), I would have expected that software companies, with their high sales and marketing costs, would pay more attention to customer satisfaction. However, sad to say that my own experience as a customer for many years taught me that most software companies treat their customers with anything from indifference to outright hostility. Perhaps in the giddy days of the late 1990s software companies could get away with this, but these days the power is definitely back with the buyer, and so it is in the industry's interest to improve customer satisfaction, if only for purely selfish reasons. Yet how many software companies even survey their customers on a regular basis? Talking to sometimes unhappy customers may not always be a comfortable experience for software executives, but it can teach you a great deal, and showing that you care enough to listen is itself a way of raising customer satisfaction.

The price of love

2006-02-14T09:48:00.000-08:00

As it is Valentine's day I will depart from my usual carping about the latest creative marketing of the technology industry and observe that, according to the BBC, the escalating price of love (or at least the price of dating services) has caused on-line dating revenues to actually drop in the US. Fortunately those romantic Europeans (perhaps the Germans or Swiss, tantalizingly it is not clear) have kept their end up, with Europe's on-line dating market growing 43% this year. It is a tribute to the power of marketing that we all feel obliged to splash out on cards, presents, overpriced flowers and squeeze into overcrowded restaurants all on the same day. However it would seem that we can blame the Romans and even their predecessors for this one rather than a modern greeting card company for this particular annual celebration.

There surely have to be a range of opportunities for technology companies here. For a start, on-line restaurant booking services like Toptable in the UK make it a little easier to find that elusive table. Perhaps someone could come up with a roses futures market that allowed one to hedge against the predictable hike in the price of roses on or around the 14th February? There are already on-line greeting card services, though good luck convincing your loved one that an on-line card is an adequate substitute for a real card.

I can't find a single software acquisition happening today - our industry just has no romance...

Happy Valentine's day to you all.

Technology Planning

2006-02-13T02:41:00.000-08:00

I spent much of my career in technology planning, both in Exxon and Shell. These organizations are quite contrasting in their culture despite being in the same industry: Exxon is the uber-centralist, with everything driven from head office, and very little room for variation allowed in its subsidiaries. Shell has changed over the years but has traditionally been extremely de-centralized, allowing its subsidiary operating companies a great deal of autonomy; in the mid to late 1990s Shell has tried to become more centralized in its approach, though still nothing like as much as Exxon.

What worked best in technology planning in these different companies? Exxon's strength was its standardization at the infrastructure level. There were rigid standards for operating systems, databases, email, telecoms protocols and even desktop applications, long before most companies even dreamed up their "standard desktop" initiatives. This avoided a lot of problems that many companies had e.g. with varying email systems, or differing database applications, and critically meant that skills were very transferable between jobs and between Exxon subsidiaries (important in a company where most people move jobs every two to three years). By contrast Shell was fairly anarchic when I joined, even having different email systems, desktops using anything from Windows to UNIX desktops, and every database, TP monitor, 4GL etc that was on the market, though often just one of each. In 1992 I did a survey and recorded 27 different BI/reporting tools, and that's just the ones I could track down. It was perhaps not surprising that Shell spent twice as much on IT as Exxon in the mid 1990s, despite the companies being about the same size (Exxon is now a lot bigger due to the Mobil acquisition). On the other hand Shell had excellent technology research groups, and many collaborative projects between subsidiaries, that helped spread best practice. Also, since operating companies had a lot of autonomy, central decisions that proved unsuitable in certain markets simply never got implemented rather than being rammed down subsidiary's throats.

It was certainly a lot more fun being a technology planner in Shell, as there were so many more technologies to tinker with, but it was also like herding cats in terms of trying to get decisions on a new technology recommendation, let alone getting it implemented. In Exxon it was extremely hard, and probably career-limiting, to be in a subsidiary and try and go against a central product recommendation; in Shell it was almost a badge of honor to do so. Shell's central technology planners did, though, make some excellent decisions e.g. they were very early into Windows when the rest of the world was plugging OS/2, and they avoided any significant forays into the object database world at a time when analyst forms were all writing the obituaries of relational databases.

Having worked in both cultures, I believe that the optimum approach for technology planners is to try and standardize as rigidly as your company will let you on things that have essentially become commoditized. For example, who really can tell the difference between Oracle, DB2 and SQL Server any more? For the vast majority if situations it doesn't matter, and it is more important to have one common standard than it is to get the "right" one. On the other hand, in an emerging area, it is just self-defeating to try and pick a winner at too early a stage. You do not want to stifle innovation, and the farther up the technology stack, the less harm a few bad decisions are likely to make. For example, get the wrong operating system (like OS/2) and it is a major job to rip it out. Get the wrong web page design tool and there is less damage done.
Moreover, at the application level there is likely to be a clearly quantified cost-benefits case for a new technology, since applications touch the business directly e.g. a new procurement system or whatever. At the infrastructure level it is much harder to nail down the benefits case, as these are shared and long-term. If your new application has a 9 month payback period, then it matters less if one day it turnes out to be "wrong", but you don't want to find this with your middleware message bus product. There are lots of hobbyists in technology, and few products at the infrastructure level are truly innovative, so getting the lower levels of the stack standard is well worth doing.

Overall, while both companies are clearly highly successful, I think on balance that Exxon's strong centralization at the infrastructure level is more beneficial from a technology planning viewpoint. Quite apart from procurement costs, the skills transfer advantages are huge: if your DBA or programmer moves from the UK to Thailand, he or she can still use their skills rather than starting from scratch. However Shell's greater willingness to try out new technology, especially at the application level, often gave it very real advantage and rapid payback, even if overall IT costs were higher.

What is perhaps interesting is how the technology planning needs to reflect the culture of the company: a company whose decision making is highly decentralized will struggle greatly to impose a top-down technology planning approach, whatever its merits.

One more time

2006-02-10T02:32:00.000-08:00

An article from Wayne Eckerson, research director at The Data Warehouse institute (TDWI) has some sound advice about how to revitalize a data warehouse, based on a case study by Greg Jones of Sprint Nextel Corp. As the article says:

"Many data warehouses are launched with much fanfare and promise but quickly fail to live up to expectations"

and indeed multiple studies have shown high failure rates for data warehouse projects. I was at a Gartner conference earlier this week when an analyst stated that "the vast majority" of business intelligence initiatives fail to deliver tangible value. Yet, as a wise colleague of my often says:

"There is never time to do it right, but always time to do it again"

By this he means that data warehouse projects cut corners and make simplifying assumptions in their design about how the business works. It is much harder to make the design truly robust to business change, and yet this inability to deal properly with major business change is what eventually leads to problems for most data warehouses. A reorganization occurs, and it takes three months to redesign the star schema, fix up the load routines, modify the data mart production process, test all this etc. In the mean time the business is getting no up-to-date information. What do they do? They knock up a few spreadsheets or perhaps something quick in MS Access, "just for now". Then another change happens two months later: the company buys another company, which of course has different product codes, customer segmentation, cost allocation rules etc to the parent. Putting this new data into the warehouse is added to the task list of the data warehouse team, who have yet to finish adapting to the earlier reorganisation. The business users need to see the whole business picture right now, so extend their "temporary" spreadsheet or MS Access systems a little bit more. Since they have control of these, they start to do more using this, and after a time it hardly seems like the data warehouse is really that necessary any more. Of course they let the IT people get on with it (it is not their budget after all) but usage declines, and they give up telling the data warehouse team about the next major new requirement, as they never seem to see results in time anyway. Eventually the data warehouse falls into disuse. Eventually a new manager comes in and finds the spreadsheet and MS Access mess to be unmanageable, and a new budget is found to have another go, either from scratch or by rewriting the old warehouse. And so the cycle begins again.

Sound familiar? The overriding issue is the need to reflect business change in the warehouse quickly, in time for the business customers to make use of it, and before they start reverting to skunkworks spreadsheets and side solutions that they can get a contractor to knock up qucikly.
Until the industry starts adopting more robust, high quality modeling and design approaches, such as that based on generic modeling, this tale will repeat itself time and time again. The average data warehouse has 72% maintenance costs i.e. if it costs USD 3M to build, it will cost over USD 2M to maintain, every year. This is an unsustainable figure. Still, there will always be a new financial year, and new project budgets to start again from scratch.....

Another one bites the dust

2006-02-09T02:52:00.000-08:00

I had written quite recently about the consolidation occurring in the data quality industry. The pace picked up today, when First Logic, having escaped the clutches of Pitney Bowes, looks as if it will be acquired by Business Objects.

This acquirer is a lot more logical for First Logic than Pitney Bowes. Data quality is a major company of any BI/data warehouse implementation, and indeed Business Objects has already been reselling First Logic for over a year, so the two companies already know each other. The bargain basement price (USD 69M purchase for a company with revenues of over USD 50M) tells you all you need to know about the health of the data quality market.

This move supports my thesis that data quality as an independent market is essentially disappearing, with the capabilities being baked into broader offerings. I believe the same fate awaits the ETL market; more on this later.

The India syndrome

2006-02-03T06:32:00.000-08:00

One of the interesting effects of the rise and rise of India as an offshore location for technology staff is the effect that it is having on the prices that IT consulting firms can charge. We had a situation at Kalido where four well-known systems integration firms bid on a project, and in the end the customer chose none of them, but went in-house with significant input from staff in India. I also encountered a situation last year where a very well known firm was charging just USD 650 a day on a large project for its junior staff, a rate that would have been unthinkably low in 2001, when nearly double that would have been the going rate even for IROCs (idiots right out of college).

If you look at Accenture, perhaps the leading IT consulting firm other than IBM, you will see that its overall business is still growing, but in fact there are two trends: consulting has declined a lot, but outsourcing has risen to take its place. Even Accenture has been unable to protect its premium consulting pricing across the board under the onslaught of lower prices from Indian firms like TCS, Wippro and Infosys. The large consulting firms have responded by setting up large operations in India themselves ("if you can't beat 'em, join 'em") but while this has no doubt helped, the daily rates that consultants can charge in the US and Europe is still affected by this downwards pricing pressure. It's not that the Indian companies are giving it away: Wippro's profit margins are twice that of IBM.

Of course not every job can be done remotely. Support call centers and programming and testing to specifications are clearly the easiest to do, while projects that require a high degree of iteration e.g. web sites, user interface design, or reporting systems are much less suitable. Still, companies are now moving more complex work overseas e.g. accounting and financial research, so the list of "safe" jobs in IT in the developed world is gradually being eroded away.

Of course this movement has caused wage inflation in India, with 20% pay increases common amongst hot skills, and turnover rates of over 20% in Bangalore being normal (these can hit 50% for call centre jobs). Nonetheless, there is a long way to go. A top programmer with five to ten years C++ experience in the UK or the US might earn well over USD 100k (more in Silicon Valley) but the equivalent in Bangalore is still around USD 15k. It is less in Chennai or Pune. It is going to be a long time before inflation brings Indian wages up to anything like US levels.

This structural price deflation effect still has a long way to play out, with off-shoring growing steadily but still by no means universal, and large companies exploiting the lower prices to push down consulting rates in the US and Europe. It isn't going to get prettier any time soon. I was in India last week, and the sense of momentum and progress is tangible.

Missing the Boat

2006-02-02T00:49:00.000-08:00

One of the things that has bewildered me over the last year or two (and there are plenty of things that bewilder me) is how the data quality vendors have seemed oblivious to the emerging trend of master data management (MDM). On the face of it, there are few sectors more in need of a fillip. Data quality, which involves a lot of human issues such as data governance, and getting business people involved, is a hard sell. Rooting out errors in data is hardly the sexiest area to be working in, and as the solution is only partially provided by technolopgy, projects and initiatives here are prone to failure (human nature being what it is). The space has seen significant consolidation on recent years: Avellino was bought by Trillium, Evoke was bought by Similarity systems, Vality by Ascential (now IBM), Group 1 also by Pitney Bowes, who also made an abortive attempt to buy First Logic (if you can figure that strategy out, answers would be gratefully received), while Trillium is owned by Harte Hanks. Now Similarity systems has in turn been acquired by Informatica. Not the sign of a flourishing sector.

Surely then data quality vendors should have seized on MDM like a drowning man would at a life-raft? Data quality issues are a significant element of master data management, and while having software that can match up disparate name and address files is a long way from having a true MDM offering, remember that this is the tinseltown world of high-tech marketing, where a product can morph into another field with just a wave of a Powerpoint wand. Data quality vendors certainly ought to have grasped that matching up disparate definitions of things like "product" and "customer" was at least related to what their existing offerings did, and could have launched new MDM-flavored offerings to pick up on the coat-tails of the nascent but burgeoning MDM bandwagon. Instead there hasn't been a peep, and vendors have resigned themselves to being picked off by in some cases somewhat odd acquirers (Pitney Bowes, for example, is a direct mail firm; does it really grasp what it takes to be an enterprise software vendor?). Having avoided the clutches of Pitney Bowes, First Logic is now making progress in talking about MDM, but it is not perceived by the market as an MDM vendor. Elsewhere in the data quality industry, the silence around MDM is deafening.

As the data quality market essentially disappears into the portfolios of integration companies like Ascential (now IBM) and Informatica (which at least make logical sense as buyers), and assorted others, the executives of some of these companies surely must be wondering whether they missed a trick.

Data Quality is so retro darling

2006-01-31T07:14:00.000-08:00

Mark Smith (CEO of research company Ventana) writes about data governance as an important area for companies. He rightly points out that technology solutions are only part of the answer, and that organizational issues are at least as important. Strikingly, just 26% of large companies include master data management in their data governance initiatives. Perhaps some of this gap is in terminology, but this is somewhat disturbing since it raises the question: exactly what numbers are most companies using to make their decisions?

From my recollection of working in two of the largest companies in the world, it was best not to dig too deeply into the management information figures much of the time; data quality was a perennial problem, as was the endless "my figures are different to your figures" discussions. As Mark Smith points out, a lot of the issue is getting the ownership of data firmly with the business. Shell carried out an excellent initiative in the late 1990s in defining a common business model (at least down to a certain level of detail) and getting some business ownership of this business model, but even after this it was still a major challenge. It is certain that other large companies have the same issues. What is clear is that data quality and ownership of definitions cannot be an IT issue; it is critical that the business people step up and take control over their data, since they are the ones best placed to understand inconsistencies and problems that someone who has an IT background may overlook.

A good thing about the emerging interest in master data management is that it highlights previously neglected issues in the "data quality" field, that was previously a tough sell internally. Hands up all those volunteers for a data quality project? It was never what you might term a fashionable subject. Yet a lot of issues in MDM are actually related to data quality, so perhaps now that MDM is trendy we can dust off some of those old data quality books and make some better progress than occurred in the 1990s.

To host or not to host, that is the question

2006-01-26T09:25:00.000-08:00

The rise and rise of salesforce.com has triggered a shift in licensing amongst a number of companies, both startups like Rightnow and "me too" defensive offerings from the likes of Oracle and SAP. Siebel missed the boat entirely here, though its problems were by no means confined to its licensing model. The fact that it was massively over-marketed, with surprisingly limited core functionality given its price tag, requiring vast consulting resources to tailor every individual implementation, may have contributed also. A friend who had spent two years implementing Siebel at a bank described it as a "million dollar compiler", since everything that he wanted to do with it required customized programming/consulting.

Leaving Siebel aside, what are the broader implications of hosting and renting software. There are issues both ways. Clearly from a customer viewpoint they don;t have the hassle of installing and doing technical upgrades in-house, so no complaints there. However it is not obvious that a hosted piece of software is any more likely to meet your needs, or to require less tailoring, that one that is installed on-site. There may be (perhaps justified) concerns about the security of your own data, and indeed on related intellectual property. From personal experience I recall needing to back out from a hosted service, and having great difficulty in getting the vendor to export my data into a form that I could easy load into a transportable form - er, hello, this was my data after all.

From the vendor viewpoint there are also pros and cons. If you start from scratch then things are easier. Hosting services are much cheaper to provide than many customers realize, so margins can be very attractive. Because you are in control of the implementation, you have less issues at particular customers, who have installed some wacko combination of middleware (usually the day before some critical business deadline) that means you can't replicate their problems. On the other hand, it is hard to grow as fast. Recurring revenue is great, but there is a danger that it can become "the software maintenance without the license" unless you are one of the vendors who have managed massive momentum (like salesforce.com). This is also a big problem for existing vendors, whose business models and sales force are geared towards license revenue. Also, as a software vendor you may not want to be in the data centre business.

What is the scale of the hosted software market? In a Red Herring article the hosted software market is reckoned to be 1.5% of the total USD 72 billion global software market in 2004 (according to IDC). This may to be a big dent overall, but it is still a fair chunk of software, and one that is growing rapidly (doubling by 2009 is IDC's estimate). Cynics would argue that the whole "ASP" model (remember that) was going to change the world in the 1990s, but the demise of Exodus and other high-profile companies domonstrated the limits of how prepared customers were prepared to go in shifting their data Centrex to a 3rd party.

My instinct is that this is a very real trend, and that the conservatism of corporate buyers may be the major inhibitor at present. Certainly the headaches that big companies have in installing new versions of software is huge - many companies have "standard desktops" that are unpopular with end users and result in an army of people ensuring that everything works (usually on some ancient version of MS Office), so removing this problem has undeniable, and large benefits. For this reason alone, I think many companies will get over their queasiness at having their data stored somewhere off-site, and so this is a trend with real legs.

Ironically this may also prove right those old mainframe die-hards in the 1980s who felt that client/server was lunacy, and that distributing applications around hundreds or thousands of desktops was going to cause far more trouble than it was worth. Certainly more trouble than it ever caused on the good old mainframe, where you knew exactly what version of the packages everyone was using. It is perhaps no accident that IBM is at the forefront of this "on demand" trend, as the desire to centralism is in IBM's corporate bones.

"Near-shoring" continues apace

2006-01-26T09:19:00.000-08:00

For European companies considering outsourcing their IT, there is a nearby alternative to India: Eastern Europe. In 2005 This market was worth 149 million Euros in Hungary, 132 million in the Czech Republic and 201 million in Poland, according to a survey by PAC. In Hungary's case this represents 16% growth, 11% for the Czech Republic and 9% for Poland. Recent high profile examples have been DHL's move to Prague and Exxon's setting up shop in Warsaw and Budapest.

There is much logic to this. The eastern European countries had a fine tradition of education e.g. in standardized tests, Hungary's student maths scores are higher than the US and the UK. Budapest IT salaries are around one third of London, and although they are rising it can be seen that they will take many years to get anywhere near Western European levels. Nonetheless, this is still something of an area for pioneers. In terms of maturity, in my own experience the Czech Republic was comfortably the best established, followed by Hungary, with Poland lagging. In Prague and Budapest it is possible to find several companies who have successful operations, and at least a few recruitment agencies etc geared up to service them. This was much tougher in Poland, let alone the "wild east" of Russia or Romania, or even Belarus or Ukraine.

However it remains to be seen whether these countries will grab much market share from India, which has great advantages of scale, many years of success in this area, and a largely English-speaking workforce. Salaries in India are still a fraction of those in Hungary, even in Bangalore (never mind Chennai, Hyderabad etc) so their economic advantage looks secure for years, while the sheer number of large companies that have trodden the path to Bangalore means that, ironically, India may actually be of lower risk than Eastern Europe, at least in terms of being "proven". The greater travel time and time-zone differences are the main drawback here, but if India can work for US companies with a 11.5 hour time difference, why not for UK companies with a 5.5 hour time difference? (yes, India's time zones are measured on half-hours, a relic of the English civil service).

Some companies will worry that the economic benefits for the more advanced/"safer" places like Budapest and Prague may be transitory. Ireland used to be a popular cheap location, but years of EU-fuelled growth at rates of 8% have now brought Dublin close to the UK in terms of IT salaries. This might indicate that, given the setup costs and risks, it is better to go the whole way and go to India, where the wage differences are so vast that it will take decades to get to Western levels, even at the current high salary growth. One interesting recent step was the Indian firm Satyam opening an office in Hungary. The idea is that customers who want to try off-shoring but are nervous can start in Hungary, and then move to India as they grow more confident. Eastern Europe also has one advantage over India - continental language skills, which would be important for markets such as France and Germany.

Certainly the "India effect" is having a structural effect on IT consultancy prices. Even Accenture has been forced to cut daily rates, and indeed Accenture's consulting revenues are actually in decline, but their overall figures are holding up to a rapid rise in outsourcing deals that is making up for the loss in consulting revenues.

Of course there have been some well-publicized problems, such as Dell's abortive Indian help-desk, which I can testify from personal experience had big problems, but there seem to have been more successes than failures. After all, it is not as if IT projects in the US or UK all go swimmingly well.

Just as manufacturing has, to a large extent, moved to China, it may be inevitable that more and more IT jobs head off to India, and to a lesser extent Eastern Europe. The economics are compelling, and as more and more companies make it work, it feels less like a pioneering activity for the mainstream, which will fuel further growth.

A data warehouse is not just for Christmas

2006-01-20T02:48:00.000-08:00

A brief article by Bill Inmon addresses a key point often overlooked - when is a data warehouse finished? The answer is never, since the warehouse must be constantly updated to reflect changes in the business e.g. reorganizations, new product lines, acquisitions etc.

Yet this is a problem because today's main data warehouse design approaches result in extremely high maintenance costs - 72% of build costs according to TDWI. If a data warehouse costs USD 3M to build and USD 2.1M to maintain annually then over five years you are looking at costs well over USD 11m (let's generous allow a year to build plus four years of maintenance) i.e. many times the original project cost. These levels of cost are what the industry has got used
to, but these are very high compared to maintenance costs for OLTP systems, which ttypically run at 15% of build costs annually. This high cost level, and the delays in responding to business change when the warehouse schema needs to be updated, contribute to poor perception of data warehouses in the business community, and high perceived failure rates. As noted elsewhere, data warehouses built on generic design principles are far more robust to business change, and have levels of maintenance around 15%.

If the data warehouse industry (and the business intelligence industry which feeds on it) is to continue to grow then it needs to grow up also, and address the issue of better data warehouse design paradigms. 72% annual maintenance costs are not acceptable.

Desperate Data Warehouses

2006-01-19T07:40:00.000-08:00

A Gartner Group report mentions that at least 50% of data warehouse projects fail. Of course on its own this sounds bad, but just how bad is it, and what is meant by failure e.g. is being one month late failure, or does it mean complete failure to deliver? How do IT projects in general do? Standish Group run a fairly exacting survey which in 2003 covered 13,522 IT projects, a very large sample indeed. Of these just 34% were an "unqualified success". Complete failure to deliver were just 15%. The rest are in the middle i.e. they delivered but were not perceived to be complete successes in some way. To be precise: 51% had "cost overruns, time overruns, and projects not delivered with the right functionality to support the business". Unfortunately the Gartner note does not define "failure" as precisely as Standish; they define the "over 50% as being "limited acceptance or be outright failures". It is also unclear whether the Gartner figure was a prediction based on hard data, or the opinion of one or more of their analysts.

The Standish study usefully splits the success rate by project size, with a miserable 2% of projects larger than USD 10M in size being complete successes, with 46% of projects below USD 750k being complete successes, 32% up to USD 3M and, 23% at USD 3-6M and 11% at USD 6-10M. The average data warehouse project is somewhere around the USD 2-5M range, with USD 3M often quoted, so indeed on this basis it would seem we should only expect around 25% or so to be "unqualified successes". Unfortunately I don't have data available for the failure rate split by size, which presumably may follow a similar pattern, and the rather loose definition that Gartner use makes it hard to compare like with like.

Even if turns out that data warehouse projects aren't any (or at least much) worse than other IT projects, this is not a great advert for the IT industry. The Standish data most certainly gives a clear message that if you can possible reduce the scope of a project to smaller, bite-sized projects, then you greatly enhance your chance of success. It has long been known that IT productivity drops as projects get larger. This is due to human nature - the more people you have to work with, the more communication is needed, the more complex things become, and the more chance of things being misunderstood or overlooked.

It is interesting that even very large data warehouse projects can be effectively managed in bite-sized chunks, at least if you use a federated approach rather than trying to stuff the entire enterprise's data into a single warehouse. Projects at BP, Unilever, Philips, Shell and others have taken a country by country approach, or a business line by business line approach, with individual warehouses feeding up to either regional ones or a global one, or indeed both. In this case each project becomes a fairly modest implementation, but there my be many of them. The Shell OP MIS project involved 66 separate country implementations, three regional and one global. Overall a USD 50M project, but broken down into lots of manageable, repeatable pieces.

So, if you data warehouse project is not to become desperate, think carefully about a federated architecture rather than big bang. This may not always be possible, but you will have a greater chance of success.

MDM Trends

2006-01-18T06:34:00.000-08:00

In DM Review we see some "2006 predictions", something that journalists cannot resist doing each January, whatever the subject. In this case the article seems curiously limited to commonest about "customer". Certainly customer is an important example of master data, and indeed there are several products out there that specialize in this (so-called CDI products, like DWL, recently bought by IBM). However it is a common misapprehension that MDM is just about "customer" and "product". It is not. One of our customers, BP, uses KALIDO MDM to manage 350 different types of master data, of which just two are customer and product. Large companies also have to worry about the definitions of things like "price", "brand", "asset", "person", "organization", "delivery point", etc, and probably don;t want to buy one software product for each one.

MDM, as an emerging area, is particularly tricky to make predictions about. For what it is worth, I predict that in 2006:

1. There will be several more acquisitions in the space, as large vendors decide that they need to have an offering of some kind, if only to fend off competitive criticism or gaps on RFI checklists. However, caveat emptor here. The better products, like Trigo, have already been snapped up.
2. At least one analyst firm will publish some form of "MDM architecture" diagram that will attempt to classify MDM into different areas, in order to try and elevate that firm's perceived "thought leadership" on the issue.
3. There will be the first "MDM project disaster" headlines as early adopters begin to move from Powerpoint into more real project implementations. Inevitably, some will not go according to plan.
4. SAP MDME will prove as problematic as the original SAP MDM, which is down pushing up the daisies in a software cemetery near Walldorf. A2i was a poor choice as a platform for a general purpose MDM tool that SAP needs, and this realization will start to sink in when customers start to try it out.
5. Management consultancies, who until mid 2005 could not even spell master data management, will establish consulting practices offering slick Powerpoint slides and well-groomed bright young graduates to deliver "program management" around MDM, with impressive looking methodologies that they are so hot off the presses that the ink is barely dry.
They will purport to navigate a clear path through the maze of MDM technologies and will certainly not, in any way, be learning on the job at the client's expense.

The next generation data warehouse has a name

2006-01-16T09:26:00.000-08:00

Bill Inmon, the "father of the data warehouse" has come up with a new definition of what he believes a next generation data warehouse architecture should follow. Labeled "DW 2.0" (and trademarked by Bill) the salient points, as noted in an article in DM Review are:

- the lifecycle of data
- unstructured data as well as structured data
- local and global metadata i.e. master data management
- integrity of integrated data.

These seem eminently sensible points to me, and ones that indeed are often overlooked in first generation custom-build warehouses. Too often these projects concentrated on the initial implementation at the expense of considering the impact of business change, with the consequence that the average data warehouse costs 72% of implementation costs to support every year e.g. a USD 3M warehouse would cost over USD 2M to support; not a pretty figure. This is a critical point that seems remarkably rarely discussed. A data warehouse that is designed on generic principles will reduce this figure to around 15%.

The very real issue of having to deal with local and global metadata including master data management is another critical aspect that has only recently come to the attention of most analysts and media. Managing this i.e. the process of dealing with master data, is a primary feature of large-scale data warehouse implementations yet the industry has barely woken up to this fact. Perhaps the only thing I would differ with Bill on here is his rather narrow definition of master data. He classifies it as a subset of business metadata, which is fair enough, but I would argue that it is actually the "business vocabulary" or context of business transactions, whereas he has a separate "context" category. Anyway, this is perhaps splitting hairs. At least it gets attention in DW 2.0, and hopefully he will expand further on it as DW 2.0 gets more attention.

The integrity of "integrated" data addresses the difference between truly integrated data that can be accessed in a repeatable way, and the "interactive" data that needs to be accessed in real-time e.g. "What is the credit rating of customer x" that will not be the same from one minute to the next. Making this distinction is a useful one, as it has caused much confusion whereby EII vendors have claimed that their way is the true path, when it patently cannot be in isolation.

I am pleased that DW 2.0 also points out the importance of time-variance. This is something that is often disregarded in data warehouse designs, mainly because it is hard. Bill Inmon's rival Ralph Kimball calls it the "slowly changing dimension" problem, with some technical mechanisms for how to deal with it, but at an enterprise level, these lessons are often lost. Time variance or "effective dating" (no, this is not like speed dating) is indeed critical in many business applications, and indeed is a key feature of Kalido.

It would indeed be nice if unstructured data mapped neatly into structured data, but here we are rather at the mercy of the database technologies. In principle Oracle and other databases can store images as "blobs" (binary large objects) but in practice very few people really do this, due to the difficulty in accessing them and the inefficiency of storage. Storing XML directly in the DBMS can be done, but brings its own issues, as we can testify at Kalido. Hence I think that the worlds of structured and unstructured data will remain rather separate for the foreseeable future.

The DW 2.0 material also has an excellent section on "the global data warehouse" where he lays out the issues and approaches to tackling deploying a warehouse on a global scale. This is what I term "federation", and examples of this kind of deployment can be found at Unilever, BP and Shell, amongst others. Again this is a topic that seems to have entirely eluded most analysts, and yet is key to getting a truly global view of the corporation.

Overall it is good to see Bill taking a view and recognizing that data warehouse language and architecture badly needs an update from the 1990s and before. Many serious issues are not well addressed by current data warehouse approaches, and I welcome this overdue airing of the issues. His initiative is quite ambitious, and presumably he is aiming for the same kind of impact on data warehouse architecture as Ted Codd's rules had on relational database theory (the latter' "rules" of relational were based on some mathematical theory and were quite rigorous in definition). It is to be hoped that acertificationtoin" process for particular designs or products that Bill develops will be an objective process rather than one based on sponsorship.

More detail on DW 2.0 can be found on Bill's web site.

Well, there's a surprise

2006-01-16T07:42:00.001-08:00

A research piece shows some facts that will not stun anyone who has had the joy of living through an ERP implementation. According to a new study:

one third of users leave large portions of ERP software entirely unused
just 5% of customers are using ERP software to its full extent
only 12% install ERP "out of the box"
over half did not measure return on investment of their IT applications.

The only thing surprising about these figures is how implausibly good they are. According to Aberdeen group, only 5% of companies regularly carry out post-implementation reviews, so "less than half" seems wildly optimistic there. Moreover, just who are these 12% of companies who install ERP "out of the box" with no modification? Not too many in the real world, I suspect. Similarly, very few companies implement every module of an ERP suite, so the figures on breadth of usage seem also unremarkable.

Many ERP implementations were banged in to try and avert the Y2k catastrophe that never happened, but there were plenty before that, and plenty since, including numerous ERP consolidation projects (though there are fewer of these that ever look like finishing). I guess the scary thing here is the expectation gap between the people who actually paid the bill for these mega-projects, and the reality on the ground. However, as I have written about elsewhere, these projects are just "too big to fail" or at least to be seen to fail, as too many careers are wrapped up in them, so this state of denial seems likely to continue until a new generation of CIOs comes along.

Easier than quantum mechanics

2006-01-16T06:26:00.000-08:00

I laughed out loud when I saw an article today with the headline "Oracle Solution- Easier to Implement than SAP", but that isn't setting the bar real high, is it? SAP may be lots of things: successful, profitable, large, but no one ever accused their software of being simple and easy to implement. What next? "Accountants less creative than Arthur Anderson at Enron" or "now, a car more stylish than a Lada"?

This particular piece of marketing spin is supposedly around an "independent" study done on SAP BW and Oracle Warehouse Builder implementations at various sampled customers. I have to say I suspect that the study might just be paid for by Oracle, though that is not stated, given that this same market research firm also brought you articles such as "Oracle is 46% more productive than DB2". We all await with bated breath further independent research pieces showing that "Oracle solves world hunger" and "Why the world loves Larry".

However, in this case I don't doubt the veracity of the material (much). SAP has become a byword for complexity, with up to 45,000 tables per implementation. Business warehouse is not quite on this scale, but still involves lots of juicy consulting hours and most likely some programming in ASP's own proprietary coding language ABAP, which I am proud to see that I once took a course in (think: a cross between IBM assembler and COBOL). I haven't got direct coding experience with Oracle's tools, but I have to assume that they can't get murkier than this.

High tech marketing has come up with some entertaining headlines and slogans over the years, but "easier than SAP" is definitely my favorite in 2006 so far.

Application vendors and SOA

2006-01-06T09:08:00.000-08:00

In an article looking forward to trends in 2006, an Oracle executive raises an interesting point. The applications market for large enterprises has now essentially reduced to a field of two giants, SAP and Oracle, with a long gap now in size between them and vendors in particular niches such as supply chain or customer relationship management. Yet CIOs are demanding, as he puts it, "hot pluggable" applications, which is another way of saying easy inter-operability between applications. For example a company might like to be able to call up a specialist pricing application from a small vendor within their SAP or Oracle ERP application.

This creates a tricky dynamic for Oracle and SAP, who ideally would like to expand their own footprint within customers at the expense of each other (and other vendors). If SOA actually works, then they will be enabling customers to easily switch out the bits of their applications that customers dislike in favor of others, which is not in their interest. Of course Oracle also sells a middleware stack, and now SAP has entered the fray with Netweaver. By doing so they hope to switch the ground: if someone is going to call up a non-SAP application from within SAP, then SAP would rather that they did it using Netweaver protocols than a rival stack, such as IBM Websphere. Indeed they would really prefer that people didn't do this at all, but instead just use more and more SAP modules. The same goes for Oracle. Hence these two application vendors need to be seen to be playing the game with regards to inter-operability, yet it is actually more in their own interest if this capability does not work properly. IBM, who does not sell applications, is in a much cleaner position here, since they can only benefit by having genuine application inter-operability via Websphere, whoever the application vendors are. IBM does not sell applications, so only has a vested interest in selling more middleware in this context (and of course the consulting to implement it).

Customers need to be very aware of the desire by the application vendors to lock them into their offerings through their middleware, and should question how genuine the commitment of application vendors to true inter-operability really is. Just as turkeys don't vote for Christmas, why would a dominant application vendor really want their application to be split into bite-sized pieces that could be each attacked by niche application vendors that would not have the reach to challenge their monolithic applications without this capability?

Of course Oracle and SAP cannot actually say this out loud. IBM (and other independent vendors like Tibco), however, should, and potentially this ought to give them an edge in the coming middleware wars.

Flexibility need not imply anarchy

2005-12-23T07:04:00.000-08:00

An article by Rick Sherman ponders how data marts are finally being supplanted by enterprise data warehouses, at least according to a new TDWI survey. Yet he muddies the waters by saying that it is quicker to produce an EDW with no data marts than one with data marts, and so suggesting that sometimes, maybe, data marts are still the way to go.

Certainly a central enterprise warehouse without data marts or a decent way of producing them is inevitably going to do nothing to reduce the plethora of departmental data marts, since people will certainly want to get data that is relevant to them, and they'll do it one way or another if the IT department has too big a backlog. But surely this misses the point. Isolated data marts are a major problem - they can never allow the "single view" which many executives need to take across divisional or departmental boundaries, as without something central there are just too many combinations to ever allow consistency. However it should not be an either/or situation. A modern enterprise data warehouse should be perfectly capable of producing useful data marts on an as-needed basis. The data warehouse cannot afford to sit in glorious isolation or it will fall into disuse, and you will end up full circle with numerous disconnected data marts arising to get around its failings.

An important point not made in the article is that in really large organizations you may not be able to get away with just one warehouse. A company with many country operations may want to have a summary global warehouse as well as one per country. Each of these will have dependent data marts. The "local" data warehouses can feed up summary data to the global one. Indeed for truly huge companies this may be the only practical solution. Examples of such an architecture can be found at Shell, BP, Unilever and others. A major advantage of such a federated approach is that political control of the local warehouse remains within the subsidiaries, which avoids "them v us" issues where the local operating units resist something being imposed on them by central office. In this scenario the local subsidiary get a warehouse that suits their needs, and the central office gets its summary information as a side-effect. Such an approach is technically more complicate, but if you use a technology that is capable of being federated (which these do) then the overhead is not great, but you have the key advantage of retaining global v local flexibility.

A Christmas Message

2005-12-22T07:15:00.000-08:00

I must have nodded off in the armchair after a glass or two of seasonal cheer when I was confronted by an apparition - a ghost of data warehouses past. He was not a pretty sight - dating back to the mid 1990s and with a grumpy attitude. In those days all you had was a database and a compiler, and a customer who wanted to somehow bring together data from multiple, incompatible systems. There were no ETL tools, no data warehouse design books and little in the way of viewing the data once you had managed to wrestle it into the new data warehouse, just reporting tools that just about saved you coding SQL but confronted users with the names of the tables and columns, usually restricted to eight characters. Those were the days, a salutary reminder of how primitive things were.

Following this was a ghost of data warehouses present. This was a much chirpier looking fellow, who could gain access to data from even devious and recalcitrant source systems via ETL tools like Ascential, and had at least some idea how to design the warehouse. These designs had suitably festive names: "snowflake schema", "star schema". If we had a reindeer schema then it would have completed the festive scene. This wraith had a sackful of reporting tools to access the data, count it, mine it and graph it. Yet not all was well - the pale figure seemed troubled. Every time the source systems changed, the warehouse design was impacted, and teams of DBA elves had to scurry around building tables, sometimes even dropping them. Moreover the pretty reporting tools were completely unable to deal with history: when reports were needed from past seasons there was no recollection of the master data at the time, and it did not reflect the current structures. The only ones really happy here were the DBA elves, who busied themselves constructing ever more tables and indices - they love activity, and they were content.

Last came a ghost of data warehouse future. In this idyllic world, changes in the source systems have no effect on the warehouse schema at all. The warehouse just absorbs the change like a sponge, and can produce reports to reflect any view of structure, present, past or even future. There are less DBA elves around, but they have discovered new things to do with their free time - with SAP up to 32,000 tables these days, there is no unemployment in DBA elf land. Best of all the customers are happy, as they can finally make sense of the data as quickly as their business changes. New master data can be propagated throughout their enterprise like so much fairy dust. I have seen the future of data warehousing, and it is adaptive.

Well, enough of that. Time to talk turkey, or at least eat it. While doing so, it may be relaxing to look back at some of the turkeys that our industry has managed to produce over the years. Human nature being what it is, there is more than one web site devoted to the follies of the technology industry, and even to the worst software bugs. Have fun reading these. I recall being told by a venture capitalist that the dot-com boom of the late 1990s proved that "in a strong enough wind, even turkeys can fly". These are some that did not.

I would like to wish readers of this blog and very happy Christmas.

Hype or Hope?

2005-12-22T06:32:00.000-08:00

In an article today Peggy Bocks of EDS asks whether MDM is all hype. I think it is fascinating how some terms in technology catch on, while others wither away. In 2002 MDM was essentially an unknown term. I recall discussing with analysts calling our new Kalido product offering "Kalido MDM" and being greeted with polite derision, since this was "not a market". Customers seemed to recognize the problem though, and I discovered that we were not alone when SAP launched their own SAP MDM offering soon afterwards. Though that product is now retired, when a vendor the size of SAP anoints a term then you can be sure that the industry will take it more seriously than when a start-up uses it. Three years on and there is a much higher level of noise around MDM, with around 60 vendors now claiming to have some sort of MDM offering, at least in Powerpoint form. Moreover further validation has come from Oracle (with its Data Hub), Hyperion (who bought Razza) and IBM, who have bought several technologies related to MDM. IDC reckon the market for MDM will be worth USD 9.7 billion within fove years.

However, I do have some sympathy with Ms Bock's point - a lot of MDM at present is froth and discussion rather than concrete projects. Certainly Kalido has some very real MDM deployments, Razza has some, Oracle presumably does, and SAP managed about 20 deployments before giving up and starting again, buying A2i and retiring its existing offering. Still, outside the tighter niches of product information management and customer data integration (arguably a subset of the broad MDM market) this hardly constitutes a landslide of software deployments.

A skeptic would argue that the industry has got these things wrong before, getting all excited over technology that fizzles out. Remember how "e-procurement" was going to take over the world? Ask the shareholders of defunct Commerce One about that trend. I recall IDC projecting some vast market for object databases in a report they published in 1992, and at the time I wrote a paper at Shell arguing that the ODBMS market would likely never get even close to a billion dollars, and indeed it never did. However object databases always struck me as a solution in search of a problem, whereas the issues around managing master data are very real, and very expensive, for large companies, and they are not well addressed today. There are major costs associated with inconsistent master data e.g. deliveries being delivered wrongly, duplicate stock being held etc. Shell Lubricants though they had 20,000 unique pack-product combinations when in fact they had around 5,000, meaning major savings to be had through eliminated duplication in marketing, packaging and manufacturing, for example.

Because it addresses a real business problem, with the potential for significant hard business savings, I believe that the MDM market will in fact catch light and grow, but there will inevitably be a confusing period while analysts get their heads around the new market and start to segment it, and customers begin to understand the various stages they need to go through in order to run an effective master data project.

Go east young man

2005-12-21T03:12:00.000-08:00

Those who think the rise in the Chinese economy can be safely ignored for a few more years or is restricted to cheap toys, sneakers and steel can think again. In 2005 the world's top exporter of high tech products was not the United States, but China.

This was quite a landmark event, and something that technology companies need to consider carefully. China has become a major force in manufacturing, but is also starting to move into off-shoring. At this stage it lags India by a long way, but 1.2 billion people constitute an awful lot of potential programmers and engineers. At present India has the huge advantage that English is the most common second language, meaning that call centers and programmers can more easily pick up US software skills and communicate with western companies. Also there are far more established technology players in India, but this advantage will not continue forever. Napoleon once said "Let China sleep, for when it awakes it will shake the world" - after 9% economic growth rates for 20 years, it looks like the alarm clock has gone off.

Bangalore and BI don't mix

2005-12-19T01:56:00.000-08:00

An article by Nakis Papadopoulous asserts that offshoring does not work for business intelligence applications, though he spends little time discussing why. Off-shoring has clearly moved beyond the pioneering days. My first experience with it was a project when I was at Exxon back in the 1980s, which in itself tells you that ideas often take a long time to really catch on. With the likes of TCS and Wipro now huge companies in their own right, off-shoring has become fairly mainstream, is it ought to be possible to look at some lessons.

While perhaps any project can be off-shored, there is a spectrum of risk that you need to consider. The big Indian companies have developed a high level of process quality, many of them certified to CMM level 5, which is more than can be said for most western IT outfits (about three quarters of CMM Level 5 IT companies are in India), so there is a major focus on quality. However in order to really feel the benefits of this you are going to need a stable, high quality specification; the CMM process is big on repeatable, documented processes. On a conventional project, questions about the specification can be dealt with by someone wandering across a corridor, but this is more difficult many time-zones and thousands of miles away. Hence the more stable and well understood the requirements, the better the chance of success. A perfect example would be an interface program between two systems. This can (and must be) tightly defined, so there is minimal ambiguity. As IT systems (unlike human beings) always behave consistently, there is a high degree of stability where each "user" is an It system. This type of application would be a low-risk one to outsource.

Building transactional systems with a well-documented set of needs should be only moderately risky, provided the requirements are well-defined, which in many cases they will be. Similarly, testing is something that requires a series of well-documented cases and responses, so again can work well in an off-shore environment.

Even with interfaces, are not guaranteed, though. We had a recent project at a very large company where the extract-transform-load was off-shored. In principle this should be relatively low risk, but it turned out badly since, among other reasons, some of the systems to extract from had complex business rules that required a lot of explanation, and because the target system was a data warehouse, where there can be a lot of changes in the data needs as users refine their understanding for what they can do. Much of this work was brought back on-shore and the project is now live, but only after a scary phase.

At the high-risk end, building user-interfaces where a lot of prototyping is needed would clearly be awkward many time-zones away, as would systems where the requirements are rather loose. This is frequently the case in business intelligence, where the users often are only partly aware of what they can get out of a system until they have seen the actual data. Here it is rare to see bullet-proof functional specification documents, so an off-shore team is at an inherent disadvantage compared to one camped out next to the business users. This is above all the reason why business intelligence projects are some of the least suitable to be off-shored.

I'd be interested to hear of any experiences, good or bad, out there with respect to off-shoring. Does you own experience match that of the above, or not?

One size does not fit all

2005-12-16T02:42:00.000-08:00

An article in Computer Weekly today by Martin Fahy contains some excellent insights into why "single instance ERP" is mostly a fantasy. An academic, he has conducted a survey of CFOs and his research has unearthed found some interesting findings. Firstly:

"CFOs and senior finance executives prefer technology architectures that are robust across a wider ranger of organizational circumstances"

Spot on! One of the limitations of ERP systems is that they impose a rigid business model - indeed this was one of the selling points: "sweep away all that inefficient duplication and standardize around a single set of processes". The trouble with this is that businesses do have unique requirements in particular territories or markets, and are at different stages of development. What may make sense in a mature market such as Germany make not do so in a fast-growing developing market like China. The more widely the scope of a single ERP implementation, the longer it takes to implement and the more functionality that needs to be loaded onto the project. Changing this becomes ever more difficult given its scale, making the business less reactive to changing circumstances since its business processes are essentially frozen.

"CFOs in particular are skeptical about making further investments in second wave ERP until payoffs from the first wave of implementations is realized"

This is a woefully under-discussed topic. Lots of people made money in the yah-fuelled ERP boom of the 1990s: certainly vendors like SAP and Oracle, and definitely the consulting firms that implemented these systems. Yet how often have these systems actually delivered the returns that they promised? Since a pitiful 5% of firms actually carry out regular post implementation reviews (according to Aberdeen) few people really know, but my own experience at two multi-nationals would suggest this is not a topic that executives want to highlight. At one company I am familiar with, an ERP consolidation project is underway that is estimated to cost a billion dollars, and will take eight years. Given the track record of projects that size, it is hard to be optimistic that something won't change in that timeframe that will affect the project.

"users tend to find ERP functionality cumbersome, complicated and not in keeping with their established work patterns"

Indeed. I have written elsewhere about this.

In my view, CIOs spend too much time worrying about "simplifying" core infrastructure, where benefits are at best difficult to pin down, and not enough on truly value-added initiatives that business people can relate to, such as ones that enable better customer understanding.

The study concludes that "For the foreseeable future single-instance ERP will be a popular rhetoric, but a scarce reality". Given the real and very high costs of getting there (Nestle, the poster child of single instance ERP reportedly spent USD 3 billion dollars on this) and the ever-elusive payback, one wonders how any of these initiatives ever get signed off at all.

MDM Business Benefits

2005-12-15T02:25:00.000-08:00

There were some interesting results in a survey of 150 big-company respondents conducted by Ventana Research as to where customers saw the main benefits of master data management (MDM). The most important areas were:

better accuracy of reporting and business intelligence 59%
improvement of operational efficiency 27%
cost reduction of existing IT investments 8%

It is encouraging that respondents place such a heavy emphasis on business issues compared to IT, since quite apart from this sounding quite right (MDM can improve customer delivery errors, billing problems etc) they will have a much better chance of justifying an MDM project if the benefit case is related to business improvement than the old chestnut of reduced IT costs (which so rarely appear in reality - surely IT departments would have shrunk to nothing by now if all the projects promising "reduced IT costs"over the years had actually delivered their promised benefits). A nice example of how to justify an MDM project can be found in a separate article today, in this example specifically about better customer information.

The survey also reflects my experience of the evolution of MDM initiatives, which tend to start in a "discovery"phase where a company takes stock of all its master data and begins to fix inconsistency, which initially impact analytic and reporting applications. Later, after this phase, companies begin to address the automation of the workflow around updating master data, and finally reach the stage of connecting this workflow up to middleware which will physically update the operational systems from a master data repository. This last phase is where many of the operational efficiency benefits will kick in, and these may be very substantial indeed.

Based on the rapidly increasing level of interest in MDM, in 2006 I expect to see a lot of the current exploratory conversations turning into more concrete projects, each of which will need a good business case. At present MDM projects tend to be done by pioneering companies, so it will be very interesting to see if the various projections prove accurate and MDM starts to become more mainstream.

Visuals for the few

2005-12-14T01:49:00.000-08:00

There is a thoughtful article today by Stephen Few in the BI Network journal. In this he discusses the ways in which data can be presented to people, and gives a nice example of how a flashy graphic can be harder to interpret than a simple bar chart. There are some interesting follow-ups to this line of reasoning. Firstly, the vast majority of users of BI software have quite simple data display needs: they probably want to see a trend in data e.g. "are sales going up or down?" or answer a simple question like "what is the most profitable distributor?". Since the vast majority of BI tool users have no background in data analysis or statistics, it is at best pointless and possible self-defeating to provide them with much in the way of statistical tools or elaborate graphical display capabilities. If they don't understand statistical significance for example, then they may reach invalid conclusions by playing around with statistical tools.

This may explain why vendors who specialize in advanced data visualization never seem to really make it to any size. There are some very interesting technologies in this area e.g. Fractal Edge,(a technology that seems to me genuinely innovative) AVS, OpenDX and the tackily named The Brain. More established vendors would include SAS, who were one of the first vendors to really go in for sophisticated graphics and statistical tools, yet built up their considerable success mainly in other areas (e.g. they were about the only software that could make sense of an IBM mainframe dump, so became a standard in data centers; they have since diversified greatly). So, two decades after the graphical user interface became ubiquitous, why are there no billion dollar data visualization companies?

I think it is simply that there are not enough people out there whose jobs demand advanced data analysis tools. I have argued elsewhere that the vast majority of business users have no need whatever of ad hoc analytical BI tools. One can debate the exact proportion of business users who need something more than a report. I reckon perhaps 5% based on my experience on data warehouse projects, while I have seen an estimate of 15% from Forrester, so let's say 10% isn't far off. Then within this population, who want to at least see some analysis of data, what proportion are serious data analysts and what proportion would find a bar chart more than adequate? I don't have any hard data here, but let's go for 10% again as a guess. In such a case then only 1% of potential BI users actually need a sophisticated data visualization or statistical toolset. Indeed this figure may not be so far off, since in order to make serious use of such tools, some background in statistics is probably important, and relatively few people have this.

This would mean that, in a large organization of 10,000 people, there is actually only a market of 100 people for advanced data visualization or statistical tools (data mining tools being one example). Assuming that it is rare to be able to charge more than a few hundred dollars for a PC tool, then even at a thousand dollars a piece our mythical company would only be worth a maximum of USD 100k to a vendor, and that assumes that the vendor could track down every one of these advanced data users, and that every one of them buys the software, an unlikely situation.

If this reasoning stacks up, then while there will continue to be fascinating niche technologies serving aspects of data visualization, we are unlikely ever to see a true mass market for such tools. A picture may tell a thousand words, but there may not be many people in businesses needing to actually produce such pictures.

A good strategy for data visualization vendors would be to avoid trying to be mass market tools and find industry niches where clever graphics really do add a lot of value. For example a tool which improved a stock market trader's view of the market would presumably be worth a lot more than a thousand dollars. The same would be true of a geophysicist working for an oil company. A good example of a targeted strategy of this kind can be seen in Spotfire, who have carved out a strong niche primarily on the life sciences industry, and seem to be thriving on it.

MDM gets the blues, or at least the blue

2005-12-12T02:34:00.000-08:00

In case anyone has any doubt about the reality of the master data management (MDM) market, it is worth noting that IBM has now set up a significant business unit dedicated to MDM, with 1,000 staff, in its vast software group division. This follows a series of acquisitions (Ascential for data movement technology, Trigo for product management, DWL for customer information synchronisation, SRD for identity management).

So far this set of tools still has gaps, though. As I noted elsewhere, BP has 350 different types of master data being managed by KALIDO MDM, and customer and product are just two of these 350 categories. It would seem excessive to expect a customer to buy 348 further technologies once they have bought their CDI and PIM products, so it seems clear to me that a more generic approach to MDM is required than tackling each specific type of data with a different technology. Moreover IBM still lacks a technology to deal with the "analytic" part of MDM, something which can help manage the semantic integration of the various business models which large corporations have, and which contribute heavily to the diversity of master data. Buying piecemeal technologies that tackles specific data-types, however clever they may be (and DWL and Trigo both had excellent reputations) is not going to solve the enterprise-wide problems that large companies face in managing their master data. It seems to me that, while incomplete, IBM has a better grasp of the issues than Oracle or SAP, which has already ditched its first MDM offering, while the SAP MDME solution, based on technology acquired from A2i, has had poor initial feedback from early prospects. "Even worse than the original SAP MDM" is one customer assessment which cannot be encouraging to the German giant. Moreover IBM, with its deliberate abstinence from application software, has the advantage of not being perceived as quite as aggressive as SAP or Oracle. One CIO memorably described IBM as the "beige of the IT industry" meaning that it was neutral and inoffensive compared to many others.

I see there being an evolution in most master data initiatives, the first stage being the analysis of the problem, classifying the various different business definitions that exist for master data in the enterprise; this goes well beyond customer and product e.g. "Asset", "brand", "person", "location" are all important types of master data. The next stage is to document the existing processes for managing change within these categories (mostly manual, involving email) and the governance and authority levels involved e.g. not everyone can authorize the creation of a new brand. This will then require either automation of this workflow, or some process redesign (probably both). Finally the new workflow will need to be linked up to some form of messaging infrastructure e.g. EAI technology, so that changes to the master data can be physically propagated throughout the various operational systems in the corporation. At present there are various technologies around to tackle elements of the problem, but they are far from joined up.

The MDM market is in a nascent state, with people still coming to terms with the issues and trying to piece together where the technology offerings fit. The business problems which it addresses are very real in terms of operational efficiency, so there should be plenty of value there for companies that have compelling offerings. IBM has realized this earlier than most.

Oracle' struggles to buy growth

2005-12-09T07:53:00.000-08:00

It is interesting that Oracle's buying binge in the last couple of years has not enlivened its share price. As pointed out in a recent Forbes article, Oracle's price/earnings ratio (a simple measure of how strongly the market views a stock) is now the lowest since 1990. This is despite Oracle's superb operating margin of 31% (up there with Microsoft and better than SAP's also excellent 27%). The problem is clearly not profitability but perceived room for growth. Even its core database software license sales were flat in the last quarter. The database business is still very much the jewel in Oracle's crown, contributing a disproportionate proportion of Oracle's profits.

The strategic issues are that the database market is somewhat saturated, with Microsoft chipping away at Oracle's market share with ever its more functional SQL Server, and IBM continuing to revamp DB2. Although insignificant now, the open source mySQL at the least creates pricing pressure, as does SQL Server. Oracle's applications business has been thoroughly outclassed by SAP. And the Peoplesoft acquisition was very important in order to inject both extra market share and superior technology. Oracle's grab of retail software vendor Retek from the clutches of SAP was an astute, albeit defensive, move. Over the years Oracle has meandered into a range of other technologies, either by development or acquisition, but rarely with much success e.g. its MPP offerings, its lackluster business intelligence products, etc. This is less surprising, as large software companies usually struggle to diversify, especially the further they move from the area that originally made them successful.

Certainly the wielding of the cheque book has bought Oracle some market share, but it also brings with it a major technology challenge in trying to integrate the various technology platforms of the companies it has acquired in with its already sprawling suite of software. Oracle has long been known for its superb marketing and aggressive sales force, but its pushy tactics have alienated a lot of customers, which in the end must damage it. When household name companies start to contemplate the massive task of moving away from Oracle to SQL Server, not on technical grounds basically because they feel themselves commercially abused, then this indicates a depth of animosity amongst customers which eventually will come home to roost.

Oracle did a fine job of winning the premier slot on the DBMS business, outpacing often superior technology (such as Ingres) through relentlessly effective marketing and sales execution. It remains to be seen whether the trail of upset customers they left along the way will continue to haunt them as they try and bring back growth. Perhaps Larry Ellison should ask the Pythia, the priestesses of Apollo at the original Oracle of Delphi, for some guidance. Their predictions were usually cryptic, but further predictions could always be bought with more gold if the initial ones didn't meet expectations. Sounds a bit like Oracle's application strategy: if you can't build a set of applications that someone wants, then just buy vendors who have. Some things never change.

2005-12-07T04:29:00.000-08:00

A new report from Butler Group finds that "less than 8% of the IT budget is actually spent on initiatives that bring value to the enterprise". This is not an encouraging number for those who believe that CIOs have their finger on the pulse of the business, but merely confirms other studies e.g. one from A.T. Kearney, which found that business executives perceive their IT departments to be out of touch with the business. Yet the same study found that "70% of respondents see IT innovation as important or critical to their company's success". I'm afraid that my own experience would side with the cynics. There are certainly some talented and hard-working people in corporate IT departments, but in general the management of IT is woefully out of synch with the needs and desires of the operational business executives. CIOs continually act as gatekeepers rather than enablers, often seeing their role as one of cost-cutting and standard-bearers in the fight to reduce the number of vendors, preferably to a set that can be represented on a single Powerpoint slide.

Unfortunately this is generally not what business executives need. Since exciting new business applications are unlikely to come from the big vendors that wine and dine the CIOs, the business will look to IT to bring them new ideas that technology can enable geneuine value, and help them sift out the genuinely interesting from the slideware and the charlatans. I had a conversation with a business executive in a large company this weekend, who explained that his IT department had recently made a software product from a small vendor "non strategic" in favor of a less functional one from SAP, despite some objections from him. A year later this same IT team has had to admit that his new global retail management information system cannot be delivered by the SAP technology. Just what sort of business value is this IT department delivering to its customers? People working in IT at that company can hardly be surprised when one-third of its IT jobs were recently shifted to India. If IT departments do not bring perceived value, then they are just a commodity cost as far as executives are concerned, in which case they can expect to be treated as such.

The management of IT badly needs to speak the language of business and to become more aligned to the priorities of its customers, rather than its own agendas and technology religious wars, most of which leave business people cold and are perceived as irrelevant. The status quo will see a lot more IT jobs headed towards Bangalore, and not just the low-level programming ones.

Weaving knots rather than nets

2005-12-06T05:28:00.000-08:00

SAP's Netweaver initiative is an astute move to try and define and so to control a standard for applications in large enterprises. It will certainly present advantages to existing committed SAP customers, who will be able to interact more easily with some non SAP applications. However there are several drawbacks. Firstly, NetWeaver's integration is skin-deep. If you are on an SAP screen you can potentially branch out to a non-SAP routine e.g. a specialist payroll calculation routine. However anything that requires the integration of data between SAP and a non-SAP system is no better off than they are today i.e. customers are still into coding. Since the higher value levels of integration will usually involve not just portal-like "put in on the same screen" integration but actually dealing with business meaning of data, this will limit the use in reality. For example Netweaver would allow you to drop out of an SAP process into a supplier system, but does not help you deal with the issue that your set of product codes, or your general ledger structure, are different from that of your suppliers. For meaningful integration you need to resolve the business meaning or semantics.

What Netweaver certainly does is to declare war on several industry players who were previously either neutral to SAP or indeed active partners. IBM is the most obvious example. IBM has a massive services arm that does huge business implementing SAP, and IBM has decided to stay out of the applications business, so relations between the firms were good. Netweaver directly attacks IBM's core websphere middleware, and so now IBM's software group is in direct competition with SAP. The same would go for the EAI and ETL vendors (e.g. Tibco) and Microsoft, who have their own middleware stack. Oracle competes here too, but of course Oracle was already the most direct competitor to SAP. SAP may not care about the EAI world, but IBM especially is a big target to take on. The reason that SAP is prepared to take this risk is that the reward is so great: Microsoft showed the power that can be exerted by controlling the critical standard, in their case Windows.

Most large corporations have multiple middleware stacks within their organization (SAP, but also IBM Websphere, Microsoft and probably Oracle as well), so the key issue for them is how to deal in a neutral way across these, rather than how to rip all but one of these out. This is where the Netweaver strategy may ultimately fail, since the sheer cost of ripping out an existing well-established software infrastructure is gigantic, and that assumes that corporations donÃ’t care about the lock-in that would give SAP. Some won't care about this, but many will. However the practical problem is the sheer scale of core infrastructure that would have to ripped out, and yet without data and business semantic integration, the benefits of doing such a thing would be very limited. Oracle and SAP are both giants locked in a battle to gain a larger and larger footprint in the enterprise, yet both are too well-entrenched to ultimately destroy the other. SAP has no database, for example, an Achilles heel for it, and SAP grieves every time they win an application account from Oracle and then see the customer deploy SAP on the Oracle platform. Oracle and SAP are always likely to optimize their applications for their own proprietary middleware, since their agenda is to expand their footprint inside large corporations, yet by doing so they rule themselves out of being an idealapplicationsn-neutral layer that can genuinely help their customers.

For most enterprises, Netweaver looks superficially attractive but does not solve the core integration problem, that of resolving the differences in business meaning embedded in their many, many core transaction systems. Netweaver's widespread deployment is certain, but its skin-deep level of integration will deliver only limited benefits to customers, yet at considerable cost. Perhaps customers should check back on the investment cases they made a few years ago before they went down the ERP route - did the benefits promised then actually materialize? The costs certainly did (billions of dollars each for global corporations), but I haven't seen too many of their IT departments shrinking away to nothing because all their integration problems were solved.