Saturday, July 19, 2008

The Evolution of Analytics



Enterprises today are awash in data. POS systems capture every transaction for a retailer, CRM systems maintain rich profiles of customer preferences and ERP systems maintain transactional information at the most granular levels. Until the mid-nineties, only very large companies, the ones who could afford SAP or Oracle, could afford a reliable data infrastructure. However, even the large companies had to rely on smaller, niche players for analytical and optimization capabilities. The late nineties saw a number of companies starting to offer analytical products around supply chain optimization (i2, Manugistics). Soon, companies followed with Price Optimization, based on advanced scientific algorithms (ProfitLogic, DemandTec, etc.). Analytical evolution had come a long way: with Oracle, SAP, Teradata providing the data infrastructure, reporting vendors (Cognos, MicroStrategy) providing ad-hoc access to information and knowledge, and specialized vendors providing prescriptive modeling and optimization capabilities.

What has changed in the last couple of years is that high level data infrastructure is no longer limited to large companies. The giants like SAP are now catering to small and medium businesses (SMBs) with simpler offerings, and players like NetSuite are now providing extremely robust ERP systems as a hosted service. While data infrastructure is now available to the masses, science-based analytical software is still very expensive to build and distribute, and mostly beyond the SMB's price-range. SMBs are still lagging in the analytical evolution, although the appetite for analytical excellence is definitely palpable.

A combination of factors is changing this picture, giving SMBs access to high quality analytics at a price point that they can afford. The biggest contribution comes from the open-source community, in the form of relational databases (
MySQL, PostGres), web-development frameworks (Ruby on Rails, Django), and optimization programs and solvers (COIN-OR, Scipy). Ruby on Rails makes it possible to go from a business meeting to a working application in a matter of weeks, not months or years. Amazon Web Services provides the physical infrastructure which allows running the complex algorithms on their compute clouds for less than a cup of coffee at Starbucks. As these innovations are driving the costs down, and enhancing the supply of high-quality analytics, the demand for these services are increasing as well. Enterprises large and small are becoming increasingly aware of the power of data and data-driven models, and seeking to gain competitive edge through analytics (Competing on Analytics).

The future centers of analytical excellence may be quite different from what we are used to today. Today's paradigm is niche analytical companies selling to large enterprises. The sales cycle is long and complex, licenses and implementation costs run in six figures, and hosting and maintenance are just as expensive. The future may be ruled by small, nimble shops, providing high-quality, low-cost intelligence over the web, in a simple Do It Yourself (DIY) mode. Are you ready for the brave new world of analytics?

No comments: