The Ups and Downs in Business Intelligence
Business Intelligence is a technology that produces actionable information to decision makers, based on the data residing in underlying transactional systems in a company.
A decade ago, the underlying OLTP (online transactional processing) systems were many, including custom or bespoke applications. Hence, development of BI or OLAP (online analytical processing) solutions were time consuming. The underlying data model was required to be studied, data warehouse model needed to be created, hardware had to be procured and OLAP cubes based on the data model to be created, data to be cleaned, and finally the BI application is required to be developed, which will work on the data aggregated in OLAP cubes and produce meaningful reports for decision makers.
However with passage of time, while many past complexities related to creating BI solution got reduced, some additional challenges have cropped up. The purpose of this article is to explore areas which have simplified with time, and areas which have become more complex.
Areas where things have got simplified
a) BI solution is not a standalone application, but is dependent on underlying transactional systems. So when corporates started standardizing on ERP applications, BI projects got a big boost. As underlying data got standardized, and corporates adopted product ERPs such as SAP, Oracle etc., time and effort required to create data model got reduced. The quality of underlying data improved, BI vendors came with standard ERP connectors’ and reports, and hence BI project became feasible to many customers unlike before.
b) Software runs on hardware and the advent of hardware technology, gave rise to new architecture, which made BI projects more affordable to customers. True to Intel cofounder Gordon Moore’s prediction, the computing power of chips is doubling every two years, RAM cost is coming down and speed is going up exponentially. In memory architecture is today a reality. To give some examples, SAP has developed HANA database completely on in memory technology. Qlik view was developed on in memory technology years back and they realized the power of this architecture before others. Now most of the vendors have solutions around in memory technology. By having the data on faster RAM, as against slower hard disk, BI solutions became faster.
If the underlying ERP runs on in memory database (such as HANA), real time analytics is possible, as against the traditional Day -1 analysis.
c) Number of BI developers, data scientists and architects increased with time. Many who started career in software development migrated and adopted BI as career. Availability of skillsets in any technology in IT is very important, so that solutions can not only be developed well but also supported properly.
d) Allocation of budget on BI projects also helped IT. Top management at board level also became aware that it’s the business intelligence system that provides them insights, for taking meaningful decisions.
a) Analysis of data from new sources are required. E.g IOT is throwing up huge amount of data, unstructured data like images and files are also required to be analyzed. Advancement of artificial intelligence and imaging technologies will require new ways of analyzing data in a fast and secured manner.
b) Big data, non SQL databases are being used to address these new age requirements. However, being new technology, skill sets in these areas are still limited.
What can be the BI strategy for a corporate today?
a) Instead of putting the cart before the horse, it’s always good to sit with the management and set the priorities which will provide maximum business benefits.
b) The solution identification will depend on various factors like complexity of landscape (single ERP Vs. multiple OLTP), level of analysis needed (Forecasting Vs. Reports) etc.
c) Putting a Business IT team is a key to any successful project. A good business analyst is needed and the vendor’s past experience in similar industry definitely adds value.
d) It’s always good to focus on one department like sales, or finance and then extend BI to other areas in the company.
Where BI projects go wrong
a) BI project goes wrong where there is an expectation mismatch between what management expects from the system and what’s feasible. I am aware of a case where a HOD expected BI to work partly as transactional system to reduce some data conversion work and the project failed, as that’s not what a BI system is supposed to do.
b) Wrong architecture and modelling can create problem from leveraging system. It is important that the data cloud is correctly designed and provisioned, so that future needs can be easily fulfilled. Wrong design can create huge problems in future, in spite of early success.
With advancement of technology and greater usage of stand ERP packages, BI systems can be developed easily these days. However it’s always important to prioritize and work closely with business to ensure maximum success in BI projects. This is one project that can provide maximum visibility to IT in any organization. However, one needs to be aware of the pitfalls and plan properly and select a good technology platform and vendor so that 100% success is ensured.