Today's request becomes tomorrow's standard report

Information is constantly changing, and this is self-generating!

The more people know, the more they want to know.
The only thing that answers produce is more questions.

Pick your platitude they are all true. The general rule that I use is that 3 years from now 2/3 of the requirements for reporting are currently unknown.

If your organization is not geared to this constant state of change then you will be falling behind, if only from your competition!

What is behind all this.

Within any organization there are those that always want something different, either a new analysis, table or just the same data presented in a different way. It is not uncommon for such requests to go unanswered, simply because the level of effort required to respond is simply beyond the resources of the responsible department. It could be either available personnel, expertise or just no money to expand the current system.

But without this constant reminder that the current reporting systems are inadequate (if not now, then in the near future), the overall system and the organization that it supports will stagnate. I don't know of any industry where doing the same thing, year after year, promotes efficiency or expansion. Do you?

Given that we always need to expand and grow, we always need new information. This information must come from new analysis on existing data. If your organization suffers from DRIP, then there is no new information coming and failure, in some form, is in the future.

An organizations health, therefore, depends on the abilities of the SME's to produce this new information. They are a critical piece of the expansion plan. But in previous posts we have seen how their abilities are stifled and limited by others.

Enter the SME!

While it may be that the IM department holds (and controls) the data it is the subject matter expert, the SME, that understands the data.

This is the person in the organization that turns data into information. If there is someone in your organization that is data rich, information rich, it is the SME.

As highlighted in the first few blogs, the control and handling of your corporate data, is solidly the purview of your IM department. They hold the access cards, they hold the software application cards, they are the home of the DBA that looks after the data.

Unfortunately they do not understand the data, that is the world of the SME.

So what has happened to the SME in the past few years. In many cases they are simply stifled in their ability. But a good SME, with a keen interest in the data and understanding its contents will work around these obstacles in many ingenius ways.

For example, standard generated reports will find additional uses in post-processing venues. Cutting and pasting to a spreadsheet is just one of many ways of extracting data from the controlling application. In this new environment the SME has absolute control and is free to manipulate the data to their heart's content.

In more imaginative cases, access to the raw data can be obtained. This can be a bit more difficult but can usually be managed with a sympathetic IM department, and limiting the access to read-only. (After all that is usually all the SME needs. They just want to see and play with the data, not change anything!)

Except in the most stifling of organizations, the transition from information poor to information rich is probably occurring in your organization. Where there is a need for information, there is someone capable of extracting that information from the data.

In most cases this need for information is beyond the scope of standard reports and requires something new. Today's new information request is tomorrows information requirement.

And this leads to the next post.

Security Enters the Picture

Databases are getting bigger, the applications that work with them are getting more complex, and the people doing the bulk of the data entry are getting less sophisticated. This is a recipe for a security nightmare, too many people accessing too much data with not enough control and clearance.

The result is a plethora of rules and controls for access and the watch phrase becomes: If you don't need to see it, you can't!

The security aspects of the database control become more and more important, different levels of access are granted to different operational requirements. After scandals such as Enron, the financial controls guidelines, protocols and regulations become more stringent. The days of a single person having access to the data, either from the front door, or through a back door, are at an end.

Security of the data and the database becomes a heavy trump card in the development and operation of the modern database and data warehouse.

Between the security and the complexity of the modern database, very few people get to the see the raw data. What anyone does get to see has been preprocessed and pre-packaged into bite-size packets. However, these packets are still only data, they are not information.

In order to turn it into information one needs interpretation. This interpretation can only come from the SME, Subject Matter Experts. These are the few that really understand what is in the data.

Enter the DBA

Organizations quickly discovered that cheaper data entry went directly to the bottom-line, reducing overhead. Systems got bigger, databases got larger and the need to make these databases more efficient in processing became a new requirement.

Enter the world of the DBA, database administrator. This is the person that understands the database structure, where everything is, how it all fits together, and how to make the most efficient structure for the various requirements of the applications.

This person understands the structure, the relationships, the tables in the complex world of database design.

But....

They don't understand the data. What does it mean. Consequently they possess no Information.

But because they control the database structure, they become the guardians of the data itself. This leads to access control. Who gets to see what.

The best way to control the data access is through the applications that access it. Limit the applications that can access the data, and limit the access within the application and you effectively control the data, all under the auspices of efficiency and security.

Yes, here is another factor in DRIP, security. We don't want the wrong people accessing the data, for a whole host of reasons.

So....

How did we get where we are today?

It wasn't always a world of data rich, information poor, it was once a world of data poor, information poor. But we have climbed out of that mire.

What happened was an evolution of systems. The advent of powerful computers and cheap programming allowed for the development of more advanced applications.

This allowed for embedding of logic and business rules into the program, and away from more expensive human interactions. It has made possible the cheap operation of large scale business operations with a minimum of experts and a larger number of "clerks".

So the cheaper clerks were kept busy entering all the data that fed these hungry systems, and it became necessary to report on the all this myriad of activity.

Thus began the slippery slide to the world of information paucity.

What is it!

So I decided to start a blog!

It took a while, but I discovered my passion the other day when someone mentioned that their operation was like many others - "Data Rich and Information Poor".

I realized that this is probably true for most large organizations, though in my particular job it was not true, and I decided to determine why. Maybe it was because I had conquered the problem, or maybe not. But regardless I realized I had a lot to offer in this area and decided to share it with everyone else, or at least those that could find me!

So how am I going to attack this issue, what will I have to say about it, how will this all evolve?

That is the beauty of a blog, one never knows. While I may have a lot to say on this subject, my ideas may get sidetracked into alternate lines of thought, all the better.

Let us begin with an attempt at defining what we mean by "Data Rich and Information Poor", DRIP.

We begin with an example, one that I hope will develop throughout this blog.

The reports from your organizations systems (ERP, BI, call it what you will), show that the West has more sales than the East. There are also less sales people in the West. That is what the data shows, but what does it mean.

There is no information here. The reports are lacking, we must find more.

Ahh, begins the process of over-systemization and the death-knell for the information age!