Reflecting on Data, Information, and Insights

It has been a while, okay a long time, since I took the time to write a new blog post. There are lots of topics I intend to write about ranging from recursion and decorators to how to configure asynchronous processes using celery and redis. However, I decided on data strategy because it is a topic I have to think about on a near daily basis.

My goal is not to outline the principles of developing an effective data strategy. That would take way too much time and there are already a number of excellent articles on the topic. I recommend "What's Your Data Strategy?" which is an excellent article in the Harvard Business Review on the topic.

Data strategy has been on my mind a lot lately. It's impossible not to think about considering I'm in the midst of a major SAP implementation where the definition and maintenance of master data is critical. However, the effective management of data has been a critical requirement for professional success in both my military and corporate careers. I spent 30 years as an artillery officer in the U.S. Army and California Army National Guard with the first 10 years in the active component and the last 20 years in the Guard. My service in the Guard allowed me to pursue a joint and parallel career in the corporate world. People often comment how incongruous it seems for a combat arms officer to also be an IT professional, but I always point out that the two careers have proven mutually beneficial. Below are a couple of vignettes from my military and corporate experience that have relevance to understanding and managing data.

Army

I'm sure many of you have heard the term "fog of war" which is a reference to the confused nature of warfare where military commanders have to deal with limited data and make decisions under timelines that are often extremely compressed. I have commanded units at the company, battalion, and brigade levels and I have also served on staffs at the battalion, brigade, division, and theater (what we refer to as "echelons above reality") so I understand the criticality of data. Military staffs at different levels operate in what is often called a tactical operations center (TOC) although it really depends on the organizational level (e.g. D-MAIN, CAOC, etc.). Regardless, the staff functions are largely the same and that is to provide the Commander with timely information to make informed decisions. A vast amount of data flow into operations centers from units and other organizations operating with various relationships (subordinate, peer, superior, NGOs, DOD, etc.). Commanders establish what are known as Commander's Critical Information Requirements (CCIR). These criteria are a relatively short list of data the Commander feels is critical to informed decision making. It is the responsibility of staff officers, NCOs, and operators to understand the CCIR and look for these data nuggets of gold...failure to do so can result in mission failure. In 2013 I was serving as the Chief of Fires for the 40th Infantry Division headquartered in Los Alamitos, California (and currently serving in Afghanistan). In my capacity as the Chief of Fires I was responsible for planning and executing all lethal and non-lethal fires within the division's area of responsibility including surface to surface and air to surface (i.e. aircraft) fires. We were undergoing a major training exercise at Fort Leavenworth, KS which the Army conducts for division-level headquarters roughly every 4 years. There are two major objectives of these warfighter exercises; 1) Ensure the evaluated headquarters is capable of performing its war time mission and 2) learning and sharing lessons learned throughout the Army.

With this background I can move on to the anecdote I have in mind...and if you're still reading thank you for your patience! One of the Division Commander's CCIR criterion was to locate the enemy's operational reserve which in this case was a tank brigade. The operational reserve was a key focus of our collection and intelligence efforts. The second day of the "battle" I went to the Analysis Control Element (ACE) to talk with the G2 targeting officer and ACE chief to get the latest intelligence picture and estimate. I asked the targeting officer where she thought the enemy's reserve and she pointed to the computer screen that had a widely dispersed array of icons representing squads and platoons which weren't even armored units. This immediately led me to the (accurate) conclusion the targeting officer had no idea how to properly develop data and transform it into information. In this case the icons are analogous to individual data elements, but I needed information (i.e. the location of the enemy's operational reserve). Even better would have been intelligence (also known as "insights" in the civilian world) informing me, and hence the Commander, on the enemy's location, strength, disposition, and intent. Needless to say it was a long training exercise...although ultimately we did locate the force and "destroy" it using air interdiction assets.

Corporate Manufacturing Environment

Several years ago I had the opportunity to work with a team tasked with implementing a dashboard which was intended to show how a production line was performing (current work order, run speed, quality status, current production count, next planned work order, etc.). The team had a very nice design layout of what the dashboard was going to look like. It quickly became apparent that the real goal was to implement a manufacturing execution system (MES). I asked to look at the data model so I could understand how the source data was going to be collected from the production line's programmable logic controllers (PLCs) and the reactions ranged from blank expressions to comments that they hadn't "got that far yet." This is a classic example of putting the proverbial cart before the horse. Not having a data model was bad enough but the problem was compounded by the fact the team did not have the right key performance indicators or metrics. A classic measure of performance for a production line is OEE which stands for overall equipment efficiencies. The OEE measure takes into account quality, availability, and performance. The team felt that operators wouldn't care about OEE because OEE is "just a number" that doesn't mean anything. Needless to say, the project was a failure, but it did plant the seeds for a successful reimplementation the following year. This time the team gathered requirements, defined appropriate KPIs/metrics, developed an appropriate data model, and then moved on to the collection and aggregation of data. The new system was such a success that the new MES system was implemented throughout the plant so the Business could move from gathering information to gaining insights to operate much more effectively.

These two vignettes are all about data and in today's world we are awash with data everywhere. Many organizations struggle with how to manage and leverage their data. Many companies that successfully harness data have been wildly successful and in some cases have killed competition and created huge competitive advantages.

Below is a chart I developed to demonstrate the relationship between data, information, and insights:

Data

We have already discussed data in a fair amount of depth. Data is used to represent or describe objects but by itself it has little meaning. That's not to say data isn't important; quite the opposite. An organization's master data is critical to sustained success regardless of industry, and it is imperative companies have appropriate processes and protections in place to manage master data. Positions such as Chief Data Officer, data steward, data architect, and others are becoming increasingly common throughout many industries. One caveat I feel is important to keep in mind is that the pendulum can swing too far to one side. An organization's master data has real value and therefore must be managed and protected appropriately, but it is also important that users have the necessary access to data in order to perform their responsibilities. Such "defensive" data strategies require the balancing of these competing interests and the right policies, procedures, and IT governance controls need to be in place.

Information

Information represents data that has been summarized, has context, and can be used to make decisions. I frequently hear customers ask for reports with the expectation that the required data will simply be available. Unfortunately, that is generally not the case and since hope is not a method it's a good idea to think of data requirements ahead. One of the first signficant efforts we did during the SAP implementation was to define the key performance indicators (KPIs) to ensure the data conditions were set during the design phase. This gives us a high degree of confidence the necessary data elements will be available once the system is implemented. The key point here is data used for reporting KPIs should be collected as employees perform routine business processes. Trying to collect and "massage" data usually results in KPIs not being reported because it becomes too difficult or the reporting becomes subjective and manipulated to meet management's expectations.

Insights

Organizations that manage data effectively and develop information should justifiably take pride in doing so. However, the true power of harnessed data is the "weaponization" of data which can be used to gain and sustain competitive advantages. Again, competitive advantages apply across any industry whether it be the military or private sector. One of the American military's competitive advantages over its state and non-state competitors is its ability to collect enormous amounts of data through human intelligence, signals intelligence, and other sources and develop intelligence that can be exploited. The fusion of data and subsequent exploitation is what ultimately led the United States to locate and kill Osama bin Laden. Similarly, private sector companies can gain and sustain competitive advantages within their respective industries. Companies like Netflix, Amazon, Facebook, Google, and others are great examples of companies that have leading edge offensive data strategies.