6 Minute Read

As you evaluate competitive intelligence tools, you will likely come across two major categories of systems: taxonomy-based systems and systems that use natural language processing for automated tagging.

Finding a system that will make the best use of your team’s time and resources, as well as help you gain the best results is important. So, we want to help you understand the differences between the two types of systems and make the right choice for your needs.

-- Article Continues Below --

New call-to-action

Let us help you find the tool that best fits your needs.

Traditional Taxonomies

Most competitive intelligence tools use keywords to tag content that is stored within the system. Some of these platforms have automatic tagging capabilities that will automatically apply a keyword tag to a piece of content whenever that keyword is identified within that piece of content. 

Often, these keyword structures may have multiple, hierarchical layers. For instance, in many platforms, you can configure the tool to automatically tag a piece of content that contains the word “China” with the tags “China” and “Asia”. 

These hierarchical keyword structures are often referred to as taxonomies. Using a taxonomy is a time-tested method to structure information for easy retrieval. In fact, librarians have been using a taxonomy known as the Dewey Decimal system for over 140 years.

Where traditional taxonomy-based systems fail is when we don’t know every detail about every piece of information available within the system. Take the library example above; in that case we know every single book that needs to be organized, making it easy to provide a comprehensive system.

But what happens when we live in a world with ever-changing and new information inputs, like your competitive marketspace. If there is information you don’t know exists, but might be very useful to you, a traditional taxonomy will fail. There will be no way to categorize and surface this information because we haven’t pre-built a category for it.

Luckily, there is another way!

Natural Language Processing

The field of Natural Language Processing (NLP), a subset of machine-learning (ML), has been around for about 50 years, coming out of the combination of the study of linguistics and enhanced computer processing capabilities.

Many newer competitive intelligence tools leverage this new field of study and use a different type of information architecture to automatically organize content. Advanced metadata tagging technologies using AI, like natural language processing and other machine-learning algorithms, can go beyond simply identifying a string of characters in a document. 

They can automatically identify and classify entities that are mentioned within a piece of content, and tag them accordingly, without anyone ever having told the system what it was looking for.

Machine learning algorithms can also be applied to train a computer over time to recognize that certain pieces of content are more valuable than others. Machines can be trained to identify content that is relevant to a particular concept or topic, or to a particular product. 

These types of systems can also be used to filter out irrelevant content automatically, drastically improving the signal to noise ratio in the platform. All of these features benefit the analyst in identifying relationships or connections that may not have been previously seen or understood.

What’s more, with AI capabilities, these systems can recognize new and interesting data that you may have never thought of, or would ever know to ask of a specific keyword-based system.

Benefits of Natural Language Processing

There are quite a few benefits to using competitive intelligence tools that use natural language processing to automatically tag data, rather than traditional taxonomy-based systems.

Context-Based vs. Keyword-Based

NLP-based systems can understand context within a document or piece of information in a way that solely keyword-based systems never could.

For instance, taxonomy-based systems will automatically tag any piece of content that contains the string of characters specified by the user as a “keyword”. When using a traditional keyword-based system, the user tells the computer to look for a specific pattern of characters within a document and, when it finds that pattern, to apply a tag for that keyword. 

If we tell a keyword-based system to automatically tag the keyword “apple”, the system will search each piece of content and if it finds the letters a-p-p-l-e, in that sequence, with no spaces, it will apply the tag “Apple”.

Obvious problems arise in cases like this, where the keyword has more than one potential meaning. We have no way to know whether the content that was tagged with the keyword “Apple” is about Apple the consumer electronics company, or a fruit that is a key ingredient in the all-American dessert pie.

By contrast, a system using advanced NLP could read a document, understand the context in which the word “Apple” was mentioned, and realize that this particular usage was a reference to the company Apple, and apply a tag of “Apple” of the type “Company” to that piece of content.

NLP-based systems help you filter out the noise, allowing your team to spend less time parsing through information to weed out every piece of content about apple (the fruit) to try and narrow to everything about Apple (the company).

Discover New Information and Reduce Missed Opportunities

Another drawback of taxonomies is that the user who is searching for relevant content must know exactly what s/he is looking for. If you’re old enough to remember when library catalogs existed in multi-drawer cabinets with countless index cards in each drawer, you’ll understand this phenomenon. 

If you wanted to find information about the science of social networks, you would consult the index to find that your topic was located in section 302.85. All well and good, but there is nothing else in 302.85 except that specific topic. A taxonomy does not lend itself to discovering relevant information that you didn’t know was relevant when you started your search.

You have to specifically and methodically lay out every single thing the system must look for and then tag appropriately. Which means you have to know everything that might ever be of interest. But, in today’s world, that is rarely possible.

Take the Apple example above. An NLP-based system will, in addition to automatically tagging, also may have identified mentions of the companies “Facebook”, “Google” and “IBM” automatically too within the same document, without anyone having told it to look for those words.

NLP-based competitive intelligence tools take the word “intelligence” to heart. While taxonomy-based systems are merely order-takers, NLP-based systems actually help surface relevant information that you may have never thought to ask for in the first place, but could play a huge role in your competitive strategy.

NLP-based competitive intelligence tools take the word “intelligence” to heart.

Easier to Setup and Manage

Because you don’t have to spend time manually detailing out the entire set of hierarchical keywords your system should look for, NLP-based competitive intelligence tools can be up and running (and providing value) much faster than a taxonomy-based system.

In addition, you have to spend a lot of time managing and maintaining your taxonomy over time. You’ll constantly want to add more items to watch for as you learn of new changes to your marketplace. This can be a very tedious and time consuming process, taking valuable hours that your analysts could be spending on more valuable tasks.

Rely Less on Internal Communication

Another great benefit of an NLP-based system is its ability to break down organizational silos and make information and knowledge available to anyone who needs it.

NLP-Based systems break down organizational silos.

Say for instance you are part of a global organization, with multiple disparate geographic office locations. You may have a member of the Asia Pacific competitive intelligence team upload a document that has information that is of interest to them. 

In a taxonomy-based system, that information would be tagged with the tags that particular team chose to apply to it. However, that document might have information that is relevant to a team over in the US, but because the specific tags and categories they search by weren’t applied to the document, they will never know about it.

Enter NLP-based systems. Anytime a document is added, that document is automatically tagged with all data that might be of interest, and suddenly that document is surfaced to the US-based team as they search the system based on their own keywords.

No longer do teams need to spend agonizing amounts of time building, agreeing upon and managing complex taxonomies across large organizations (or failing to do so, in most cases). Your competitive intelligence tools will do the work for you, and let you spend time analyzing more complete data.

Making the Switch from Traditional Taxonomies to Natural Language Processing

Taxonomies are a great fit when you know everything about every piece of data that will ever live in the system, and you can control every user of that system. Typically for competitive intelligence functions, this simply isn’t the case.

There are too many changes and unknowns in the market to know every possible input. And there are too many users within the system to be able to control every detail or force exactly the same processes and tagging.

However, many organizations are still wary of moving away from taxonomy-based systems. This is primarily because it is a departure from “the way it has always been.” A big change like this can be daunting, and can be difficult to gain support for the decision internally.

Many organizations worry they will lose the level of “customizability” they have in a taxonomy-based system, but this is simply not true. There are ways to have the best of both worlds if you choose the right NLP-based competitive intelligence tools.

You can combine the automation of natural language processing and automated tagging with custom tags and searches. If you have specific terms or company vernacular you want to include, all you have to do is add customized searches for those terms.

In very rare cases, organizations can choose to build their own custom NLP architectures. In most cases this will provide little-to-no additional value over the simple custom searches combined with standard NLP to justify the undertaking.

In all cases, the best way to get comfortable with the switch from a taxonomy-based system to an NLP-based system is to simply pilot an NLP-based tool. When piloting a new system you’ll want to make sure you are also getting training and hands-on help to learn and effectively use all of the new features of the system.

If you are used to a taxonomy-based system, piloting an NLP-based system with no additional training or support won’t truly allow you to learn and understand how you can use NLP features to your advantage.

Final Thoughts

Don’t let fear of the unknown or a mentality of “that’s the way we’ve always done it,” keep your team from realizing the benefits of using competitive intelligence tools that are truly intelligent.

With the switch to an NLP-enabled system, your team will gain back valuable time that they can spend on in-depth analysis, you’ll surface opportunities you may have otherwise missed, and find more value in the data you add to your own system.

New call-to-action