Enterprise Search – How Far Until We Hit Warp Speed?

I’m currently working with a colleague on doing a whole bunch of stakeholder consultations for an intranet redesign and a common thread has emerged.

Enterprise search sucks.

Hard.

I’ve done consultations for intranet overhauls a for two clients and in both cases, this has been the one thing on the lips of the people I’ve talked to. In fact, at iStudio, when we ask “what is on your wish list for the new intranet?” some people can barely wait for us to finish before telling us how bad the internal search tool is compared to, say, Google.

Which goes to show just how high Google has set the bar in terms of search.

Background

Businesses that need large intranets tend to operate in a data driven environment and are therefore well known as uber content producers. So, while the value of the intranet grows along with its knowledge bank, the harder it becomes to find what you need. In 2005, research firm Gartner estimated that “80 per cent of an organization’s information is unstructured data (such as Word documents, presentations and rich media files).”

Therefore, it is vital for an intranet’s search engine to accurately and consistently locate information – including archived and unstructured data. If there is a huge amount of content, all of which has an associated cost, and no sophisticated way to find that content, organizations are essentially losing or significantly degrading their content investment.

Along with the frustration of not being able to find what you know is saved on the intranet, other costs associated with poor quality search include duplication of efforts around documents (such as creating new documents after not finding existing ones), time spent searching for documents/pages and the need to rectify mistakes due to the wrong SOP being used.

Working with one client, we got our own data saying that an incredible 98 per cent indicated they could only find information “most of the time” or “sometimes.”

Why So Bad?

Why is internal search so sucky? Certainly there are some really good, high quality search products out there – former client BEA’s AquaLogic Interaction Search product springs to mind – and the motivation to make a killer enterprise strength search product is well documented: It has been reported that Ford Motor Company yielded $4m pa (U.S.) in productivity benefits simply through employees finding information more easily – showing the tangible benefits of high quality search. (sorry, no linky for this one – but it is a fact. I even put it in a client report so it must be true.)

But the main reason internal search engines are so bad is, ironically, the lack of data to work with.

Think about Google – the search Behemoth (which competes with current and former clients of both iStudio and FH) currently indexes an astounding 700 trillion pages (according to my colleague Neil) all of which are subject to the complicated PageRank algorithm to give you precisely the right results for your search.

Even the most complex and convoluted of intranets that I’ve seen won’t have any where near that sample size to work with.

What’s more, the most important thing about data in the “real” world is that it’s importance is defined, in part, by the amount of links into that page. Within the enterprise, it seems, that sort of interlinking just doesn’t happen.

So when I search on my clients’ intranets, I get a list of all the pages which contain all or part of my search query – there’s no authority algorithm to filter my results for me. As a result, the bigger the intranet, the more useless the search function, until the intranet hits a critical mass (which is never) or starts to interlink more heavily (which is unlikely).

A Solution?

Given Google has set the search bar so high, it’s on natural to look to the company for a solution. Well, perhaps not directly to Google, but to the company whose patent Google infringed upon on its way to making a brazilian dollars – Overture.

Here’s my solution. It’s a little more work than slapping down a few hundred grand on a search system that likely won’t work but less work than actually building on that does.

Each department that has a presence on the intranet must define the top X amount of documents within it’s content area and then assign key words to those documents. So far you’ve just cleaned up the meta data.

Now, give each department a set amount of “credits” and have them assign a budget of credits to each document. If you have 10 documents and 100 credits you can either give each document 10 credits or you can assign more resources to one particular document – this resource allocation will imply the authority or importance for each document.

Now, when you conduct a search, you can get not only the “organic” search results, but you also see the “sponsored” search results, based on the credits, alongside them.

What do you think? Will this fly? [insert Star Trek joke/pun here]

6 Responses to Enterprise Search – How Far Until We Hit Warp Speed?

  1. James says:

    I think you’ve nailed the problem, but, alas, not the solution. Asking anyone to “rank” their documents would bog down immediately. I work at a large professional services firm with just this problem and getting any input at all on web or intranet content is like pulling teeth.

    By the way, how do you like Liverpool’s chances next season? Do they need David Villa?

  2. Ed Lee says:

    as always, the problem is governance. if the content owners saw the value in having their content easily accessible to their internal clients, i suspect it would be useful. after all, you’re asking for a small amount of time from the content owner each month in return for not having to give up the hours of time that would be spent in person-to-person requests.

    think of the great productivity savings!

    plus, it would also allow newer content to float to the top: if you’re uploading a new version of a document, you’d automatically want to assign importance to it….wouldn’t you?

    as a sidebar, this could also be a rationale for internal blogs within the enterprise as a way of increasing the amount of linking between departments…
    ————–
    edit: i’m not sure if we do need david villa – he’d detract from our shining star right now (torres) and would likely force rafa into a 4-4-2 formation which i don’t think plays to our strengths. i’d like to see a 4-3-3, personally.

  3. Armand says:

    …or just use Google Enterprise Search? (disclaimer: a client of mine)

    Never think the metadata ranking systems work that welll. You have to know how people think in order to tag stuff effectively… and who knows how people think to search better, Joe in Accounting or someone like Google?

  4. mira says:

    great assessment of the situation. the importance of metadata makes me twitchy.

    having lead a team of forms nerds and webmasters for a spell, we were militant about document metadata and keywords. our own internal shared drive searches rocked but the results of our intranet search engine sucked. i waited with baited breath until an application caught up with us. especially because we did user testing for the keywords used by this particular audience but alas, nothing ever materialized.

    all these comments are salient, epecially ed’s on accessibility and customer service. and why not use existing web metrics to determine document importance? that way human bias is removed from the equation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: