If a picture tells a thousand words, do you need a thousand search terms to find an image? Thankfully, no.

But while Haystack WS technology is prominent – we’d argue dominant – in some areas of media search (particularly video and audio), the approaches to image search are far more diverse. Why?

This starts with the question of how image search works. Basically, there are five kinds of image search engines. While Haystack WS technology can be integrated with all, it is best-suited to platform-level search (on platforms like Pinterest) where user behavior information is typically more rich.

The five basic species are: categorical/tagcloud, traditional machine vision, machine learning, deep learning, and palette. Some, like Google Images, are a mixture of approaches – machine learning and deep learning are used to generate categories and tags, which are then fed back into a proprietary sort (this generates the hierarchy of results seen by the Google Images user).

Historically, categorical search was the strongest, but required (until recently) human tagging. This meant either crowdsourcing enormous amounts of human labor (“Hey, that’s a cat! Let’s tag it with the word #cat!”) or buying archives that were already categorized in some way (“This whole folder is pictures of cats!”).
About fifteen years ago, machine vision got to the point where it could distinguish in a human-like way between pictures of, for instance, cats and dogs (even embracing positive and negative identification frameworks: “This is a dog and is also not a cat!”). Traditional machine vision, which uses categorical, human-crafted buckets to categorize images, continues to make substantial progress in a wide variety of applications from geology to medical imaging to industrial lithography.

Now, machine vision and deep learning dominate the image search space. This is partly because while training data (from labeling startups like CrowdFlower and crowdsourcing platforms like Mechanical Turk) produced by humans is valuable, its value can be multiplied with the help of machine learning and automation systems. Once a (well-designed) engine has seen 10,000 pictures of cats (and many pictures of “not cats”), it can begin to locate and categorize new pictures it has never seen before. It can then return those new pictures to users who search for “cats” without any human ever having tagged or explicitly identified the result image as containing a cat.

The newest area, and one that shows promise, is palette search. Palette search examines patterns of color, hue values, and even identifies proprietary colors. While this is primarily a toolset used by graphic designers and advertising agencies, it is a technology seeing increasing use in areas like litigation support (is that Coca-Cola red?) and advances in machine learning mean applications are being developed to support areas like interior design (what goes with this color of sofa?) and even fashion design (what are popular colors for dresses like this one among customers?). Palette is interesting in that it’s an area where very quickly the technology progressed from diagnostic to decision-making.

At Haystack, we believe images are an important part of the online ecosystem and that uniting users with image content that interests them is a central task in search.

Are you interested in licensing our technology for better-organizing your image platform or image library? Let’s talk.