Skip to content

How DemandSpot Works

September 28, 2009

This post will tell the story of what happens behind the scenes in DemandSpot – how relevant tweets find their way to our website for you to search. While we’re a software at the core, this is going to be a non-technical explanation. The intended audience is our customers – people who sell stuff rather than program computers.

It all begins with Twitter Search. Our system runs queries for some keywords (like “house hunting“) every few minutes. We’re running a whole bunch of queries – to match the various ways in which people refer to the process of searching for a house: in addition to “house”, there are also apartment, apt, home, condo, pad and flat. The verb can be huting, searching or looking, in various tenses. Finally, there are some implicit indications – the tweet: “we got pre-approved for a mortgage” makes a very good lead. In short, the idea here is that we do the job of figuring out which queries are worthwhile and then do the tedious work of continuously running those searches for you automatically so that you don’t have to do it. You can roughly tell which expressions we search for by looking at the high-lighted part of the tweets on demandspot.com.
In order to retrieve the tweets in close to real-time, while not creating a heavy load on Twitter’s servers, we change the time interval between consecutive searches dynamically – according to how fast tweets are coming in. So at times when tweets are few and far apart (like night time in America) our queries slow down as well.

Next up, every tweet goes through a “semantic engine”. This is a nifty piece of software that has some understanding of human language. The problem we’re solving here is that search by keywords alone retrieves many irrelevant tweets. For instance, “how’s the house hunting going?” and “good luck house hunting!” mean that someone else is doing the hunting, not the tweeter herself. In fact, search retrieves more irrelevant tweets than relevant ones, including references to the past or future (“last year when we were buying a house”), spammers, bots (tweeting listings from Craigslist for example) and more. We also differentiate between supply and demand (“House hunting in Denver? check this out..” is probably written by a Realtor, not a home buyer).
Last but not least, the semantic engine also identifies location names in the text, and assigns latitude/longitude values so we can place the tweet on a map. This is pretty difficult since many names are ambiguous: Los Angeles is commonly refered to as LA, L.A, but the La in “La Perla” is something else. Another example: Louisville is probably Louisville KY, but could also be Louisville IL… (more on this subject below).

At this point we already did quite a lot, but we’re still far from done! Twitter users provide relatively few details about themselves – name, location and a short bio are all there is, and while professionals usually fill these up, more than half of tweeps don’t bother providing all of them. Nevertheless, it’s worthwhile to know a little more about a person before you contact them. Other social networks to the rescue: we use some clever algorithms to beef up our leads by adding information about them from their other social media profiles. Right now we’re able to find Twitter users on other social media in about 40% of cases – and we keep getting better at this. You can when we’re successful from the lead’s contact info in DemandSpot – if we managed to match someone to their other online profiles, you’ll see a bunch of links to those other websites.
Why is this important? Let’s take location for example. The lead’s location is probably the single most important factor when searching for real estate leads. However, far less than 10% mention where they’re looking in the tweet itself, and less than 50% specify their location inside their Twitter profile. This means that many leads are lost unless we can tell where they’re from. By locating their profiles on other social media, such as MySpace, Facebook, LinkedIn, Digg, Yelp and many more, we’re sometimes able to determine their location after all. Automatically locating these profiles with a high level of accuracy is no easy feat of course – names for example are too ambiguous so we don’t use them, but this isn’t the main focus of this post, so I might expand on this issue at another time.
Another good use for alternate sources of location is disambiguation. In the example above, when someone says they’re house hunting in Louisville and their MySpace profile says that they’re from Illinois, then we make an educated guess that they’re searching in Louisville IL, not KY.
In short, this trick allows us determine pretty accurately the location of another 30% or so of leads – above what you’d be able to get through Twitter alone.

Now comes the last part in this “behind the scenes” exposition. At this point we index the lead information in our database making it easy to search for.

Our website itself is simply a Twitter web-client that takes advantage of all the extra information we’ve collected to make searching for leads fast and simple. There’s more to it – we keep your prospecting history for you, provide analytics and more, but that’s a subject for another blog post – stay tuned.

Now if you’ve come all the way to the end of this post, you must be thinking to yourself that all that I’ve described could be applied to other kinds of Twitter searches – not just house hunting. You’re absolutely right, and we’d love to hear your ideas on the matter. Thanks!

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

<span>%d</span> bloggers like this: