Documentation

 

"Ludicrous Speed": Where We Get Our Address-Processing Superpowers

The "Fast Lane" Answer

Whether you're looking for geocodes, address validation, or a new license from the DMV, you probably want your results fast. While nothing can really be done about the DMV, something can be done about how fast you get your data back when you process an address. Here at SmartyStreets, we're always working to improve our already-awesome address processing engine so we can get things back to you faster than a speeding bullet.

Our SLA guarantees sub-100ms response times for address lookups, not including external network latency (like we said above—we can do something about our speed, but not the speed of other things like your internet, or the DMV). We promise those same levels of speed on batch requests and we never throttle. And our speed is reliable; we promise a 99.98% uptime. In short, we can take your 100,000 addresses and have them processed in seconds.

We're betting that you're curious how we do that, considering most other places will take anywhere from half an hour to four business days to give you your results. Well, the short answer is that we were once struck by lightning and granted superspeed, and now we spend our days processing addresses and our nights fighting crime.

As for the long answer, let us explain:

  1. We're fast because we designed our system to be fast.
  2. Other service providers are slow because they put up roadblocks to slow you down on purpose.
  3. There may be a slowdown on your end—if you're stealing your neighbor's 1mbs wifi, or you're using your Uncle Stan's Windows '95 desktop, that's on you.
  4. What kind of address processing are you looking for? Some providers are offering economy-sized gift baskets full of information, but they're asking for hours or days to put together everything you're asking for

It's as simple as that. We're fast because we make ourselves fast so you don't have to worry about it.

The "Scenic Route" Answer

The speed you want out of address processing services like address validation/verification and geocoding can easily be compared to the speed you want when you drive from place to place. And because we like things to be easy here at SmartyStreets, we're going to do that.

Let's imagine that your address processing journey is a trip on the freeway. You drive up the ramp and onto the freeway, navigate appropriately through traffic, then take the off-ramp toward your destination. Car size, speed limit, number of lanes, traffic levels, and distance to destination all affect the speed at which you complete your trip. These are all features similar to attributes that can make your trip down Validation Lane either faster or slower. We'll go over them in this order:

  1. How far to your destination?
  2. How fast is your car?
  3. How fast can you get on the freeway?
  4. How fast can you go on the freeway?

How Far to Your Destination?

This one is entirely inside your control, because you're the one who picks it. If you're just headed down the street, then great. If you're headed all the way to Anchorage, Alaska, you're going to run out of freeway long before you run out of road. The point here is, if you're traveling a long distance, you can't expect to arrive there after a 15-minute drive.

How does this relate to our topic of doing things like getting the geocode for an address? Simple: it's like asking yourself just what kind of services you need. Some providers offer more comprehensive data and services. Some can even give you things like email address information for the people living there, or average climate and weather patterns for the area.

Sure it's cool to have the census data regarding the locations you have addresses for. But do you really need it? And do you need it enough to justify the three days it's going to take for your 10,000 addresses to come back to you? That difference in work to be done is like an addition in distance to be traveled—an increase in either work or distance means an increase in the time it takes.

Part of the reason we're so fast at SmartyStreets is because we've narrowed our focus. We do address validation and geocoding. We provide some supplemental data beyond that because we like being helpful and throwing in extra stuff, but those two services are the core of what we do.

Not every provider does what we do. Differences in how they offer validation or what version of geocodes they return affects their speed. So choosing a provider that does more, or who provides data that takes an unnecessarily long amount of time to put together, is like choosing a destination that's three towns over—it's going to take longer.

Here's a brief breakdown of why our address validation and geocoding can be done so fast.

Address validation

Address validation (in case you didn't know already) involves using an address database like the one kept with the USPS to verify if a particular address is real. You check the address in question against the authoritative database, and if it's on the list, it's real. This process is relatively simple—and thus really quick—because all you're doing is trying to find a match. You hold up your address and say "Is this bad boy on file?" and we look at the list and give you either a thumbs up or a thumbs down. That's the gist of it, anyway.

However, in order to compare your address to the address in the database, both your input and the addresses in the database have to be formatted the same way. That way, they'll actually match each other; if they didn't match, the address wouldn’t validate. So before being validated, addresses are standardized, meaning they are reformatted to meet the specifications of the database they're being checked against. Misspelled words are corrected, capitalizations and abbreviations are set straight, and missing data is filled in if possible. If you ask real nice, we may even parse it for you.

Who are we kidding; we're gonna parse it for you anyway.

The point is, what we're doing is a set system, and we've designed our process such that we can accomplish all of these tasks quickly and accurately—from standardizing and parsing to validating and geocoding. We use the databases and the format guides to our best advantage, and the result is a wicked-fast process speed.

Geocoding

Geocoding is a fun little matching game, where you connect a physical/mailing address to GPS lat/long coordinates. The purpose is so that you can do things like use your Google Maps app on your phone to navigate to that new steakhouse you read about in The Daily Planet. It's a good thing to pair with address validation, because if the address is valid, that means the location actually exists, so the geocode won't lead you to the middle of a cornfield.

The thing is, there are two types of geocodes: interpolated and "rooftop level" accurate. "Rooftop level" geocodes were acquired by someone taking a device and walking right up to the premises in question, then recording the coordinates. Interpolated geocodes are acquired by taking two geocodes already on file (from either category), and dividing the space between them evenly. The space is then applied to the addresses in between to estimate the geocode.

"Rooftop level" geocodes take longer to acquire beforehand, and are thereby more expensive, a cost that's usually passed on to you when you ask for the geocodes. Interpolated geocodes can be produced at the time of request, and don't require someone to be physically present at the location to record them, so they're cheaper. But they're being processed when you ask for them, so it takes longer to get your answer than it would for a "rooftop level" code.

Unless you're working from a fixed database.

At SmartyStreets, we use interpolated geocodes, which as we mentioned are good enough to get you line-of-sight of the destination about 99% of the time. But we also validate, which means we never give you a geocode for an address that isn't real. Moreover, it means we don't have any geocodes on file that aren't tied to a real address. That means all the interpolated codes are already on file; they've been interpolated in advance.

So when you validate an address with us, and we tell you that everything's hunky dory and hand you a geocode telling you exactly where the place is, it happens quick. You order a "number 5," and we turn around and give you the whole meal, because the fries are already in the little cup, and the burger patty is pre-grilled.

How Fast is Your Car?

Your car can only travel so fast on the freeway. It has a top speed, after all. Even if there was no one else, no cops, no speed limits, and it was just a straight shot from here to there, you'd eventually hit a barrier. Drop your foot to the floor all you want, but your Prius just isn't going to break the sound barrier anytime soon.

This is a limitation on your end. The car you choose to drive has an engine that can only spin at so many RPMs. You could get a car with a more powerful engine (a fire-engine red Ferrari might do nice). But even then, there are limits. You will never find a way to get a car that has limitless speed. You need to be hit by lightning to produce an effect like that. Or be born on Krypton.

This is the same as the hardware limitations and connection speeds on your computer. The hamster on a wheel inside your desktop can only run so fast before it has to stop and take a drink out of that upside-down water bottle. And if your internet connection has to "turn on," accompanied by sounds of static, phone rings, and electronic screeches ... odds are you're going to have a hard time reading this article, let alone process addresses.

In short: there's nothing we can do about how fast your car can go. We can only control how fast your car may go within our system.

How Fast Can You Get on the Freeway?

Now here's a fun one: when you get on the freeway, how long does it take you? See, in life, the on-ramp and the freeway itself are never truly empty. But sometimes it is the exact opposite of empty. Some places will have a meter—like a stoplight you would see at an intersection—that only lets cars through the on-ramp one at a time. What's worse, regardless of the Ferrari you might own or the jet-assisted take-off engine you've strapped to the back to get the most speed you can out of it, you still have to wait to put pedal to the metal, since there are a bunch of cars in front of you.

These circumstances correlate to several things. The most important is called throttling: a tool used by some providers to reduce strain and load on their servers by limiting how many requests are being thrown at it at a time. They say "Ok, use our service, but you can't send more than ten requests a second." That means you're going to be spending a long time processing those 5 million addresses you have.

Sometimes throttling comes in the form of usage limits—i.e., "Use as much as you want per second, but you can only give us 2,500 requests a day." It gives you more freedom over small batches, but over the long haul you are still similarly hamstrung.

SmartyStreets doesn't throttle. Ever. At all. Why? Because we're not afraid that you're going to overwork our servers and cause lag. In fact, we dare you to try.

How Fast Can You Go on the Freeway?

We don't turn the meter on at the on ramp for a good reason: we've got plenty of space on the freeway for everybody.

Our servers are like a freeway with dozens of lanes; ample space for thousands to drive at high speeds. We built it this way so that we can say "We don't throttle, and we don't have to." You can throw as much as you like at us, and grab a hundred of your closest friends to get them to do the same. It won't matter. You'll still get the wicked awesome speeds we promised upfront.

What's more, the hardware we use and the software/code we use is optimized for the roles they fill. We're giving you the Ferrari, so to speak, and taking away all the speed limits. So as long as you can make it to our on ramp, you're set.

Not every provider has the capacity to just waive the flag and let you run like this. And many aren't willing to commit the capacity they do have. But we think you're worth the extra effort, so we're making a exception for you. Ok, maybe we do that for everybody, but we wanted you to feel special because, hey, everybody deserves a little validation.

All the Junk We Left in the Backseat

Batches

A batch is anytime you lump a bunch of requests together to process them faster. So, say, uploading an excel file full of addresses. You don't want to be entering all that stuff manually; That'd be like walking from New York to LA.

We support batch processing here at SmartyStreets. Some providers do, some providers don't, but we make a point of offering it to make your lives much easier and simpler. It's kinda like if you showed up at the on ramp, and we gave you a minivan, full-size van, or a bus to drive, depending on your needs.

Uptime

It doesn't matter how fast our service is if it's never up and running. We have stellar uptime reliability (99.98%), and we have a long series of redundancies built in on behalf of our system, to make sure every "just in case" is covered. No need to worry that someone's going to trip over the power cord out here, and that you're suddenly going to lose access to our service.

API

The last detail we're going to mention is bonus functionality, including API support, among other things. Sometimes being fast means being available in the most efficient form. If you only need processing one at a time, but you've got lots of little guys spread throughout your day, our API might be right up your alley. Tie directly into our sweet system, and find out just how fast your life can be when we're doing all the validating for your addresses as fast as they come to you.

Are We There Yet?

By doing everything in our power, and by giving you options for optimizing the above features, we're able to give you the super-fast service we're known for. How fast? Like, "make the Kessel run" fast. We're known as the fastest in the industry, and it's a moniker we're proud of. We combine this infamous speed with real human beings who actually answer phones (that shouldn't be a superpower, but it is).

The point is, like superheroes, we're here to help. We want you to know you can rest easy at night with us watching over the city (or at least the addresses in it). You deserve a provider who will fight supervillains for you, and that's why your friendly neighborhood SmartyStreets is here to save the day.