bitly blog

Do kittens really rule the Internet?

  We long ago crowned cats the kings of the internet, but is   that really the case? What data supports their reign?

  And, with Thanksgiving approaching, might the turkey be   competitive?

  We went deep into the bitly data to discover which   animals really rule the internet, and the answer might   surprise you:

While cats have a respectable lead, the winner is clear — dogs are 37% of the total results in the cute animal set, and feature in 50% more pages than cats!

  Our favorite dinner bird gave a strong showing, however, hovering   between monkeys and bears. If we dive into the turkey-based content,   we see an amazing knitted hat (left), the basic roast turkey recipe   from RealSimple and, of course, a LOLTurkey.

  We wish you a happy Thanksgiving filled with delicious food, cute   animals, and entertaining URLs!

bitly and Verisign

One of our investors brought his teenage daughter by the office the other day. She was puzzled.

“Bitly has an office?” she asked her father. “I thought it was, you know, just part of the Internet.”

We took that as a great compliment — over the last three years, bitly links really have become a ubiquitous part of the web.  A recent Microsoft Research report even claims that short links can account for as much as 1% of the new URLs created on any given day. 

With that kind of scale comes a great responsibility in terms of technical reliability and redundancy. Which is why we’re pleased to announce a new agreement with Verisign, which operates two of the Internet’s root nameservers and much of the web’s DNS infrastructure. If there’s a single company that qualifies as the steward of the Internet, it’s Verisign.

Verisign and bitly already work in tandem nearly every time a short URL is clicked. On any given day, bitly translates hundreds of millions of short URLs into standard web addresses; last month alone, we handled 8 billion such redirects. Verisign takes those long URLs and translates them into IP addresses, resolving over a trillion monthly DNS queries, including every URL hosted on a .com, .net or .gov domain. 

These two steps constitute the core infrastructure of the social web, and bitly’s relationship with Verisign aims to make them completely reliable and blazingly fast. 

Beginning this fall, bitly’s primary data center will be hosted on Verisign’s global infrastructure. That’s saying a lot, since Verisign’s infrastructure has maintained the .com namespace for more than a decade with zero percent downtime.  

Verisign’s architecture has been integral to the growth and stability of the Internet at large, and we could not be more excited to work with them. Scientists at both companies are already poring over volumes of DNS resolution data — data that will help us answer fundamental (and fundamentally awesome) questions like: “what actually are the most popular websites on the Internet?” and “just how big is the Internet, anyhow?” 

Stay tuned, as we’ll be sharing our findings here.

We See into the Future

bitly sees the future!

Not in a time-travel, causality-breaking way, of course. bitly sees the future by knowing what is popular and exciting on the internet before anyone else does.

Scientifically-minded purists will argue that we only know the present. Fair enough.  Yet we can see much of the future by understanding the present while others are still catching up with the past.

bitly has been working on real-time search powered by bitly data for quite some time. Unlike other search technologies, which weight results by the quantity of references or links, bitly search weights results by cross-platform social engagement. Collaborative filtering (the wisdom of crowds) is an obvious value of social media, and real-time search is an obvious way to package that value.   

Finally, we are ready to release several different services based on this new type of search technology.  The first, launching today, is called “reputation monitoring.” What it lacks in catchy naming, it makes up for in awesomeness: given a search term, it will alert you when there is a sudden change in engagement and/or sentiment around content involving that term. Because these alerts are built on bitly’s realtime social data, you can find this content long before you see it on a search engine based on crawling web pages, and probably long before you see it on a news site. 

We created some interesting monitors and thought we would share them with everyone. Try these: politics, products, and startups.  You won’t get the email alerts when these sample terms trend, but you can click on the graph and explore the content.  The monitors live off of real-time bitly data, which on average lead major search engines by 12 hours or more.  If you have any ideas on what else we should be monitoring, let us know. Of course, you can monitor any terms you like with a bitly Enterprise account. 

Extracting the ebb and flow of trending topics and identifying viral content hours or days before you will see it in the news, on Google, or on Bing is quite amazing. We don’t know where it will lead. The internet made the world flatter. Social media and real-time content sharing are making the world thinner.

You just shared a link. How long will people pay attention?

How long is a link “alive” before people stop caring? Does it matter what kind of content it is, or where you shared it? At bitly we see a lot of links, and while every link is special, we’re learning a few general principles that we can share.

Let’s take a look at one particular story - Baby otter befriended by orphaned kittens - which was first shared by StylistMagazine on Facebook on Tuesday at 7:12am.  If we plot clicks over time for this link, we see:

Rate of clicks per 10 minutes on “Baby otter befriended by orphaned kittens”

We can evaluate the persistence of the link by calculating what we’re calling the half life: the amount of time at which this link will receive half of the clicks it will ever receive after it’s reached its peak. For this link the half life was 70 minutes, which captures all the clicks between the grey lines on the graph above.

Let’s look at a second link - East Coast earthquake: 5.8 magnitude epicenter hits Virginia - , this one first shared by the Washington Post on Twitter.

Rate of clicks per minute on “East Coast earthquake: 5.8 magnitude epicenter hits Virginia”

While the exact details of the traffic are a little different, and the scale of the traffic to this link is much larger, we see essentially the same pattern: a fast rise, and a more relaxed drop-off. Noticeably though this link a half life of only 5 minutes: after 5 minutes this link had seen half of the clicks it would ever see.

This link is associated with a very timely event (an earthquake on the US East Coast) as opposed to the previous link (pictures of otters and kittens are clearly interesting all the time). We think that this difference in content drives the difference in dynamics of these two links. However, one alternative theory that comes up again and again is that the dynamics of the link traffic depend on where the link is posted: do links posted on facebook last longer than they do on twitter?

So we looked at the half life of 1,000 popular bitly links and the results were surprisingly similar. The mean half life of a link on twitter is 2.8 hours, on facebook it’s 3.2 hours and via ‘direct’ sources (like email or IM clients) it’s 3.4 hours. So you can expect, on average, an extra 24 minutes of attention if you post on facebook than if you post on twitter.

Distribution of half-lifes over four different referrer types. Facebook, twitter and direct link (links shared via email, instant messengers etc.) half lifes follow a strikingly similar distribution.

Not all social sites follow this pattern. The surprise in the graph above is links that originate from youtube: these links have a half life of 7.4 hours! As clickers, we remain interested in links on youtube for a much longer period of time. You can see this dramatic difference between youtube and the other platforms for sharing links in the image above.

The graph shows the distribution of half lifes for each referrer. So we’d expect to see link half lifes of less than 20K seconds (5.5 hours) for facebook, twitter and links shared directly, and we’d be very surprised to see any link maintain significant traffic for a lot longer than 60K seconds (16 hours). But for youtube, we’d be a little surprised to see half lifes of less than 5 hours!

In general, the half life of a bitly link is about 3 hours, unless you publish your links on youtube, where you can expect about 7 hours worth of attention. Many links last a lot less than 2 hours; other more sticky links last longer than 11 hours over all the referrers. This leads us to believe that the lifespan of your link is connected more to what content it points to than on where you post it: on the social web it’s all about what you share, not where you share it!

This post brought to you by the bitly science team! Questions or comments? Email us.

bitly Has an iOS 5 SDK!

As you may have heard, the forthcoming fifth version of Apple’s iOS has some very cool share-to-Twitter functionality baked directly into the operating system. We have been hard at work on an SDK that will allow iOS applications to automatically use branded bitly short links with this functionality. (For example, news.me shares to Twitter with on.news.me branded links.) Our SDK also supports iOS 4 via Twitter OAuth.

Since iOS 5 has not yet been released, we can currently only offer this SDK to developers who have signed the applicable NDAs from Apple. If you’re working with iOS 5 and want to check out our SDK, drop us a line at api@bitly.com!

bitly Acquires Twitterfeed

For years now, we’ve worked closely with our friends at Twitterfeed to enhance their social media publishing tools with branded short links and realtime data. Today, we are excited to announce that we have acquired Twitterfeed, and to welcome both Twitterfeed and its users to the bitly family.

The publishing workflow provided by Twitterfeed constitutes a core part of the bitly ecosystem. Along with other products (such as SocialFlow and dlvr.it) that use bitly to share content and track engagement, Twitterfeed is both creating and consuming gobs of bitly data every day. Twitterfeed had over a million active users last month, and we look forward to empowering them with even more actionable insight from bitly data. We also look forward to bringing the enhanced sharing functionality of Twitterfeed directly to bitly.com, making it easier than ever to collect, organize, shorten and share links.

We are already hard at work with the Twitterfeed team thinking through ways that combine our efforts to make the social web a better place. We will keep you in the loop as we roll out changes to both Twitterfeed and bitly, and we welcome your thoughts and feedback.

Some Hacks from the 1.USA.gov Hackathon!

Many thanks to everyone who came out to our 1.USA.gov hackathon! It was great to see so many friendly hackers and open data enthusiasts collaborating and sharing ideas (and devouring 11 pizzas in about as many minutes).

Our friends at USA.gov posted a summary of the hacks created at all four nationwide hackathons, including an awesome visualization of NASA’s popularity worldwide created by Adam Laiacano at our New York office. Here are a couple more hacks that came out of the New York hackathon:

* A fascinating set of data visualizations by Harlan Harris.

* Sonification of 1.USA.gov data by Niki Yoshiuchi.

Want to hack on the 1.USA.gov data? It’s still publicly available on the USA.gov website. We’ll be hosting another hackathon soon — keep an eye on the hackabit Meetup group for more information!

Our Data Could Be Your Life

Do you know exactly what you were reading 10 years ago this very moment? 10 years from now, you may. The data we create is a living archive of our interests and intentions, and we’re creating more of it every day — a staggering 1.2 zettabytes last year alone. At bitly, we are always looking to learn from our data, and we are thrilled that a particularly interesting set of bitly data is being made publicly available for hacking and analysis.

This Friday evening, bitly will host the New York branch of a nationwide, 4-city 1.USA.gov open data hackathon. The USA.gov team has made the realtime clickstream data for all 1.USA.gov URLs open to the public, providing an unprecedented window into the way that we engage with government content. We are extremely excited to see what talented hackers and coders can do with such a rich and fascinating data set.

In advance of the hackathon, the Measured Voice team has put together a microsite at govclicks.measuredvoice.com that ranks the most popular 1.USA.gov URLs per day by click count (an average of about 56,000 total daily clicks over the last few months). The diversity of content at the top of the list is fascinating; a NASA Tweetup announcement alongside information about the recent FAA furloughs  and a strongly worded FDA warning to Diamond Foods about walnut packaging.

This basic ordered list provides a compelling look at the popularity of specific government URLs – and it is only scratching the surface of the underlying data. Where in the country are most people looking up information about FAA furloughs? (And, using a complementary API like SimpleGeo’s Context, what do we know about the demographics of those regions?) Which social networks drive the most traffic to NASA’s website? What US government content is most frequently accessed outside the US? There’s a wealth of insight encoded in the 1.USA.gov data just waiting to be discovered. 

Indeed, working with data isn’t just a matter of crunching numbers and writing code; it also means knowing what you’re looking for and knowing what you’re looking at. Through the right lens, the data from the 1.USA.gov project could answer questions that we hadn’t even thought to ask – and some of the data that answers those questions may very well be yours.

If you’d like to explore this data with us, then come to our NYC office this Friday (or visit the other 1.USA.gov hackathons in San Francisco, San Diego and Washington D.C.) and build something new!

Where does your traffic really come from?

Lately there’s been a lot of discussion of the relevance of referrers as a way of discovering the source of traffic to a site. At bitly, we see billions of clicks per month on across multiple platforms, instant messages, emails and mobile text messages. This gives us a unique window onto the way people share social objects, and we see how they propagate through human networks and across the various social networks.

 
For example, you can see this graph of a typical link spreading through multiple networks; in this case, Twitter, Facebook, and Google+. This link was first shared on Twitter. It’s possible to undercount traffic from the originating service by up to a factor of five. We demonstrate why below.

The reality of content sourcing and proper attribution is a bit more complicated. What happens when someone sees a link on one network, then re-shares the same link? It’s simple to track this behavior within one social site, but what about when it spans multiple sites and sharing mechanisms?
 
Unfortunately, referrers are a poor metric for assigning proper attribution when they work, and  many links are viewed on clients or mobile devices (approximately 11%) that don’t even support the referrer protocol!
 
First, let’s examine referrer the data that a typical publisher will see. Since traditional and online-only publishers tend to use different methods of promoting their content, we broke them out separately:

This tells us that different platforms drive different amounts of social sharing to different publications.
 
Next, we located the top 100 publishers with active web traffic. This Bitly100 represents a “Dow500”-like index of top publishers, and shows how different social web sites contribue traffic in an aggregate view that any web publisher can use to calculate the impact of various networks on their traffic.
 
We saw traffic from 153,338 distinct referring domains to the top 100 publishers in the month of June. Twitter.com was 11.4% of the referrers to that traffic.  As you can see in this graph, the greatest percentage of referrer traffic can be classified as “direct.” Direct traffic is traffic that comes without a specific referrer code. In the complex social web environment, how might we trace the original source of this traffic?

As in the first figure, we can assume that links propagate from one network to another, as you can observe from the correlations in click times between twitter.com traffic and direct traffic. If the initial link was shared on twitter, and you see 60-70% direct clicks, it’s possible to conclude that the twitter traffic is undercounted by a factor of ~5.

This demonstrates that traffic observed from multiple networks may originate in one particular network. In the case of the URL examined at the beginning of this post, it originated on Twitter and spread out through events on other, diverse platforms. At bitly, we tie it all together.

Visualizing The NASA Shuttle Launch with Public 1.USA.gov Data

A couple weeks ago, we announced a fantabulous 1.USA.gov open data hack day to be held on Friday July 29th. Since then, our data team has been playing with the clickstream data for the 1.USA.gov project, which has been publicly mirrored on the usa.gov site. Here’s a nifty visualization of click traffic to 1.USA.gov links on July 8th, the day of the last manned NASA space shuttle launch:

Since NASA.gov is one of the many US government and military sites that shortens to 1.USA.gov, you can see the entire world light up as the launch takes place around 16:30 GMT.

If you’re interested in playing around with this data at our office on July 29th, you can sign up for the New York branch of the 1.USA.gov hackathon on our Meetup page here. Information about the San Diego, San Francisco and Washington D.C. hackathons (hosted by Measured Voice, SimpleGeo and USA.gov respectively) is available via the USA.gov blog. Wherever you are, we encourage you to hack away at the 1.USA.gov data, and share your project via Twitter using the #1USAgov hashtag.

To help get you started, our friends at Measured Voice have put together a tool called “gogogon,” which produces a list of the daily list of the most-clicked 1.USA.gov URLs. You can play around with the gogogon code over at github.