"Find something you love, and go for it."
DIY Dollars header image 2

Don’t Waste Backlinks: Tell Googlebot About Unindexed Referrers with a 3rd-Party Sitemap

June 2nd, 2007 by shawn · 16 Comments

Many of you readers are also bloggers – I know this because I sometimes get comments to posts like “Great post!” and there’s a link in there. (That’s fine. I’m glad you stopped by. But say something useful or ask a question if you could – we don’t take kindly to spam ’round these parts.)

As a fellow blogger, I want to let you in on something you may not have thought of before – a way that you can increase the value of your site by getting more backlinks that the search engines don’t know you have. There are backlinks just sitting there, links on other people’s sites that point to your site but don’t “count” in Google or other search engines because they simply aren’t indexed.

So let me break it down. You might know the following, but let’s review some basics of links and ranking:

  • The ranking of your site is muy importante in the crazy world of websites
  • The overall most important factor in being ranked high by search engines is the number of backlinks you have
  • A backlink is a link on another site to your site
  • The more backlinks you have, the better your ranking, and the more people find you in search, the more traffic you receive, and the more valuable your site is
  • You can optimize your site like crazy (it’s a very important factor), but at the end of the day I’d say it comes down to the number of backlinks you have

So, the task at hand is to get backlinks. Right? There are a few ideas about that. These are things you’ve seen everywhere else, too…

  1. Design a blog/CMS theme. Put a link to your site in the footer. Give your awesome theme away, with the stipulation that no one messes with the footer. Every time someone uses your theme, you got a nice link to your site on every page of their site. Too hard for most of us.
  2. Sign up for every link-exchange program you can and trade links. A good thing to do, but only accomplishes so much.
  3. Buy links. Go to eBay and you can find people who will put your link on 100′s of sites. Can be expensive and you don’t always get what you pay for. (Yes people get away with selling PR4 links that aren’t really PR4 links.)
  4. Comment on blogs, one at a time, with things like “Great post!” and put your link in there. Okay method, except it’s very time-consuming and if you make throwaway comments just for the sake of getting a link there’s a good chance your comment will just get deleted.

Not to say there’s no value in these methods and other methods. Use ‘em all. We need lots of backlinks so that when Google decides to do a PR update your blog can actually get a decent rank. So where do we get these backlinks??

There are a few additional methods I know of, some of them sneaky and some of them common sense. Today, I’ll tell you a common sense one. The sneaky methods will have to wait but I will (for now) just mention that social networking sites are a great way to get massive backlinks if you know a few ways to use their systems.

Also I have to clarify a couple things: This method will not get you new backlinks but it will ensure that no existing backlinks are being wasted. Also, don’t expect massive amounts of backlinks (I’m talking 100′s or more at a time).
Another thing I have to say is “backlinks != traffic”. You might not notice a traffic increase – or if you do, it will be a quick jump and then it will level off. Remember what backlinks really help with is Google’s view of your site’s popularity which increases your PR.

Anyway, here’s what to do

First, you will need to extract a list of all your referrers from your site logs. This can be done from Urchin (export to a CSV file), but you can get this from the raw server logs as well.

  • If you have cPanel hosting, go to Raw Log Manager and make sure “Archive Logs in Your Home Directory” is checked. You can access them here every day.
  • If you have Urchin, you can use Urchin to export a file.

Whichever method you use, you want to end up with a text file full of your referrers. Now, we need to make all of these plain text URLs look pretty so Googlebot will make sweet love to your site, get this list of URLs, and index them. (There are a few ways to get this part to happen.)

So now you have a messy file that needs to be cleaned up. You want anything with your URL, Google, Yahoo, etc. to be removed and keep only unique URLs from other sources. You also want the file to end up with one URL per line, with no extra characters. (Ideally you would remove any already-indexed referrers from this list as well. If you know a scripting language like PHP, this can be done. Anyone care to help?)

If you’re a regular ‘ol lunkhead like me and need a tool to help you do this stuff, download Crimson Editor and open up the textfile of referrers in Crimson Editor (CE).

  1. You first will need to take out all the junk using the Find & Replace function (CTRL-R). You should have one URL per line in your file, but each line will look like ” * www.someurl.com/index.html”. So you will do a CTRL-R and put ” * ” in the Find field, and put nothing in the Replace field – hit backspace. Crimson Editor will then take out the extra spaces and the * symbol.
  2. Next, we need each of these plain URLs to be actual HTML-formatted links. If you have a lot of time on your hands (or if you only have a handful of URLs in the list), you can manually type ‘<a href=”http://www.someurl.com”>www.someurl.com/index.html</a>’ for each one. But we’re all smart & lazy so you want to use Find and Replace in Crimson Editor to do this work for you. Enter these in the Find & Replace and check “Use Regex”.

Find What: (http[-_a-zA-Z0-9:./.?]+)


Replace with: <a href=”1″>1</a>

You’ll have to wait about two entire seconds, and all the URLs should be properly formatted. Just like magic.

You now have a list of links to all URLs that link to you. Basically, you have the makings of a third-party sitemap.

Save it as an HTML file, upload the file to your site, and tell Google about it (via your preferred method). Or, break up the list into manageable chunks and post it to one of your blogs (you do keep a few free blog accounts on the side, don’t you?) in multiple posts and ping Pingoat. Or…well, you can get creative with it and figure it out.

So to recap: we all have referrers that link to us that aren’t indexed. These backlinks have the potential to count for something if they get indexed by the search engines. By putting all these backlinks in a third-party sitemap page and getting the third-party sitemap indexed, you’ve given value to your referrers which adds some additional value to your site.

And everyone lives happily ever after.

Related Posts:

    Tags: SEO · WordPress/Blogging

    RSS feed


    Comment by Sanjeev
    2007-06-02 15:38:07
    (Comments wont nest below this level)
    2007-06-03 05:24:34

    Hi, great post ;)
    I found a plugin that creates a sitemap automatically. Won’t that work? I’m really bad at this tech stuff :(

    (Comments wont nest below this level)
    Comment by shawn
    2007-06-03 05:46:17

    Yes – any of the sitemap generators will work fine as far as getting all the pages on your site indexed. So you make your page of referrer URLs, let’s call it “backlinks.html”. From your homepage, you create a link to backlinks.html. Then you use your sitemaps generator to make a new sitemap of your site (which of course will now have backlinks.html) and you then tell Google to come visit your site because you have something to tell it. It grabs your sitemap and looks at this new page on your site called backlinks.html and then investigates all the links contained therein and hopefully indexes all of them.

    So to answer your question…yeah, should work. :)

    Comment by pf101
    2007-06-03 17:37:23

    I think I’m going to have to read that again (and again…) to get it all to sink in. Lots of good information. I’m just learning about all this stuff and reading posts like this just make me realize how LITTLE I know! This one is going into my (ever expanding) list of bookmarks of things to come back and implement once I know what they mean. :-)

    (Comments wont nest below this level)
    2007-06-06 06:34:00

    [...] you read my previous post about link saturation, unindexed referrers, and third-party sitemaps (Don’t Waste Backlinks – Tell Googlebot About Unindexed Referrers…) then you probably had either one of the following three [...]

    (Comments wont nest below this level)
    2007-09-05 05:54:33

    [...] is an Interesting Technique. I thought I’d take a look at generating these. As with anything that looks tedious, I [...]

    (Comments wont nest below this level)
    Comment by Lucio
    2007-09-21 05:37:49

    This is one of the best posts I have read about in a long time. I always wondered how I could get backlinks linked in google. Great idea!

    I post in one particular forum, I have about 1000 posts with my link in the signature. What I’m going to do is get the links for thread and do just this.

    Thanks a million! This is scrapbooked and bookmarked.

    (Comments wont nest below this level)
    Comment by SkyPenguin
    2007-12-23 00:05:24

    Ive already done this on my site.
    The only thing that worried me was that if Google sees a page with nothing but hundreds of links I might get penalised for being a link farm. So Ive added the “nofollow” tag to all the links. (I believe that Google still follows “nofollow” links – just doesnt give them weighting).

    Ive also wondered whether just having adsense on a page allows Google to automatically see all the incoming referer/back links.

    (Comments wont nest below this level)
    Comment by SkyPenguin
    2007-12-23 00:09:58

    One other point. If you do create a page like this its probably a good idea to make sure you have no keywords like “referer” or “backlinks” that indicate the page content.
    These keywords are very likely what the referer spambots will be looking for via Google searches.

    (Comments wont nest below this level)
    Comment by Fedmich
    2008-01-09 17:17:20


    That regex find-replace for url is what amazes me more :) very useful.

    Im a bit scared to use this, because Google might really penalize the site, I think

    Thanks for the tip again :)

    (Comments wont nest below this level)
    Comment by c5
    2008-05-29 01:29:21

    I’ll have to try this. Unfortunately, when I had to reformat my HD my username and password of my cpanel got lost. :( I’ll have to extract it first then I’ll try this. I’m just a bit confused (more than a bit, actually)…

    So, as I understand…
    -I’ll make a list of all those who visited my site.
    -Post that list as html somewhere else that’s already indexed.

    Is that it?

    (Comments wont nest below this level)
    Comment by Rob Schmidt
    2008-08-11 22:21:00

    Even though I have made the sitemaps for google they are still slow to index my site , any suggestions to speed them up?

    (Comments wont nest below this level)
    Comment by fernando
    2009-02-04 15:51:18

    Even though I have made the sitemaps for google they are still slow to index my site , any suggestions to speed them up? how

    (Comments wont nest below this level)
    Comment by Jeff Gravely
    2009-02-23 11:29:51

    Seems like a good idea. I wonder if its best to post the file on the site that it is for. It seems like that might make all your links count as reciprocal links, possibly lowering their value in Google’s eyes.

    (Comments wont nest below this level)
    Trackback by Seo
    2011-08-12 03:01:06

    Niche Sites…

    Don’t Waste Backlinks: Tell Googlebot About Unindexed Referrers with a 3rd-Party Sitemap…

    (Comments wont nest below this level)
    Trackback by cPanel Web Hosts
    2011-09-15 10:48:30

    cPanel Web Hosts…

    [...]Don’t Waste Backlinks: Tell Googlebot About Unindexed Referrers with a 3rd-Party Sitemap[...]…

    (Comments wont nest below this level)

    Sorry, the comment form is closed at this time.