Hiding links with the robots meta tag


If you’ve read my posts before you might of notice I’m a little paranoid about competitors mining my links. In part that’s because I don’t want them to see some of the stuff I’m up to! But more importantly when you’re working in competitive sectors like gambling or travel every link counts and the difference between ranking 1 or 2 could be a matter of a couple of decent links- and that can be worth hundreds of thousands of pounds.

I’m always playing around with ways to throw competitors off the scent and here’s one I’ve been testing lately. My testing is never as rigorous as it could be but hopefully its a bit of insight you can go away and play around with yourself.

The hyphothesis

If I put a link on a page which isn’t indexed (using the robots meta tag) will it:

a) Pass link weight

b) Show up on backlink analysis tools like YSE or OSE

The test

What I did was find 3 pages at random on decent domains. I didn’t want pages with thousands of links because it would be harder to see if my links were showing up in the backlink reports. I’ve given the original link numbers for each page below in the brackets…

http://en.wikipedia.org/wiki/Comparison_of_the_imperial_and_US_customary_measurement_systems (YSE47, OSE46)

http://onlineservices.bournemouth.ac.uk/courses/Course.aspx?course=74&school=HSC&level=ug&code=BSOT&mode=ft (YSE1, OSE1)

http://www.bbc.co.uk/scotland/music/celticconnections/2009/artists/dirk_powell/index.shtml (YSE0, OSE0)

I then linked to each page using unique ID numbers as the anchor text as you can see below.

This page sat on a little microsite I own. Not the best site in the world but its well indexed, homepage PR 4, few links, yada yada – that’s not really important. What’s important is that the page used a robots meta tag noindex, follow:

<meta name="robots" content="noindex,follow" />

For anyone that doesn’t know, what this setting should do is let Google (and other robots like SEOmoz) see the page, follow any links on it but not return the page itself in any search results.

I then linked to the page sitewide so it got plenty of internal link juice.

I then gave it about 3 months. Primarily that was because I completely forgot to write this up until now but also gave plenty of opportunity for the link data sources to update themselves. The linkscape index in particular takes quite a while to update but there’s been a couple of index updates since then.

The results

So there was 2 elements to this. The first was whether those links would ‘count’ to Google. We can tell these links are being counted because if you search for the unique ID numbers in Google it returns the target pages for each link as you can see on the links below:

BBC page

Bournemouth uni page

Wikipedia page

Nothing too surprising here. This is exactly how Google should treat the robots tag, but still reassuring to know it works.

The second and more important part of this test is checking the backlinks of those pages in our favourite link analysis tools.

I checked each of the 3 pages in Yahoo Siteexplorer, SEOmoz’s Open Site Explorer and Majestic SEO. In none of these tools did my page show up in the backlink reports. I’ll show you the link reports on the BBC page below…

The majestic report

Open site explorer report

YSE report

Again this is basically what we know should happen. The crawlers which provide this data should be obeying the robots tags. Still its worth testing these things rather than relying on blind faith.

So how can we use this in our link building?

Well for starters anytime you’re paying for a link you’d rather your competitors didn’t see like an advertorial, sponsored review or presell page you could get the webmaster to robots noindex the page with the links on. If you’re buying up high value links this should help reduce the chance of this kind of unwanted attention.

Second if you’re running a few (or hundreds) of your own sites or blogs you’d probably rather your whole network wasn’t exposed in your backlink profile so only put your links on certain pages and noindex, follow those.

Basically it’s going to be far harder for competitors to map your link networks and copy your tactics or spam report you if those pages aren’t in the web index. Who knows you might even be able to throw Google’s  manual reviewers off the scent if you were worried about that sort of thing.

What we don’t know is whether a page which isn’t indexed will pass the same amount of link weight as a link on an indexed page. That’s for another day…

10 Comments Hiding links with the robots meta tag

  1. Gary

    Hi John

    Nice post, my first thought would be is their such a thing as a high value no indexed link?

    I don’t think this is something ive heard anyone else talk about, it would be interesting to see if a non indexed page passes pagerank.

    1. admin

      Hey Gary

      I certainly think they pass toolbar pagerank, and they seem to pass anchor text. But whether they can pass real ranking value needs to be properly tested- working on this at the moment but its always a tricky thing to measure!

  2. Paula Albocino

    Hi John,

    Thanks for sharing. We met at the Pro SEO, by the way. :)

    I have one question for you. If the page was only blocked with ‘noindex’, without the follow command, would the links inside this page still count for Google?

    Thanks and regards,

    1. admin

      Hey Paula, good to meet you this week:-)

      If you just used meta ‘noindex’ the links would still be followed by default.

      Follow is the default state so unless you specifically add ‘nofollow’ to the meta tag yes in theory the links should count to Google. Hope that answers your question!

        1. admin

          It was a decent conference although heard quite a few people say it wasn’t as good as last years. Have to catch up at another one soon!

  3. Paula Albocino

    Hi John,

    Thanks for your answer. About your reply to Gary above mine “I certainly think they pass toolbar pagerank, and they seem to pass anchor text. But whether they can pass real ranking value needs to be properly tested”. Is there such a thing that a link can pass toolbar pagerank and anchor text and not pass ranking value?

    1. admin

      Well bit of a grey area I guess…

      In my experience toolbar pagerank is pretty crude in that if a PR3 page links to a PR 0 page for example the linked to page will pickup a PR score of 1 or 2 in the toolbar regardless of any other factors which could limit the value of the link (penalties etc)

      So what I mean is that we can’t simply test whether noindexed pages pass ‘value’ just by looking at toolbar pagerank so we’ll need to test another factor like the ranking of a page for a particular keyword. If that makes any sense at all!

  4. Justin Knightley

    it’s great to see real data John, good work!

    ‘What we don’t know is whether a page which isn’t indexed will pass the same amount of link weight as a link on an indexed page. That’s for another day…’

    Good luck with testing that!

    You mention some good backlink checkers, have you tried SEO Spyglass? It’s my current fav. – give the free version a go – you’ll be hooked!

    I’d be really interested to learn what tools/networks you regularly pay for in your backlink aresenal. Particularly backlinking your backlinks.

  5. admin

    Cheers Justin

    I haven’t tried SEO spyglass actually but been meaning to give it a look.

    In terms of tools/ networks the public one’s I use most regularly would be things like content crooner, IM automator and article ranks. All really low level stuff but I still think there’s value in knowing you’re getting a certain volume of links on near auto-pilot so you can concentrate on more valuable stuff. I’ve talked a bit more about that stuff over here – http://www.johnmcelborough.com/ecommerce-product-pages

    Give us a shout if you’re ever in brighton by the way! J


Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>