A comment from Mark Middleton perfectly illustrates the point I was making the other day about visualizing your published surface area. I started this blog in December, and ever since I’ve been running with a robots.txt file that reads:
User-agent: * Disallow: /
In other words, no search engine crawlers allowed. Of course that’s not what I intended. I’d simply assumed that the default setting was to allow rather than to block crawlers, and it never occurred to me to check. In retrospect it makes sense. If you’re running a free service like WordPress.com, you might want to restrict crawling to only the blogs whose authors explicitly request it.
WordPress.com’s policy notwithstanding, the real issue here is that these complex information membranes we’re extruding into cyberspace are really hard to see and coherently manage.
For the record, the relevant setting in WordPress.com is Options -> Privacy -> Blog visibility -> I would like my blog to appear in search engines like Google and Sphere, and in public listings around WordPress.com. Interestingly, although I’ve made that change, it’s not yet reflected in the robots.txt file. I wonder how long that’ll take?
It’s interesting that .com made the default that direction, while the software as it “ships” from wordpress.org has (I believe) the default set for content to be visible.
Yes, Steven is correct.
Just checked my own. WordPress (from .org) installs locally without a robots.txt at all.
Your ‘visualizing’ link to the earlier post has a slightly incorrect URL.
Hmm…I co-incidentally started up a WordPress blog just today and I find that it’s already set to “I would like my blog to appear in search engines like Google and Sphere, and in public listings around WordPress.com.” by default.
On creating a blog there is a check mark (defaults to checked) with the text “I would like my blog to appear in search engines like Google and Technorati, and in public listings around WordPress.com.” Maybe there was a problem where the check mark was not respected, but I can’t reproduce the problem now.
Many of our customers do not start with an understanding of the permanence or searchability of content published online. They have no comprehension of “published surface area”.
It’s the first time I commented here and I must say you share us genuine, and quality information for bloggers! Good job.
p.s. You have a very good template for your blog. Where did you find it?