A comment from Mark Middleton perfectly illustrates the point I was making the other day about visualizing your published surface area. I started this blog in December, and ever since I’ve been running with a robots.txt file that reads:
User-agent: * Disallow: /
In other words, no search engine crawlers allowed. Of course that’s not what I intended. I’d simply assumed that the default setting was to allow rather than to block crawlers, and it never occurred to me to check. In retrospect it makes sense. If you’re running a free service like WordPress.com, you might want to restrict crawling to only the blogs whose authors explicitly request it.
WordPress.com’s policy notwithstanding, the real issue here is that these complex information membranes we’re extruding into cyberspace are really hard to see and coherently manage.
For the record, the relevant setting in WordPress.com is Options -> Privacy -> Blog visibility -> I would like my blog to appear in search engines like Google and Sphere, and in public listings around WordPress.com. Interestingly, although I’ve made that change, it’s not yet reflected in the robots.txt file. I wonder how long that’ll take?