Audanbal
334 days ago in Features & Bugs
Tags: dolphin, security, search
Reactions: 7 comments •0 points•0 reports 

Robots.txt

I have created a robots.txt in root directory...

User-agent: *

Disallow: /

Isn't this supposed to tell any bot not to index anything from the site?

Google still indexing even for a year. Am I entitled to file a lawsuit against Google simply because they ignored robots.txt? Maybe robots.txt isn't enough? Maybe Dolphin was made to be "public" and ignores robots.txt? Whatever the case may be, it needs to be fixed. Wordpress has option if want your-site searchable or not. I think Dolphin should include this.

Plus

Comment•Report

Plussed by

See All ›

Comments

Threaded

Please login to post a comment.

Nathan Paton•334 days ago

You should be able to have it removed through a number of options. I recommend you read this page: http://www.google.com/support/webmasters/bin/answer.py?answer=164734

Permalink•Reply•Plus•0 pluses

Audanbal•334 days ago

Thank you, Nathan.

Obviously, robots.txt is a poor choice. I just created password-protect directory in home folder. Do I need to add in .htaccess in dolphin root as well?

Permalink•Reply•Plus•0 pluses

Nathan Paton•334 days ago•in reply to Audanbal

If you want to disallow all bots from indexing your site, you'd have to add the following to your installation's root directory:

User-agent: *
Disallow: /

I think the reason why you're having problems is because you used "User-Agent" instead of "User-agent." I don't know how bots interpret it, though.

If you want to make your site password-protected through your web server, you'll have to edit the root .htaccess file. I should warn you that some parts of Dolphin will see more

Permalink•Reply•Plus•0 pluses

Audanbal•334 days ago•in reply to Nathan Paton

Actually, it's User-agent: * in the robots.txt in dolphin root directory and untouched for a year. I must have gone dyslexic whilst typing in here. :oþ Thus, edited in the post. Anyway, I am still puzzled to why googlebot ignores this.

Permalink•Reply•Plus•0 pluses

dolphin_jay•333 days ago

Permalink•Reply•Plus•0 pluses

gameutopia•333 days ago

A while back I read a post by Matt Cutts from Google on his blog, I can't locate the exact link at the moment, but he claimed that technically if a site links to yours they can still include it in the index without actually crawling your site. Something about they can gather enough data based on the content, description, and your link on another site that links to your site, and use that info as the search result without touching your site.

Adding noindex is an option, preventing anyone from viewing see more

Permalink•Reply•Plus•0 pluses

Audanbal•331 days ago

By default, without robots.txt. Click the Dashboard and choose Privacy Settings. Here you can customise who has access to all of your data. I recommend changing your default privacy settings to, at least, members so that non-members cannot access your information. On your profile page, click the Edit button to customise the privacy settings on your profile.

What's the differences between Default and Public found in Edit Profile Privacy? So, choosing Members/Friends/Only Me... does this too prevent see more

Permalink•Reply•Plus•0 pluses