Quotempole Logo
Share Your Intelligence! Collect your own quotes, create custom quotelists and more.
What is Quotempole?down
Why Use Quotempole?down
How Can I Capture Quotes?down

Quote

quote icon But robots.txt is not a legal document — and 30 years after its creation, it still relies on the good will of all parties involved. Disallowing a bot on your robots.txt page is like putting up a “No Girls Allowed” sign on your treehouse — it sends a message, but it’s not going to stand up in court. Any crawler that wants to ignore robots.txt can simply do so, with little fear of repercussions.
Share

Sign In to comment on this quote

Extra Quote Detail

comment
Comments
reference
Reference
tag icon
Tags
url
URL
quoteList icon
Quotelists