Document Actions
Spider-Bot, Spider Bot, does whatever a spider-bot is allowed to do according to your robots.txt file
Evidently, Social Edge is to Chinese search engine spider bots as David Hasselhoff is to Germany. The bots love us!
In wading through our recent slowdown, we found more than one cause for our woes. Turns out there were a couple of bots from search engines in China that were crawling our site, but they were hitting inaccessible pages, leading to, well, some huge error logs among other problems.
As a result, I've temporarily turned off the wiki and added a couple of exceptions to our robots.txt file. A little more testing and I'll turn the wiki back on, but the fact that it can be edited with a login was problematic.
In wading through our recent slowdown, we found more than one cause for our woes. Turns out there were a couple of bots from search engines in China that were crawling our site, but they were hitting inaccessible pages, leading to, well, some huge error logs among other problems.
As a result, I've temporarily turned off the wiki and added a couple of exceptions to our robots.txt file. A little more testing and I'll turn the wiki back on, but the fact that it can be edited with a login was problematic.
