- Aug 4, 2008
- Reaction score
Why does a .pdf for a website come up before any of the other pages of the same site? Any way to fix that?
robots.txt is a file that can be placed in the root of your website that search engine spiders look to in order to see if they have permission to crawl everything on the site. If you go to www. yoursite.com/robots.txt you can see your robots.txt file. I would say 90% of websites don't have them, which is fine. They are only used to exclude pages, so if there is nothing to exclude, then you don't need one.Okay, #4 has me worried. I really hope that it isn't my fault that the .pdf comes up first because I don't know how to fix it.
Can you expand on the robot.txt file explanation please?.