Web Design & Development

Robots.txt allows you to specify which pages should not be crawled. Pages that don’t get crawled can still rank for keywords and show up in search results. Robots.txt has been with us for over 14 years, but how many of us knew that in addition to the disallow directive there’s a no index directive that […]