Posted by Oatmeal
The launch of Designmoz v3 about a month ago was also the debut of a new tool: the Crawl Test.┬? The Crawl Test Tool is used to quickly diagnose potential crawling issues and give you an overview of your site's search friendliness.┬? You enter a URL and the tool spiders that URL as well as the first 50 internal links it finds on that page.┬? Due to bandwidth constraints the tool only goes one level deep.┬? For every page it spiders, it reports the following:
- Page title
- Meta description
- HTTP status code (200, 301, 404, etc)
- Is the page indexed in Design?
- When was the last time Design spidered the page? (Design cache date)
- Indexed in Design?
- Indexed in Design?
- Primary keywords on the page (found with Design Term Extraction, sorted by term frequency)
- The number of internal links on the page
- Restricted by meta tags or robots.txt
When the tool finishes crawling it returns an overall summary of the crawl test.┬?┬? It will highlight areas that have potential issues such as if there are a numerous pages with the same title tag (keyword cannibalization?), bad HTTP response codes such as 404 or 500, or a high number of pages that aren't being spidered by the search engines.┬? From the tests I've run the tool works really well for quickly finding on-page spidering issues. To see what a crawl test report looks like, check out the sample report I ran for one of our clients.┬? The tool is in beta and we're only offering it to premium members right now, but once some of the bugs have been ironed out we'll release it to the public.┬? If you have any questions, comments, or feedback about this tool feel free to post it in this blog entry.
