01
Choose the site to crawl
Add a website to a Clearcrawl account to start a crawl. The site’s own crawler rules still apply.
Crawler behaviour / Clearcrawl
A crawler that makes its rules clear.
Clearcrawl is built to inspect the websites you choose without adding mystery to the process. It identifies itself, checks site-owner instructions, moves at a measured pace and leaves useful findings in your reports.
Identifies clearlyOne stable, public user-agent
Follows site rulesrobots.txt is fetched and honored
Adapts to serversPaces requests and backs off when needed
The short version
Three rules shape every crawl.
Clearcrawl is a practical tool for the people responsible for a website. That means showing our work and respecting the boundaries around it.
02
Robots.txt comes first
The crawler fetches robots.txt using the same Clearcrawl identity used for page requests, then honors the instructions it finds.
03
Response-aware pacing
A crawl starts with one connection and can ramp up to four for larger audits when responses stay healthy. Slow, throttled or failed responses make it slow down and return to one.
A crawl, explained
From website to useful report.
Discovery is more than following links. Clearcrawl uses the site’s own signals alongside the page journey, then keeps the result visible in the report. Requests start at one connection and can increase to four when the server responds well.
01
Add a website
Choose the website you want to inspect in Clearcrawl and start the crawl.
02
Read the site rules
Clearcrawl fetches robots.txt and applies the rules for its branded user-agent.
03
Reconcile discovery
Declared sitemaps and the conventional sitemap.xml location help reconcile what the site says exists.
04
Pace the requests
Requests share one crawl-wide pacing schedule. Clearcrawl starts with one connection and may use up to four when server responses stay healthy, backing off when they slow or signal throttling.
05
Review the report
Findings, redirects, exclusions and other results are organised for review in Clearcrawl reports.
Site rules are the boundary
Crawl the sites you choose.
Add a website to Clearcrawl and review its technical findings. The important boundary is the site’s own crawler policy: robots.txt instructions still apply to every crawl.
Robots.txt is fetched and honored before page checks begin.
Website added / site rules
Adding a website starts a crawl in your workspace.
Review its technical findings in a clear report.
Adding a website does not bypass robots.txt.
Site-owner instructions still apply to the crawler.
robots.txtrules
sitemap.xmldiscovery
Same-site URLs reconciled for review
Discovery signals
Sitemaps help us compare the map with the journey.
Clearcrawl checks sitemap files declared by robots.txt as well as the conventional sitemap.xml location. Sitemap indexes can be followed, and same-site listed URLs can be queued for normal crawler validation.
A URL can appear in a sitemap without being linked in the page graph. Bringing both signals together helps reports distinguish a missing link from a redirect, a robots block or another actionable result.
See how findings are reviewedWhat you get back
Useful detail, without a request log to decode.
Crawl findings can be reviewed in Clearcrawl reports. Look at the affected pages, see how discovery and access affected the result, and use the plain-English context to decide what deserves attention.
- Review crawl findings by issue and page
- See redirects and other navigation outcomes in context
- Recheck changes and compare your progress
Crawl behaviourharborandpine.co
ReviewedAccess signals reconciled
Robots rules, sitemap entries and page navigation are ready to review.
robots.txtHonored
Sitemap entries18 reviewed
Redirects2 in context
Report ready to reviewUpdated just now
Clearer checks start here
See what your website needs next.
Start with a 7-day free trial. A card is required, and your selected plan will be billed automatically after the trial unless you cancel before it ends.