Good questions.
Clear answers.
Practical answers about crawling, reports, schedules, accounts and technical SEO. Search a topic or browse by section.
Crawls & access
What Clearcrawl can reach, how crawling works, and what affects coverage.
How long does a website crawl take?
There is no guaranteed duration. It depends on how many URLs are in scope, how quickly the site responds, and whether JavaScript rendering is enabled. Clearcrawl paces requests for site safety. Follow the live progress in the workspace and avoid starting duplicate crawls just to check status.
Can Clearcrawl crawl JavaScript-rendered websites?
Yes. Enable JavaScript rendering when important page content appears only after the browser runs the site’s JavaScript. It can take longer and is limited to 500 URLs. If the content is already present in the normal page HTML, leave it off.
Are there limits on how many URLs Clearcrawl can crawl?
Standard crawl URL limits depend on the selected plan; check the plan comparison for the current limit. JavaScript-rendered crawls are separately capped at 500 URLs. Fetched resource URLs count toward the standard crawl limit. View plan limits
Why might a crawl take longer than expected?
Slow site responses, a large URL set, careful request pacing, and JavaScript rendering can all extend a crawl. Check the live progress and any crawl messages. If the site’s content does not need browser rendering, turning JavaScript rendering off can help.
What should I check if Clearcrawl cannot crawl my site?
Check that the site is publicly reachable and that robots.txt does not disallow the pages you want crawled. Clearcrawl does not sign in or bypass site rules. If access is allowed, check your host, CDN, or firewall logs for blocked requests, 403 or 429 responses, rate limits, or bot challenges. Cloudflare Security Events can help identify a rule that is stopping the crawler. See the crawler identity
Why didn’t Clearcrawl crawl every page?
Clearcrawl discovers pages through links and sitemaps, so unlinked pages may not be found. Robots.txt rules, URL exclusions, redirects, unavailable pages, and the selected plan’s crawl limit can also affect coverage. Review the crawl’s excluded and unreachable URL findings and make sure important pages are linked or listed in a sitemap.
Does Clearcrawl follow robots.txt?
Clearcrawl reads the site’s robots.txt and follows applicable crawl rules. A missing robots.txt file does not by itself prevent a crawl. If the file cannot be read or has a problem, review the crawler behaviour information and the report before drawing conclusions. Crawler behaviour
Does Crawl-delay in robots.txt slow Clearcrawl down?
Yes. Clearcrawl honors a valid Crawl-delay directive and waits that many seconds between page requests, so a larger value can make a crawl take much longer. Check the directive in both the ClearcrawlBot and * user-agent groups. Only lower or remove the delay if your site can safely handle requests arriving faster.
What is Clearcrawl’s crawler called, and how can I allow it?
Clearcrawl identifies itself as ClearcrawlBot/1.0 (+https://clearcrawl.app/crawl-behaviour/) when requesting robots.txt and pages. In robots.txt, check that rules for User-agent: ClearcrawlBot or User-agent: * do not disallow the pages you want crawled. If Cloudflare or another firewall blocks it, use your security logs to find the matching request and add a narrowly scoped exception if appropriate; do not disable protections globally. A user-agent is a useful matching signal, not proof of a request’s origin. Crawler behaviour
Can Clearcrawl crawl password-protected pages?
Clearcrawl does not sign in to a target website or use private credentials. It can check pages that are publicly reachable without logging in; do not send Clearcrawl a website password.
Does Clearcrawl change or fix my website?
No. Clearcrawl reports findings and suggested next steps; it does not edit or publish changes to your site. With JavaScript rendering enabled, the browser runs site scripts to display the page, so only enable that mode when needed.
Reports & progress
Understand scores, history, exports, shared reports, and AI assistant access.
What does my website health score mean?
The score is a summary of technical findings from the crawl. It is not a Google score, a ranking position, or a guarantee of search performance. Compare it across similar crawls to see how the checked site changes.
Does a higher health score guarantee better Google rankings?
No. Technical fixes can remove crawl, indexing, or page-quality problems, but rankings depend on many factors. Clearcrawl is not a rank tracker and cannot promise a ranking or traffic increase.
How do I track changes in my website over time?
Review saved crawls in Reports & history to compare scores and findings. You can also schedule a recurring weekly crawl to create a regular check-in.
What can I export to CSV?
CSV exports are available for the selected report table and reflect its current filters. They are data exports, not a complete image or PDF copy of every workspace screen.
What does a shared report link show?
A shared link shows a compact summary and saved history for one website. It does not include page-by-page audit rows, crawl controls, account details, or other websites.
Who can see a shared report?
Anyone who has the link can view its shared summary, so treat it as a public link rather than an access-controlled invitation. Revoke the link in report-sharing settings when you no longer want it to work.
Can I connect Claude or ChatGPT to my Clearcrawl reports?
Yes. Connect Claude or ChatGPT to Clearcrawl through your AI tool’s MCP or custom connector settings. In Clearcrawl, open Account settings → AI assistant access, copy the server address, add it in your AI tool, then sign in to Clearcrawl and approve the connection. Setup menus can vary between clients and workspaces. No API key is shared.
What can a connected AI assistant see or change?
It can query sites, completed report summaries and findings for the Clearcrawl account you approved. It cannot start crawls or change or delete data. A separate optional permission allows affected URLs for one finding, up to 500 results per finding; query strings and fragments are removed. You can revoke a connection at any time.
Is MCP server access included in every plan?
Yes. Solo, Pro and Max plans include MCP access. Your selected plan’s website and crawl limits still apply. View plans and pricing
Schedules & account
Recurring crawls, trial billing, account details, and workspace preferences.
Can I schedule recurring crawls?
Yes. Clearcrawl supports weekly schedules. Set the preferred day, time and time zone in Scheduled crawls; each scheduled run uses the account’s crawl allowance and standard crawl settings.
What happens when my 7-day free trial ends?
A payment card is required to start the trial. At the end of seven days, the selected plan and billing interval are charged automatically unless you cancel before the trial ends.
How do I change my account email?
Open Account settings and manage addresses in Profile. Verify a new address before making it primary. This updates the same account, so saved websites and reports stay with it.
Does Clearcrawl have dark mode?
Dark mode is available in the signed-in workspace. The preference is saved in that browser; the public marketing pages remain in their standard light presentation.
SEO help
Prioritise fixes and find practical guidance for technical SEO.
Which SEO issues should I fix first?
Start with critical and high-priority findings, then consider how many important pages are affected and whether the issue is intentional. Clearcrawl’s explanations help you decide what applies to your site; re-crawl after making a change to verify it.
Where can I learn more about technical SEO?
The Clearcrawl Academy has practical guides on crawling, indexing, technical SEO, on-page checks, site structure and more. Explore the Academy
Is Clearcrawl a replacement for Google Search Console?
No. Clearcrawl audits technical issues it can observe while crawling a site. Search Console provides Google-specific search performance and indexing data for a verified property. The tools answer different questions and can be used together.