Đoàn Lê Minh Anh
OnCrawl is a technical SEO platform that helps businesses assess their websites from a search engine’s perspective, analyze server logs, and uncover opportunities to improve content. When a website has thousands of URLs, multiple page versions, or frequent updates, manual checks can easily miss issues that affect crawling and indexing. OnCrawl provides a more systematic way to find them.
This guide walks through a practical OnCrawl workflow: preparing a project, running a crawl, combining crawl data with log data, and using content reports. The goal is not just to read the metrics, but to prioritize work that can improve SEO performance.
What is OnCrawl, and who is it for?
OnCrawl is a technical SEO tool focused on crawling websites, analyzing page structure, and combining crawl data with server logs and other data sources. This enables users to compare how search engine bots discover a website with the structure the SEO team intends, rather than relying on a one-off audit.
The tool is particularly useful for ecommerce websites, news sites, platforms with many categories, and businesses changing their website structure. Smaller sites can also use it for regular audits, but the greatest benefits tend to emerge when a site has a large number of URLs, many filtered or paginated pages, or automatically generated content.
OnCrawl does not replace an SEO strategy, rank-tracking tools, or the process of evaluating content quality. Instead, it provides data that helps teams identify technical issues, understand bot behavior, and find underperforming page groups. To use it effectively, define your business questions before opening the reports: Which pages are not being crawled? Is crawl budget being wasted? Which content groups need improvement?
Prepare a project and crawl your website
Before creating a project, define the website scope, primary domain, preferred protocol, and areas to exclude. If your site has a staging version, subdomains, or login-protected areas, check these carefully to avoid including out-of-scope data in the results. Also agree on crawl speed limits with your technical team to reduce the risk of placing unnecessary load on the server.
- Create a project: Enter the domain and configure the basic settings to suit your audit objectives.
- Choose URL sources: You can start with an XML sitemap, a list of known URLs, or a starting point on the website. A sitemap is useful for providing canonical URLs, but it should not be treated as a complete list of every page that exists.
- Set the scope: Define inclusion and exclusion rules, such as URLs with filter parameters, internal search pages, or areas that do not need to be checked.
- Configure the crawl: Choose an appropriate speed, configure the bot type if needed, and check the robots.txt rules to understand which pages the crawler is allowed to access.
- Run the crawl and review the results: Once the crawl is complete, review the URL count, response statuses, connection errors, and completion rate before analyzing the data in depth.
For the first run, start with a manageable scope instead of crawling the entire website right away. This lets you test the configuration, detect unexpected URLs, and consult the technical team early if the crawl creates a significant load. Once you have confirmed that the data is reliable, you can expand the scope and schedule regular crawls to monitor changes.
Read crawl data to find technical SEO issues
After the crawl is complete, start with the overall picture and then examine individual URL groups. HTTP status metrics can help identify 4xx page errors, 5xx server errors, unnecessary redirects, and redirect chains. However, a URL that returns a status other than 200 is not automatically an error: check whether the page is intentionally redirected, removed, or blocked from access.
Next, check indexability and on-page signals such as canonical tags, robots directives, page titles, headings, and click depth. An important page that cannot be reached through internal links may be difficult for bots to discover consistently. Conversely, large numbers of parameter-based URLs with near-identical content can make the site structure more complex and consume crawl resources.
Classify findings by impact and scope. A canonical issue on a key revenue-generating page usually deserves higher priority than a warning on a few less important pages. For each issue, record the number of affected URLs, representative URL patterns, possible causes, and the team responsible for resolving it. This approach turns a report into an actionable task list rather than a collection of difficult-to-act-on alerts.
Analyze logs to understand bot behavior
Server logs record requests made to a website, including visits from Googlebot and other search engine bots. By combining log data with crawl data, OnCrawl can help answer important questions: Which URLs are bots visiting? Which pages receive few visits? Are many requests going to URLs that do not need to be indexed? Are bots discovering new content promptly?
Log analysis typically begins with preparing log files from the server or a storage system in a supported format. Make sure the logs include the requested URL, timestamp, response status, and user agent. Check the time zone, duplicate data, and date range before importing. If the logs cover only a short period or part of the server, the conclusions may not represent the entire website.
Once the data has been processed, compare the URLs in the logs with those found in the crawl. Important page groups that receive frequent bot visits are generally discovered regularly, but a high crawl rate does not automatically mean strong SEO performance. If bots spend many visits on filter pages, parameter-based URLs, or error pages while rarely visiting new product pages, this may signal a need to improve internal linking, control duplicate URLs, or update the sitemap.
Log analysis also needs to be interpreted in context. Bot traffic can fluctuate with seasonality, content changes, website updates, or server configuration. Do not conclude that crawl budget is being wasted based on a single metric. Combine log data with URL importance, indexation status, internal links, and business objectives.
Optimize content and turn data into a plan
Crawl data can help identify pages with missing titles, duplicate metadata, thin content, similar page structures, or limited accessibility. When analyzing the results, group pages by content type, category, URL pattern, or level of importance. Comparing similar groups is often more useful than reviewing URLs individually, because an issue in a page template can affect hundreds of URLs.
For each group, assess the relationship between content quality and bot accessibility. For example, a category with useful content but a deep position in the site structure may need more internal links. A set of product pages with similar descriptions may need more distinctive content, information that supports purchase decisions, or clearer guidance. The tool helps identify signals, but editors still need to assess search intent, accuracy, and the practical value for readers.
To make sure the findings lead to action, turn them into a task tracker that lists the issue, affected URLs or page patterns, priority, owner, deadline, and verification method. Once the technical or content team has completed the work, run another crawl over the same scope to compare results before and after. LilyTech can help businesses build a technical SEO process, analyze data, and connect findings to a content plan that fits available resources.
Conclusion: Use OnCrawl as part of an ongoing process
OnCrawl delivers the most value when used as part of a continuous improvement process: crawl the site to check its structure, analyze logs to understand bot activity, and then prioritize technical and content optimization based on impact. Choosing the right scope, checking data quality, and interpreting reports in the context of the website helps businesses avoid addressing alerts mechanically.
If your business needs help auditing its website, creating a technical SEO plan, or turning crawl and log data into a concrete list of actions, contact LilyTech for advice on a solution that fits your goals and team resources.
