Đoàn Lê Minh Anh
For websites with hundreds of thousands or even millions of URLs, technical SEO audits cannot rely on opening pages one by one or running a small crawl. Businesses need a controlled data-gathering process, a way to prioritize issues, and a method for connecting findings with data from other sources. Lumar (formerly DeepCrawl) helps SEO teams audit websites at scale and monitor their technical health. This guide to Lumar introduces a practical approach for Enterprise SEO.
This article explains how to use Lumar in an Enterprise SEO environment, focusing on three key areas: setting up large crawls, analyzing server logs, and integrating data. The platform is most effective when it is part of a clear process with measurable objectives and designated owners for follow-up.
What is Lumar, and which businesses is it suited to?
Lumar is a website auditing platform for technical SEO, formerly known as DeepCrawl. It crawls accessible URLs according to your configuration, then surfaces signals such as response status, indexability, internal links, page structure, and technical issues. Depending on the configuration and subscription, businesses can organize projects, schedules, and reports to monitor website health over time.
Lumar is often a good fit for businesses with large websites, multiple page types, or multiple markets, where a small change can affect a substantial number of URLs. SEO, engineering, content, and product teams can share audit findings to identify issues and agree on what to address first. For a small website, a lightweight crawler may be enough. Choose an enterprise platform based on scale, complexity, and collaboration needs—not simply the number of features.
Keep in mind that crawl data reflects what the platform can see during a run with specific settings. It does not automatically prove that Google has indexed a page, nor does it replace expert judgment. Combine crawl findings with Search Console, analytics, server logs, and an understanding of your website architecture before drawing conclusions.
Plan a large crawl before you run it
Crawling a large website takes more preparation than entering a domain and clicking a button. First, define the business question the crawl needs to answer: Are many pages returning errors after a release? Do category pages have enough internal links? Are canonical tags correctly configured on international versions? Clear objectives help you choose the right scope, settings, and reports—and avoid generating more data than you can use.
Next, map the URL scope. Review your sitemap, robots.txt, URL patterns, and areas that do not need crawling, such as internal search results, filters that can generate an infinite number of URL combinations, or account pages. If you need to assess a specific area, such as a blog or product catalog, consider running a segmented crawl rather than including everything in one run. This makes it easier to isolate causes and compare changes across page groups.
Before crawling the entire website, test your settings on a small sample. Confirm that the platform can access the pages you need to check, detects the correct response statuses, and respects robots directives. For JavaScript-based websites, decide whether to crawl the initial HTML or the content rendered after the browser executes JavaScript. This choice affects runtime and what the crawl can detect. If the area you need to check is behind a login, verify that the supported access method meets your security requirements.
At scale, coordinate with your infrastructure team to set a reasonable crawl rate. A crawl that runs too quickly can put unwanted load on servers or distort results if the system responds slowly. Agree on when to run it, request limits, and how to stop the crawl if it causes an impact. Record key settings—including scope, user agent, crawl rate, and run date—so future comparisons are based on equivalent conditions.
Review results and prioritize technical issues
Once the crawl is complete, do not start by exporting every issue into one long list. Group findings by URL pattern, page type, and impact. For example, an issue affecting a product page template used by tens of thousands of URLs usually deserves more attention than an isolated problem on a low-priority page. On the other hand, a single URL with high conversion value or one serving as a campaign landing page may still need immediate attention.
A useful classification process considers four factors: the effect on crawlability and indexability, the number of affected pages, the business value of the page group, and confidence in the finding. Then identify the root cause, owner, priority, and criteria for confirming completion. This turns SEO reports into actionable work rather than just another data sheet.
Distinguish genuine technical errors from signals that need further investigation. For instance, a URL that is not indexed may be intentional if it is a filter page with no search value. But it may indicate a problem if an important category page is blocked or has an incorrect canonical tag. Compare crawls run with the same scope and settings to identify trends, and note major releases to help explain changes.
As part of this process, LilyTech can help businesses design a technical SEO audit structure, connect findings to business priorities, and coordinate marketing and development teams. The goal is to turn crawl data into an ordered action plan—not to run the platform regularly without completing the improvement cycle.
Analyze server logs to understand bot activity
A crawl shows what an auditing tool can find on your website; server logs show which requests actually reached the server during the recorded period. Combining these sources helps SEO teams understand which areas search bots visit, what responses they encounter, and whether crawl budget is being spent on low-value URLs. Log analysis is particularly useful for large websites, where a single crawl cannot fully represent bot behavior over time.
Before analyzing logs, clarify their scope, retention period, data format, and how bots are identified. Do not assume that every request with a name or user-agent string resembling a search bot is authentic; verify it using a method appropriate to your infrastructure and the provider's documentation. Also standardize time zones, normalize URLs, and remove irrelevant data to avoid inaccurate counts.
When reviewing reports, focus on specific questions: Do bots frequently visit parameterized URLs? Are important pages visited regularly? Which page groups account for 4xx or 5xx responses? Which crawled URLs should not be prioritized? Compare the findings with internal linking, sitemaps, and Lumar crawl data. If logs show bots frequently visiting an area that the crawl cannot reach through links, investigate where those URLs originate and how users navigate to them.
Logs contain sensitive operational data, so follow your business's security, access, and retention policies. Avoid uploading complete datasets to external systems before reviewing their data-processing terms. If logs are sent through an integration platform, confirm exactly which fields are shared, how the data is stored, and who can access it.
Integrate Lumar with other tools
The platform becomes more valuable when crawl findings are viewed alongside data from the other systems a business uses. Depending on the capabilities included in your subscription and your organization's configuration, you may be able to connect or compare data with Google Search Console, Google Analytics, log analysis platforms, task management systems, or an internal data warehouse. Before implementation, check the current product documentation to confirm available integration methods, access requirements, and any related limitations.
Each data source answers a different question. A crawl provides a technical view of the URLs it has collected; Search Console reflects search data and information related to Google; analytics shows how users interact with the site, subject to your measurement setup; and logs record requests reaching the server. When you connect these sources using normalized URLs or consistent identifiers, SEO teams can prioritize pages that have technical issues as well as high traffic or business value.
For effective integration, establish data conventions first: standardize domains, protocols, trailing slash rules, URL parameters, and duplicate page handling. Assign access by role, grant only the permissions needed, and check connections regularly. If you need to send data to a dashboard or data warehouse, define update frequency, how synchronization errors will be handled, and who is responsible for maintenance.
Finally, do not integrate systems just to add more charts. Choose a few specific use cases, such as finding pages that receive traffic but return errors, prioritizing important URLs that are not being crawled effectively, or monitoring the impact of each release. Start with a small page group or workflow, assess accuracy, and then expand. A phased approach reduces risk and helps teams trust the data.
Conclusion: Turn Lumar data into action
Lumar helps enterprise SEO teams audit large websites, monitor technical issues, and put findings in context alongside server logs and other data sources. To use it effectively, define your objectives before crawling, set an appropriate scope, control server load, prioritize by business impact, and establish a process for verifying fixes.
Every business has a different website architecture, data stack, and set of resources. If you need to build an Enterprise SEO process, assess integration options, or turn technical findings into an implementation plan, contact LilyTech for advice on a solution suited to your business goals and infrastructure.
