There are many motives you could want to seek out every one of the URLs on an internet site, but your precise aim will decide Whatever you’re trying to find. As an example, you may want to:

Identify every indexed URL to investigate difficulties like cannibalization or index bloat
Accumulate recent and historic URLs Google has observed, especially for site migrations
Obtain all 404 URLs to Get better from article-migration problems
In Each and every scenario, only one Resource gained’t Provide you with everything you need. Sad to say, Google Research Console isn’t exhaustive, along with a “site:case in point.com” lookup is limited and tricky to extract knowledge from.

On this post, I’ll wander you through some applications to develop your URL checklist and before deduplicating the information using a spreadsheet or Jupyter Notebook, based upon your web site’s dimension.

Aged sitemaps and crawl exports
Should you’re searching for URLs that disappeared within the Dwell site not long ago, there’s a chance anyone with your team could possibly have saved a sitemap file or perhaps a crawl export ahead of the improvements had been built. Should you haven’t already, check for these data files; they're able to normally give what you need. But, in case you’re reading this, you most likely didn't get so Fortunate.

Archive.org
Archive.org
Archive.org is a useful tool for Website positioning duties, funded by donations. When you look for a website and choose the “URLs” alternative, you could access approximately ten,000 detailed URLs.

On the other hand, There are some constraints:

URL Restrict: You are able to only retrieve as many as web designer kuala lumpur ten,000 URLs, that is inadequate for larger sized websites.
Excellent: Many URLs could be malformed or reference source data files (e.g., pictures or scripts).
No export selection: There isn’t a crafted-in way to export the checklist.
To bypass The shortage of the export button, use a browser scraping plugin like Dataminer.io. Nonetheless, these constraints signify Archive.org may not offer a whole Resolution for larger sized sites. Also, Archive.org doesn’t show no matter if Google indexed a URL—however, if Archive.org found it, there’s a great likelihood Google did, way too.

Moz Professional
When you might usually use a website link index to seek out exterior web-sites linking to you personally, these applications also discover URLs on your site in the process.


How you can use it:
Export your inbound hyperlinks in Moz Pro to obtain a rapid and simple list of concentrate on URLs out of your web page. In case you’re dealing with a huge website, consider using the Moz API to export details past what’s workable in Excel or Google Sheets.

It’s crucial to Observe that Moz Professional doesn’t verify if URLs are indexed or learned by Google. Nevertheless, due to the fact most websites implement the exact same robots.txt guidelines to Moz’s bots because they do to Google’s, this technique generally operates very well as a proxy for Googlebot’s discoverability.

Google Lookup Console
Google Lookup Console presents several beneficial sources for constructing your listing of URLs.

One-way links reports:


Similar to Moz Pro, the Backlinks area delivers exportable lists of concentrate on URLs. Unfortunately, these exports are capped at 1,000 URLs Every. It is possible to implement filters for precise web pages, but due to the fact filters don’t utilize into the export, you might need to depend on browser scraping resources—limited to five hundred filtered URLs at any given time. Not excellent.

Effectiveness → Search engine results:


This export provides you with a summary of webpages receiving look for impressions. Though the export is proscribed, You should utilize Google Research Console API for more substantial datasets. In addition there are cost-free Google Sheets plugins that simplify pulling a lot more substantial information.

Indexing → Internet pages report:


This part supplies exports filtered by concern kind, though these are definitely also minimal in scope.

Google Analytics
Google Analytics
The Engagement → Webpages and Screens default report in GA4 is a wonderful supply for collecting URLs, which has a generous limit of a hundred,000 URLs.


Better still, you are able to utilize filters to produce different URL lists, proficiently surpassing the 100k limit. Such as, if you'd like to export only weblog URLs, abide by these measures:

Step 1: Increase a segment into the report

Stage two: Simply click “Produce a new phase.”


Stage three: Determine the segment that has a narrower URL sample, including URLs made up of /weblog/


Observe: URLs found in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they provide valuable insights.

Server log information
Server or CDN log documents are Potentially the ultimate Resource at your disposal. These logs seize an exhaustive list of every URL path queried by people, Googlebot, or other bots throughout the recorded time period.

Criteria:

Information measurement: Log documents can be huge, a great number of web pages only retain the last two weeks of data.
Complexity: Analyzing log files could be hard, but many tools are available to simplify the process.
Mix, and very good luck
When you finally’ve collected URLs from these resources, it’s time to mix them. If your web site is sufficiently small, use Excel or, for larger sized datasets, tools like Google Sheets or Jupyter Notebook. Ensure all URLs are regularly formatted, then deduplicate the checklist.

And voilà—you now have a comprehensive listing of recent, previous, and archived URLs. Excellent luck!