PythonSeverity: MediumStatus: Published
Unbounded sitemap index following in AsyncUrlSeeder (cycles, depth, gzip bombs)
AsyncUrlSeeder follows sitemap indexes with no list of sitemaps it has already fetched, no depth limit, and no size limit on the sitemap body (CWE-409, CWE-674). A site the operator chooses to seed can serve a sitemap index that points back to itself, or a small gzip file that inflates to hundreds of megabytes. The seeding job then does not finish, and its memory grows until the process fails. This affects the library: AsyncUrlSeeder with the sitemap source, and DomainMapper with the sitemap source. The Docker server does not expose the seeder. An application that seeds URLs from sitemaps of domains it does not control can be made to hang and to run out of memory by one hostile site. No data is exposed and no code runs. Seeding only trusted domains is not affected. Affected versions are crawl4ai (pip) <= 0.9.5. Fixed by the maintainer in 0.9.6: fetched sitemap URLs are kept for the discovery run, index following stops after 5 levels, at most 10,000 sitemaps are fetched per run, and a sitemap body is refused once it passes 50 MB, including after a .gz sitemap inflates.
Timeline
- Found
- Reported via email
- Advisory published
- Entry last updated
Machine-readable: JSON · Markdown · ID security/python/unclecode__crawl4ai/2026-11-20-unbounded-sitemap-index-following
If this work is useful to you, you can support it on Buy Me a Coffee.