bingoss/owned-growth-engine
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Technical Audit supports sitemap indexes and bounded discovery #4

New issue
Closed
opened 2026-05-28 18:14:28 +01:00 by bingoss · 1 comment
bingoss commented 2026-05-28 18:14:28 +01:00
Owner
Copy link

What to build

Type: AFK.

Extend site-level Technical Audit discovery so sitemap indexes are parsed safely, bounded by the configured crawl limit, and reflected in stage diagnostics. This should preserve same-host filtering, public URL checks, private path rejection, and fallback behavior already established by the current Technical Audit foundation.

Acceptance criteria

  • Sitemap index files are parsed to discover child sitemap URLs and page URLs within the configured crawl limit.
  • Off-host, private, malformed, or excessive URLs are skipped with safe diagnostics.
  • Priority URL fallback still works when the sitemap index cannot be fetched or parsed.
  • Tests are written first for sitemap index success, malformed index, off-host child sitemap, and fallback behavior.

Blocked by

None - can start immediately

## What to build Type: AFK. Extend site-level Technical Audit discovery so sitemap indexes are parsed safely, bounded by the configured crawl limit, and reflected in stage diagnostics. This should preserve same-host filtering, public URL checks, private path rejection, and fallback behavior already established by the current Technical Audit foundation. ## Acceptance criteria - [ ] Sitemap index files are parsed to discover child sitemap URLs and page URLs within the configured crawl limit. - [ ] Off-host, private, malformed, or excessive URLs are skipped with safe diagnostics. - [ ] Priority URL fallback still works when the sitemap index cannot be fetched or parsed. - [ ] Tests are written first for sitemap index success, malformed index, off-host child sitemap, and fallback behavior. ## Blocked by None - can start immediately
bingoss commented 2026-05-28 22:49:33 +01:00
Author
Owner
Copy link

Issue #4 completion audit

Requirements checked:

  • Sitemap indexes are parsed recursively and child sitemap page URLs are discovered within the configured crawl limit.
  • Off-host, private/malformed, duplicate, and excessive sitemap URLs are skipped with safe diagnostics in the Technical Audit stage output summary.
  • Priority URL fallback still works when the sitemap index cannot be parsed or no sitemap URLs are discovered.
  • Tests were written first for sitemap-index success, off-host child sitemap diagnostics, malformed index fallback, and existing unavailable-sitemap fallback behavior.

Verification on codex-server:

  • python3 -m unittest tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_discovers_urls_from_sitemap_index_with_crawl_limit tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_skips_off_host_child_sitemap_with_diagnostics tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_falls_back_when_sitemap_index_is_malformed -> OK
  • python3 -m unittest tests.test_site_crawler tests.test_technical_audit_worker -> 21 tests OK
Issue #4 completion audit Requirements checked: - Sitemap indexes are parsed recursively and child sitemap page URLs are discovered within the configured crawl limit. - Off-host, private/malformed, duplicate, and excessive sitemap URLs are skipped with safe diagnostics in the Technical Audit stage output summary. - Priority URL fallback still works when the sitemap index cannot be parsed or no sitemap URLs are discovered. - Tests were written first for sitemap-index success, off-host child sitemap diagnostics, malformed index fallback, and existing unavailable-sitemap fallback behavior. Verification on codex-server: - python3 -m unittest tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_discovers_urls_from_sitemap_index_with_crawl_limit tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_skips_off_host_child_sitemap_with_diagnostics tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_falls_back_when_sitemap_index_is_malformed -> OK - python3 -m unittest tests.test_site_crawler tests.test_technical_audit_worker -> 21 tests OK
Commenting is not possible because the repository is archived.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
codex-token-write-test-1780004082
Temporary token write test; safe to delete.

No labels
codex-token-write-test-1780004082
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
bingoss/owned-growth-engine#4
No description provided.