bingoss/owned-growth-engine
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Technical Audit records robots.txt and X-Robots-Tag blockers #5

New issue
Closed
opened 2026-05-28 18:14:30 +01:00 by bingoss · 1 comment
bingoss commented 2026-05-28 18:14:30 +01:00
Owner
Copy link

What to build

Type: AFK.

Extend Technical Audit evidence collection to check robots.txt and X-Robots-Tag signals. Findings should explain crawl/index blockers for affected URLs or site-wide scope and preserve the approval boundary by producing findings and handoff guidance only.

Acceptance criteria

  • robots.txt is fetched safely for site-level audits and recorded in stage diagnostics.
  • Disallow, missing sitemap, fetch failure, and malformed robots outcomes become safe diagnostics or Technical Findings where evidence supports them.
  • X-Robots-Tag noindex/nofollow headers are captured as normalized Technical Findings for affected URLs.
  • Tests are written first for robots blockers, unavailable robots, and X-Robots-Tag page findings.

Blocked by

None - can start immediately

## What to build Type: AFK. Extend Technical Audit evidence collection to check robots.txt and X-Robots-Tag signals. Findings should explain crawl/index blockers for affected URLs or site-wide scope and preserve the approval boundary by producing findings and handoff guidance only. ## Acceptance criteria - [ ] robots.txt is fetched safely for site-level audits and recorded in stage diagnostics. - [ ] Disallow, missing sitemap, fetch failure, and malformed robots outcomes become safe diagnostics or Technical Findings where evidence supports them. - [ ] X-Robots-Tag noindex/nofollow headers are captured as normalized Technical Findings for affected URLs. - [ ] Tests are written first for robots blockers, unavailable robots, and X-Robots-Tag page findings. ## Blocked by None - can start immediately
bingoss commented 2026-05-28 22:53:48 +01:00
Author
Owner
Copy link

Issue #5 completion audit

Requirements checked:

  • Site-level Technical Audit now safely fetches robots.txt and stores safe robots diagnostics in the run stage output summary.
  • robots.txt Disallow matches and missing Sitemap directives create normalized Technical Findings when evidence supports them; unavailable or malformed robots data stays as safe diagnostics.
  • X-Robots-Tag noindex/nofollow headers create normalized indexability Technical Findings for affected URLs.
  • No raw response bodies, secrets, website edits, CMS writes, deployments, or publishing actions are performed.
  • Tests were written first for robots blockers, unavailable robots diagnostics, and X-Robots-Tag page findings.

Verification on codex-server:

  • python3 -m unittest tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_records_robots_txt_blockers_and_missing_sitemap tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_records_unavailable_robots_txt_as_safe_diagnostic tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_records_x_robots_tag_page_findings -> OK
  • python3 -m unittest tests.test_site_crawler tests.test_technical_audit_worker -> 24 tests OK
Issue #5 completion audit Requirements checked: - Site-level Technical Audit now safely fetches robots.txt and stores safe robots diagnostics in the run stage output summary. - robots.txt Disallow matches and missing Sitemap directives create normalized Technical Findings when evidence supports them; unavailable or malformed robots data stays as safe diagnostics. - X-Robots-Tag noindex/nofollow headers create normalized indexability Technical Findings for affected URLs. - No raw response bodies, secrets, website edits, CMS writes, deployments, or publishing actions are performed. - Tests were written first for robots blockers, unavailable robots diagnostics, and X-Robots-Tag page findings. Verification on codex-server: - python3 -m unittest tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_records_robots_txt_blockers_and_missing_sitemap tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_records_unavailable_robots_txt_as_safe_diagnostic tests.test_technical_audit_worker.TechnicalAuditWorkerTests.test_technical_audit_worker_records_x_robots_tag_page_findings -> OK - python3 -m unittest tests.test_site_crawler tests.test_technical_audit_worker -> 24 tests OK
Commenting is not possible because the repository is archived.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
codex-token-write-test-1780004082
Temporary token write test; safe to delete.

No labels
codex-token-write-test-1780004082
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
bingoss/owned-growth-engine#5
No description provided.