Robots.txt Analyzer

Name: Robots.txt Analyzer
Price: 49 USD
Author: AAIO Inc

v1.0 documentation

Parses and validates robots.txt files. Checks for crawl issues, blocked resources, sitemap declarations, and common mistakes.

URL inputFile inputXLSX export

robots_txt_analyzer.py167 lines5 paramsPython 3.8+

Quick start

terminal

pip install -r requirements.txt

terminal

python robots_txt_analyzer.py --url https://example.com

terminal

python robots_txt_analyzer.py --file robots.txt

terminal

python robots_txt_analyzer.py --urls https://a.com https://b.com --output robots_audit.xlsx

Add --output report.xlsx to save results as a spreadsheet.

Parameters

Flag	Description
`--url`	Url
`--urls`	Urls. Multiple values allowed
`--file`	Local robots.txt file
`--test-path`	Test if a path is blocked
`--output`	Save as XLSX

help

python robots_txt_analyzer.py --help

Use cases

Technical audit

Pre-launch check

Ongoing monitoring

Run as part of a full site audit. Export issues to XLSX, prioritize by severity, and create a fix roadmap for the dev team.

Run before launching a new site or after a migration. Catch technical issues before Google crawls the new version.

Schedule regular checks and compare outputs over time. Catch regressions early.

Works well with

Combine with other tools for a complete workflow:

Dependencies

Requires: pandas, requests. All included in requirements.txt.

One-time payment. Lifetime access. No monthly fees.
Learn 25 tools and get 25% back. Earn from client work and get 50% back.

AAIO Inc — aaioinc.com/tools/robots_txt_analyzer/