robots.txt-regel-parser
Fetch robots.txt and test whether specific URLs are allowed or blocked for any user-agent, including wildcard and path rules. Use it to debug crawler access and verify disallow rules.
Tool-resultat genereras på engelska.
Relaterade sidor
Serverproblem? Hackad webbplats? Långsam sida?
Kritiska incidenter besvaras inom 30 minuter — dygnet runt, 7 dagar i veckan.