Technical SEO and Algorithmic Search

Python Log File Analysis: Writing Scripts to Track Googlebot Budgets and Isolate Internal Crawl Waste Patterns

Autor Kinetikatech Operations Team · Aktualizacja 2026-09-29

Server logs reveal what crawlers actually do, not what Search Console reports. Googlebot's crawl budget is wasted on low-value URLs, redirect chains, and error responses. This guide writes Python scripts to parse logs, identify waste, and optimize crawl allocation.

Kluczowe punkty

Parse log format. Apache and Nginx logs have standard formats; extract timestamp, URL, status, and user agent.
Filter for Googlebot. Use the verified Googlebot IP ranges and user agent strings.
Calculate crawl budget. Count requests per day and identify trends over time.
Find crawl waste. Identify 404s, redirect chains, and low-value URLs consuming budget.
Analyze crawl depth. Track how deep Googlebot crawls and where it stops.
Correlate with Search Console. Compare log data with GSC indexing reports.
Generate reports. Automate daily or weekly reports on crawl efficiency.
Act on findings. Fix 404s, reduce redirect chains, and block low-value URLs.

Architecture

Logs are parsed into structured records, filtered for Googlebot, and aggregated by URL pattern, status code, and time. Reports highlight waste and trends. Actions are derived from the analysis and tracked over time.

Production configuration

Parse and filter Googlebot logs python

import re, gzip, collections

LOG_PATTERN = re.compile(
    r'(\S+) - - \[(.+?)\] "(\S+) (\S+) (\S+)" (\d+) (\d+) "([^"]*)" "([^"]*)"'
)

def parse_googlebot_log(path):
    entries = []
    with open(path) as f:
        for line in f:
            m = LOG_PATTERN.match(line)
            if m and 'Googlebot' in m.group(9):
                entries.append({
                    'ip': m.group(1), 'time': m.group(2),
                    'method': m.group(3), 'url': m.group(4),
                    'status': int(m.group(6)), 'ua': m.group(9),
                })
    return entries

Analyze crawl waste python

def analyze_waste(entries):
    waste = collections.defaultdict(int)
    for e in entries:
        if e['status'] == 404:
            waste['404'] += 1
        elif e['status'] in (301, 302):
            waste['redirect'] += 1
        elif e['status'] >= 500:
            waste['server_error'] += 1
    total = len(entries)
    for k, v in waste.items():
        print(f'{k}: {v} ({v/total:.1%})')
    return waste

Generate crawl report python

def daily_report(entries):
    by_day = collections.defaultdict(int)
    for e in entries:
        day = e['time'].split(':')[0]
        by_day[day] += 1
    for day in sorted(by_day):
        print(f'{day}: {by_day[day]} requests')
    # top URLs
    urls = collections.Counter(e['url'] for e in entries)
    print('\nTop 10 URLs:')
    for url, count in urls.most_common(10):
        print(f'  {count:5d} {url}')

Edge cases from production

Log rotation can split sessions; ensure your parser handles rotated files.
Googlebot IPs change; verify against published ranges, not just user agent.
Crawl budget varies by site authority; compare against your own baseline.
CDN logs may not include origin requests; use origin logs for accuracy.

Częste pytania

What is crawl budget?

The number of URLs Googlebot will crawl on your site per day.

How do I reduce crawl waste?

Fix 404s, reduce redirects, and block low-value URLs with robots.txt.

Can I see Googlebot in real time?

Yes, by tailing server logs and filtering for Googlebot user agent.

How often should I analyze logs?

Weekly for trends, daily during site changes.

What is the biggest waste source?

Usually parameterized URLs and redirect chains.

Jak Kinetikatech pomaga w technical seo and algorithmic search

Tym zajmujemy się na co dzień: Crawl budget, rendering, and structured data engineered for search engines and AI answer systems. Nasza usługa Optymalizacja prędkości i Zarządzanie serwerami obejmuje to kompleksowo: ocenę, wdrożenie, monitoring 24/7 oraz reakcję awaryjną w 30 minut.

Porozmawiaj z inżynierem