Skip to content

    Wget Download Pro

    Download files and mirror sites with wget.

    Quick recipes

    # Idempotent download with retries
wget --tries=10 --retry-connrefused --timeout=15 --continue \
  https://example.com/file.tar.gz

# Mirror with globs (exclude heavy dirs)
wget --mirror --no-parent --reject "*.iso" --exclude-directories=/media \
  https://mirror.example.com/

    Production checklist

    • Use --continue and --timestamping for idempotency.
    • Set --tries and sensible --timeout values.
    • Validate artifacts with checksums after download.

    Mirror Entire Website

    Mirror HTML and dependencies for offline use; respect robots and server load.

    Warning: This can download massive amounts of data and may overwhelm servers. Always check the site's robots.txt and respect rate limits. Consider using --wait and --random-wait to be respectful. 
    wget --mirror \
  --convert-links \
  --adjust-extension \
  --page-requisites \
  --no-parent \
  https://example.com

    What it does: Downloads entire site, fixes links for local viewing, gets CSS/JS/images, and doesn't go up to parent directories

    Resume Interrupted Download

    Resume a partial download instead of fetching from byte zero.

    wget -c https://example.com/large-file.zip

    Retries: GNU wget: --tries=0 means retry until stopped; other builds may differ.

    Download from List of URLs

    Read URLs from a file, one per line.

    # Create urls.txt with one URL per line
wget -i urls.txt

# With custom output directory
wget -i urls.txt -P /downloads/

    Use case: Bulk pulls when you already have a URL list.

    Limit Download Speed

    Cap throughput so a download does not saturate the link.

    # Limit to 200 KB/s
wget --limit-rate=200k https://example.com/file.iso

# Limit to 1 MB/s
wget --limit-rate=1m https://example.com/file.iso

    Units: Use k for kilobytes/sec, m for megabytes/sec

    Background Download with Logging

    Detach and append progress to a log file.

    wget -b -o wget.log https://example.com/large-file.tar.gz

# Check progress anytime
tail -f wget.log

    Tip: Add -c for resume capability: wget -bc -o wget.log https://example.com/file