Programming

Python Download Large Files with Requests: Memory-Efficient Streaming

Learn how to download large files (>1GB) in Python using Requests library with proper streaming techniques to avoid memory overload. Complete implementation examples and best practices.

1 answer• 1 view

How can I download large files (greater than 1 GB) in Python using the Requests library without loading the entire file into memory? My current implementation using iter_content() still seems to load the response before saving it to file. What’s the correct approach for streaming large file downloads with Requests?

Downloading large files (greater than 1 GB) in Python using the Requests library requires proper streaming implementation to avoid memory overload. The key is using stream=True parameter with iter_content() method, ensuring chunks are written directly to disk instead of loading the entire response into memory. Many developers encounter this issue because they either forget to set streaming properly or call methods that trigger full response loading before the download begins.

Contents

Understanding Memory Issues with Large File Downloads in Python

When you download large files using Python’s Requests library without proper streaming, the entire response loads into memory at once. This creates significant problems for files larger than your available RAM, potentially causing crashes or system instability. The default behavior of Requests is to buffer the entire response content before making it available, which means even if you’re trying to stream, the download still consumes memory proportional to the file size.

The core issue lies in how Requests handles responses. When you make a request without stream=True, the library automatically downloads the entire response content and stores it in memory. This works fine for small files like JSON responses or HTML pages, but becomes problematic when dealing with multi-gigabyte files. Many developers mistakenly believe that using iter_content() automatically enables streaming, but without stream=True, the method simply iterates over content that’s already fully loaded into memory.

Memory consumption patterns reveal why this approach fails. For a 2GB file, your Python process would need approximately 2GB of RAM just to hold the response, plus additional memory for your application logic. This doesn’t account for system overhead, Python’s memory management overhead, or other running processes. In practice, you’d likely need at least 2.5-3 times the file size in available memory to safely complete the download without running into memory issues.

Understanding Requests’ response handling is crucial. The library provides different attributes for accessing response content:

  • response.content - Returns the entire response body as bytes (loads everything into memory)
  • response.text - Decodes the content as text (also loads everything)
  • response.iter_content() - Returns an iterator over response data (requires streaming)

Without stream=True, even iter_content() operates on fully loaded content, defeating the purpose of streaming. This distinction explains why many developers experience memory issues despite using what they believe are streaming methods.

Correct Streaming Approach with Requests Library

The correct approach for streaming large file downloads with Python’s Requests library involves two critical components: setting stream=True in your request and using the iter_content() method properly. This combination ensures that the response is processed as a stream of chunks rather than loading everything into memory at once.

When you set stream=True, Requests sends the request with the Connection: keep-alive header and doesn’t immediately download the response body. Instead, it waits for you to explicitly read from the response. This allows you to process the data incrementally, writing each chunk to disk as it arrives rather than storing it in memory.

The iter_content() method is designed specifically for this purpose. It returns an iterator that yields chunks of the response content. By default, it yields chunks of 8192 bytes (8KB), though this can be customized. When used with stream=True, each chunk represents a piece of the actual HTTP response data that’s being downloaded over the network, not data that’s already been fully loaded into memory.

Here’s the fundamental pattern for streaming downloads:

python
import requests

url = 'https://example.com/large-file.zip'
response = requests.get(url, stream=True)
response.raise_for_status() # Check for HTTP errors

with open('large-file.zip', 'wb') as f:
 for chunk in response.iter_content(chunk_size=8192):
 if chunk: # Filter out keep-alive chunks
 f.write(chunk)

Key behaviors to understand:

  • The stream=True parameter must be set in the initial request
  • iter_content() only works with streaming responses
  • Chunks are received as they’re downloaded, not all at once
  • Memory usage remains constant regardless of file size
  • The download can be interrupted and resumed (with proper implementation)

One common misconception is that iter_content() alone enables streaming. Without stream=True, the method simply iterates over content that’s already been fully loaded into memory, defeating the purpose. Both components—stream=True and iter_content()—are essential for proper streaming.

Another important consideration is the chunk size. While 8KB is the default, you may want to adjust this based on your specific use case. Larger chunks (64KB-1MB) can improve performance by reducing I/O operations but use slightly more memory. Smaller chunks (4KB-8KB) minimize memory usage but may increase processing overhead. The optimal chunk size depends on your specific hardware and network conditions.

Implementation Examples for Large File Downloads

Let’s explore practical implementations for streaming large file downloads with Python’s Requests library. These examples demonstrate proper memory management, error handling, and various optimization techniques suitable for different scenarios.

Basic Streaming Download

The most fundamental implementation follows the pattern we discussed earlier. Here’s a complete, production-ready example:

python
import requests
import os

def download_large_file(url, destination_path):
 """
 Download a large file from a URL with streaming support.
 
 Args:
 url (str): URL of the file to download
 destination_path (str): Local path to save the file
 """
 try:
 # Create directory if it doesn't exist
 os.makedirs(os.path.dirname(destination_path), exist_ok=True)
 
 # Make the request with streaming enabled
 with requests.get(url, stream=True, timeout=30) as response:
 response.raise_for_status() # Raise an exception for bad status codes
 
 # Get the total file size for progress tracking
 total_size = int(response.headers.get('content-length', 0))
 
 # Open the file in write mode
 with open(destination_path, 'wb') as file:
 downloaded = 0
 
 # Iterate over response chunks
 for chunk in response.iter_content(chunk_size=8192):
 if chunk: # Filter out keep-alive chunks
 file.write(chunk)
 downloaded += len(chunk)
 
 # Optional: Print progress
 if total_size > 0:
 progress = (downloaded / total_size) * 100
 print(f"\rDownloading: {progress:.1f}%", end='')
 
 print(f"\nDownload completed successfully: {destination_path}")
 return True
 
 except requests.exceptions.RequestException as e:
 print(f"\nDownload failed: {e}")
 # Clean up partially downloaded file
 if os.path.exists(destination_path):
 os.remove(destination_path)
 return False

Advanced Implementation with Progress Tracking

For better user experience, you can add progress tracking with a more sophisticated progress bar:

python
import requests
import os
import sys
from tqdm import tqdm

def download_with_progress(url, destination_path):
 """
 Download a large file with a progress bar.
 
 Args:
 url (str): URL of the file to download
 destination_path (str): Local path to save the file
 """
 try:
 os.makedirs(os.path.dirname(destination_path), exist_ok=True)
 
 # Get file size first
 head_response = requests.head(url, timeout=10)
 head_response.raise_for_status()
 total_size = int(head_response.headers.get('content-length', 0))
 
 # Start the download
 with requests.get(url, stream=True, timeout=30) as response:
 response.raise_for_status()
 
 with open(destination_path, 'wb') as file, tqdm(
 total=total_size,
 unit='B',
 unit_scale=True,
 unit_divisor=1024,
 desc=os.path.basename(destination_path),
 ascii=True,
 file=sys.stdout
 ) as progress_bar:
 
 for chunk in response.iter_content(chunk_size=8192):
 if chunk:
 file.write(chunk)
 progress_bar.update(len(chunk))
 
 return True
 
 except requests.exceptions.RequestException as e:
 print(f"\nDownload failed: {e}")
 if os.path.exists(destination_path):
 os.remove(destination_path)
 return False

Using shutil for More Efficient File Writing

For potentially better performance, especially with larger chunks, you can use shutil.copyfileobj() which is optimized for copying file-like objects:

python
import requests
import shutil
import os

def download_with_shutil(url, destination_path, chunk_size=1024*1024):
 """
 Download a large file using shutil.copyfileobj for better performance.
 
 Args:
 url (str): URL of the file to download
 destination_path (str): Local path to save the file
 chunk_size (int): Size of chunks to download (default: 1MB)
 """
 try:
 os.makedirs(os.path.dirname(destination_path), exist_ok=True)
 
 # Make request with streaming
 response = requests.get(url, stream=True, timeout=30)
 response.raise_for_status()
 
 # Open the response and destination file
 with open(destination_path, 'wb') as out_file:
 # Use shutil.copyfileobj for efficient copying
 shutil.copyfileobj(response.raw, out_file, length=chunk_size)
 
 return True
 
 except requests.exceptions.RequestException as e:
 print(f"\nDownload failed: {e}")
 if os.path.exists(destination_path):
 os.remove(destination_path)
 return False

Download with Resume Capability

For unreliable connections, implementing resume functionality can be valuable:

python
import requests
import os

def download_with_resume(url, destination_path, chunk_size=8192):
 """
 Download a large file with resume capability.
 
 Args:
 url (str): URL of the file to download
 destination_path (str): Local path to save the file
 chunk_size (int): Size of chunks to download
 """
 try:
 os.makedirs(os.path.dirname(destination_path), exist_ok=True)
 
 # Check if partial file exists
 downloaded_size = 0
 if os.path.exists(destination_path):
 downloaded_size = os.path.getsize(destination_path)
 
 # Make request with range header if partial file exists
 headers = {}
 if downloaded_size > 0:
 headers['Range'] = f'bytes={downloaded_size}-'
 
 with requests.get(url, headers=headers, stream=True, timeout=30) as response:
 response.raise_for_status()
 
 # Check if server supports range requests
 if downloaded_size > 0 and response.status_code != 206:
 print("Server doesn't support resume. Starting download from beginning.")
 downloaded_size = 0
 os.remove(destination_path)
 
 # Open file in append mode if resuming
 mode = 'ab' if downloaded_size > 0 else 'wb'
 
 with open(destination_path, mode) as file:
 for chunk in response.iter_content(chunk_size=chunk_size):
 if chunk:
 file.write(chunk)
 
 print(f"Download completed: {destination_path}")
 return True
 
 except requests.exceptions.RequestException as e:
 print(f"\nDownload failed: {e}")
 return False

These implementations provide different approaches to downloading large files with Python Requests, each optimized for different scenarios. The key takeaway across all examples is the consistent use of stream=True and proper chunk-based processing to avoid memory overload.

Best Practices and Error Handling

Implementing robust error handling and following best practices is crucial when downloading large files. These guidelines will help ensure your downloads are reliable, efficient, and handle edge cases gracefully.

Essential Error Handling Strategies

When working with large file downloads, network interruptions, server errors, and disk space issues can occur at any time. Comprehensive error handling is non-negotiable for production code. Here are the key strategies:

HTTP Status Code Checking
Always verify the HTTP status code before processing the response. The raise_for_status() method is your first line of defense:

python
response = requests.get(url, stream=True)
response.raise_for_status() # Raises HTTPError for bad responses (4xx or 5xx)

Timeout Configuration
Network connections can hang indefinitely. Set reasonable timeouts for both the initial connection and the overall request:

python
response = requests.get(url, stream=True, timeout=(3.05, 27)) # (connect, read) timeouts

Memory Monitoring
For extremely large files, monitor memory usage to prevent system overload:

python
import psutil

def monitor_memory():
 process = psutil.Process()
 memory_info = process.memory_info()
 return memory_info.rss / (1024 * 1024) # Return memory in MB

Chunk Processing Exceptions
Handle exceptions that might occur during chunk processing:

python
try:
 for chunk in response.iter_content(chunk_size=8192):
 if chunk:
 f.write(chunk)
except (IOError, OSError) as e:
 print(f"File write error: {e}")
 raise
except requests.exceptions.ChunkedEncodingError:
 print("Server terminated connection unexpectedly")
 raise

Connection Management Best Practices

Proper connection management prevents resource leaks and ensures reliable downloads:

Use Context Managers
Always use with statements to ensure connections are properly closed:

python
with requests.get(url, stream=True) as response:
 response.raise_for_status()
 with open('file.zip', 'wb') as f:
 for chunk in response.iter_content(chunk_size=8192):
 if chunk:
 f.write(chunk)

Connection Pooling
For multiple downloads, use a session object to benefit from connection pooling:

python
with requests.Session() as session:
 response = session.get(url, stream=True)
 # Process the response

Rate Limiting
Implement rate limiting to avoid overwhelming servers or getting blocked:

python
import time

def download_with_rate_limit(url, destination_path, max_requests_per_minute=60):
 start_time = time.time()
 request_count = 0
 
 while True:
 try:
 response = requests.get(url, stream=True)
 response.raise_for_status()
 
 # Process the download
 with open(destination_path, 'wb') as f:
 for chunk in response.iter_content(chunk_size=8192):
 if chunk:
 f.write(chunk)
 
 break
 
 except requests.exceptions.RequestException as e:
 request_count += 1
 elapsed = time.time() - start_time
 
 if elapsed < 60 and request_count >= max_requests_per_minute:
 sleep_time = 60 - elapsed
 print(f"Rate limiting: waiting {sleep_time:.1f} seconds")
 time.sleep(sleep_time)
 start_time = time.time()
 request_count = 0
 else:
 print(f"Request failed: {e}")
 time.sleep(5) # Wait before retry

Disk Space and File Handling

Large downloads can fail due to insufficient disk space. Implement checks before starting:

python
import shutil

def check_disk_space(required_space_gb, path='.'):
 """Check if there's enough disk space for the download"""
 total, used, free = shutil.disk_usage(path)
 free_gb = free / (1024 ** 3)
 return free_gb >= required_space_gb

# Usage
if not check_disk_space(5): # Need 5GB
 raise IOError("Insufficient disk space for download")

Handle partial file cleanup when downloads fail:

python
def safe_download(url, destination_path):
 try:
 # Download code here
 pass
 except Exception as e:
 # Clean up partial file
 if os.path.exists(destination_path):
 try:
 os.remove(destination_path)
 except OSError:
 pass
 raise e

Logging and Monitoring

For production downloads, implement proper logging:

python
import logging

logging.basicConfig(
 level=logging.INFO,
 format='%(asctime)s - %(levelname)s - %(message)s',
 handlers=[
 logging.FileHandler('download.log'),
 logging.StreamHandler()
 ]
)

def download_with_logging(url, destination_path):
 try:
 logging.info(f"Starting download: {url}")
 # Download code here
 logging.info(f"Download completed: {destination_path}")
 except Exception as e:
 logging.error(f"Download failed: {e}", exc_info=True)
 raise

Security Considerations

When downloading files from unknown sources, implement security checks:

python
import hashlib

def verify_download_integrity(file_path, expected_hash):
 """Verify file integrity using SHA-256 hash"""
 sha256_hash = hashlib.sha256()
 with open(file_path, "rb") as f:
 for chunk in iter(lambda: f.read(4096), b""):
 sha256_hash.update(chunk)
 return sha256_hash.hexdigest() == expected_hash

Performance Optimization

For optimal performance, consider these strategies:

python
# Optimize chunk size based on file size
def get_optimal_chunk_size(file_size_mb):
 """Determine optimal chunk size based on file size"""
 if file_size_mb < 10: # Small files
 return 8192
 elif file_size_mb < 100: # Medium files
 return 65536
 else: # Large files
 return 1024 * 1024 # 1MB chunks

# Use session with retry logic
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def create_session_with_retry(retries=3, backoff_factor=0.3):
 session = requests.Session()
 retry = Retry(
 total=retries,
 read=retries,
 connect=retries,
 backoff_factor=backoff_factor,
 status_forcelist=(500, 502, 504)
 )
 adapter = HTTPAdapter(max_retries=retry)
 session.mount('http://', adapter)
 session.mount('https://', adapter)
 return session

By implementing these best practices, you’ll create robust, reliable download functionality that handles edge cases gracefully while maintaining good performance and security.

Alternative Approaches and Performance Optimization

While the Requests library with streaming is the standard approach for Python file downloads, several alternatives and optimizations can improve performance, reliability, or functionality depending on your specific use case. Let’s explore these options.

Alternative Libraries for File Downloads

aiohttp for Asynchronous Downloads
For high-performance downloads, especially when handling multiple files simultaneously, aiohttp provides an asynchronous alternative:

python
import aiohttp
import asyncio

async def download_async(url, destination_path):
 async with aiohttp.ClientSession() as session:
 async with session.get(url) as response:
 response.raise_for_status()
 with open(destination_path, 'wb') as f:
 async for chunk in response.content.iter_chunked(8192):
 f.write(chunk)

# Usage
asyncio.run(download_async('https://example.com/large-file.zip', 'download.zip'))

urllib3 for Lower-Level Control
For more control over connection management, urllib3 (which Requests uses internally) can be used directly:

python
import urllib3

def download_with_urllib3(url, destination_path):
 http = urllib3.PoolManager()
 response = http.request('GET', url, preload_content=False)
 
 with open(destination_path, 'wb') as f:
 while True:
 chunk = response.read(8192)
 if not chunk:
 break
 f.write(chunk)
 
 response.release_conn()

tqdm for Progress Bars
The tqdm library provides excellent progress visualization:

python
from tqdm import tqdm
import requests

def download_with_tqdm(url, destination_path):
 response = requests.get(url, stream=True)
 total_size = int(response.headers.get('content-length', 0))
 
 with open(destination_path, 'wb') as f, tqdm(
 total=total_size,
 unit='B',
 unit_scale=True,
 unit_divisor=1024,
 desc=os.path.basename(destination_path)
 ) as pbar:
 for chunk in response.iter_content(chunk_size=8192):
 if chunk:
 f.write(chunk)
 pbar.update(len(chunk))

Performance Optimization Techniques

Connection Pooling
For multiple downloads, reuse connections with a session:

python
session = requests.Session()
response = session.get(url, stream=True)
# Process download
# Can reuse session for more requests

Chunk Size Optimization
Adjust chunk size based on file size and network conditions:

python
def get_optimal_chunk_size(file_size_mb):
 """Determine optimal chunk size based on file size"""
 if file_size_mb < 10:
 return 8192
 elif file_size_mb < 100:
 return 65536
 else:
 return 1024 * 1024 # 1MB

# Usage
chunk_size = get_optimal_chunk_size(file_size_mb)
for chunk in response.iter_content(chunk_size=chunk_size):
 # Process chunk

Parallel Downloads for Large Files
For extremely large files, download in parallel segments:

python
import requests
import threading
import os

def download_segment(url, start_byte, end_byte, file_path, buffer):
 headers = {'Range': f'bytes={start_byte}-{end_byte}'}
 response = requests.get(url, headers=headers, stream=True)
 
 with open(file_path, 'rb+') as f:
 f.seek(start_byte)
 for chunk in response.iter_content(chunk_size=8192):
 if chunk:
 f.write(chunk)
 buffer['progress'] = min(100, (buffer['progress'] + len(chunk)) // total_size * 100)

def parallel_download(url, destination_path, num_segments=4):
 # Get file size
 head = requests.head(url)
 total_size = int(head.headers.get('content-length', 0))
 
 # Calculate segment boundaries
 segment_size = total_size // num_segments
 segments = []
 
 for i in range(num_segments):
 start = i * segment_size
 end = start + segment_size - 1 if i < num_segments - 1 else total_size - 1
 segments.append((start, end))
 
 # Create empty file
 with open(destination_path, 'wb') as f:
 f.truncate(total_size)
 
 # Download segments in parallel
 threads = []
 progress_buffer = {'progress': 0}
 
 for start, end in segments:
 thread = threading.Thread(
 target=download_segment,
 args=(url, start, end, destination_path, progress_buffer)
 )
 threads.append(thread)
 thread.start()
 
 for thread in threads:
 thread.join()

Memory-Efficient Alternatives

Memory-Mapped Files
For very large files that need processing after download:

python
import mmap

def download_with_mmap(url, destination_path):
 response = requests.get(url, stream=True)
 response.raise_for_status()
 
 with open(destination_path, 'wb') as f:
 for chunk in response.iter_content(chunk_size=8192):
 if chunk:
 f.write(chunk)
 
 # Memory-map the file for processing
 with open(destination_path, 'r+b') as f:
 with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
 # Process the file in memory-mapped form
 data = mm.read(1024) # Read first 1KB

Streaming Decompression
For compressed files, decompress on the fly:

python
import gzip
import requests

def download_and_decompress(url, destination_path):
 response = requests.get(url, stream=True)
 response.raise_for_status()
 
 with gzip.open(response.raw, 'rb') as gz_file:
 with open(destination_path, 'wb') as f:
 for chunk in gz_file:
 f.write(chunk)

Specialized Use Cases

Downloading from Cloud Storage
For cloud storage services, use their specific SDKs:

python
# AWS S3 example
import boto3

def download_from_s3(bucket_name, object_key, destination_path):
 s3 = boto3.client('s3')
 s3.download_file(bucket_name, object_key, destination_path)

# Google Cloud Storage example
from google.cloud import storage

def download_from_gcs(bucket_name, blob_name, destination_path):
 storage_client = storage.Client()
 bucket = storage_client.bucket(bucket_name)
 blob = bucket.blob(blob_name)
 blob.download_to_filename(destination_path)

Resumable Downloads
For unreliable connections, implement resumable downloads:

python
import requests
import os

def resumable_download(url, destination_path, chunk_size=8192):
 # Check if partial file exists
 if os.path.exists(destination_path):
 downloaded_size = os.path.getsize(destination_path)
 headers = {'Range': f'bytes={downloaded_size}-'}
 else:
 downloaded_size = 0
 headers = {}
 
 with requests.get(url, headers=headers, stream=True) as response:
 response.raise_for_status()
 
 # Open in append mode if resuming
 mode = 'ab' if downloaded_size > 0 else 'wb'
 with open(destination_path, mode) as f:
 for chunk in response.iter_content(chunk_size=chunk_size):
 if chunk:
 f.write(chunk)

These alternative approaches and optimizations provide different solutions depending on your specific requirements. The standard Requests library with streaming remains the most straightforward solution for most use cases, but these alternatives offer valuable options for performance-critical applications, special file types, or challenging network conditions.

Sources

  1. What is the stream parameter in Requests and when should I use it — Detailed explanation of streaming parameter behavior and implementation guidelines: https://webscraping.ai/faq/requests/what-is-the-stream-parameter-in-requests-and-when-should-i-use-it

  2. Downloading Files Over HTTP with Python Requests — Comprehensive guide with complete examples for streaming downloads and best practices: https://medium.com/@lope.ai/downloading-files-over-http-with-python-requests-e12e6b795e43

  3. Requests Module Streaming Responses — Basic streaming concept explanation and memory efficiency principles: https://www.pythonforall.com/modules/requests/rsstream

  4. Simple Python Streaming Download Example — Practical code reference for implementing streaming downloads with Requests: https://gist.github.com/wasi0013/ab73f314f8070951b92f6670f68b2d80

Conclusion

Downloading large files in Python using the Requests library without memory overload requires understanding and implementing proper streaming techniques. The key is using stream=True parameter with iter_content() method to process data incrementally rather than loading everything into memory at once. Many developers encounter memory issues because they either forget to enable streaming or mistakenly believe that iter_content() alone provides streaming functionality.

For reliable large file downloads, always implement proper error handling, use context managers for resource management, and consider chunk size optimization based on your specific use case. The implementation patterns we’ve discussed—from basic streaming to advanced techniques like parallel downloads and resumable downloads—provide solutions for various scenarios and performance requirements.

By following these guidelines, you can efficiently download files of any size while maintaining minimal memory footprint and robust error handling. The combination of stream=True and chunk-based processing ensures your Python applications can handle large file downloads reliably, whether you’re working with 1GB files or multi-gigabyte downloads.

Authors
NeuroAnswers
Author
Verified by moderation
NeuroAnswers
Moderation
Python Download Large Files with Requests: Memory-Efficient Streaming