Fix boto3 RecursionError and ImportError in Python 3.11

RecursionError and ImportError issues commonly occur with boto3 in Python 3.11 due to compatibility problems between boto3, botocore, and urllib3 versions. The proper solution involves using the correct version combinations instead of downgrading, as older versions introduce different import errors.

Contents

• Understanding the RecursionError in boto3 with Python 3.11
• Understanding the ImportError after downgrading boto3
• Compatible Version Combinations for Python 3.11
• The SSL/urllib3 Recursion Issue Explained
• Proper Solutions Without Downgrading Python
• Step-by-Step Fix Implementation
• Verification and Testing
• Best Practices for Future boto3 Implementations
• Sources
• Conclusion

Understanding the RecursionError in boto3 with Python 3.11

When you encounter a RecursionError: maximum recursion depth exceeded in botocore/httpsession.py while using boto3 with Python 3.11, this is typically related to SSL context creation issues. The error occurs because newer versions of urllib3 (2.0+) introduced changes to the SSLContext API, while botocore versions before 1.34.55 still rely on the old API implementation.

This creates a deep recursion loop when boto3 attempts to create an S3 client or resource. The problem manifests specifically when initializing boto3 services like boto3.client(‘s3’) or creating boto3 S3 resources. The stack trace consistently points to the SSL context creation process within the botocore library.

Understanding the ImportError after downgrading boto3

When you downgrade boto3 to version 1.9.164 to resolve the RecursionError, you encounter a different error: ImportError: cannot import name 'Mapping' from 'collections'. This error occurs in botocore/vendored/requests/packages/urllib3/_collections.py and happens because older boto3 versions bundle vendored urllib3 that expects to import collections.Mapping.

In Python 3.10 and later, the Mapping class was moved from collections to collections.abc. When your code tries to import Mapping from the old location, Python raises an ImportError because the import path no longer exists. This is a common issue when using older versions of boto3 with newer Python versions.

Compatible Version Combinations for Python 3.11

The compatibility matrix for Python 3.11 with boto3, botocore, and urllib3 is well-documented in the boto3 issue tracker. According to the GitHub issue #4061, here are the recommended version combinations:

Python 3.11 Compatibility Matrix:
- boto3: 1.34.0 minimum, recommended 1.34.55+
- botocore: 1.34.0 minimum, recommended 1.34.55+
- urllib3: 1.26.0 minimum, recommended 1.26.18

This combination ensures compatibility between the libraries while avoiding both the RecursionError and ImportError issues. The recommended versions specifically address the SSL context changes in urllib3 2.0+ while maintaining compatibility with Python 3.11’s collection changes.

When working with boto3 s3 operations, using these version combinations ensures that your code will work reliably without hitting these compatibility issues.

The SSL/urllib3 Recursion Issue Explained

The root cause of the RecursionError is a dependency conflict between botocore and urllib3. Here’s what happens:

1. urllib3 2.0+ changed its SSLContext API, removing the DEFAULT_CIPHERS and modifying how SSL contexts are created
2. botocore versions before 1.34.55 still use the old API when creating SSL contexts
3. When you call boto3.client(‘s3’), botocore attempts to create a urllib3 SSL context using the old API
4. This creates a recursive call loop because the new urllib3 API is designed to call back to itself differently
5. Python hits its maximum recursion limit and raises the RecursionError

This is a known issue that affected many users when they upgraded to Python 3.11 without updating their boto3 dependencies appropriately. The problem is particularly common in Docker containers where different versions of the libraries might be pinned.

Proper Solutions Without Downgrading Python

Solution 1: Upgrade to Compatible Versions

The recommended solution is to upgrade your dependencies to the compatible versions rather than downgrading boto3. This approach addresses both the RecursionError and potential ImportError issues:

pip install --upgrade boto3>=1.34.55 botocore>=1.34.55 urllib3==1.26.18

This command installs the minimum required versions of boto3 and botocore while pinning urllib3 to a version that works with both the newer Python 3.11 and the updated botocore API.

Solution 2: Update urllib3 to 2.0+ with Latest botocore

If you want to use the latest features of urllib3 2.0+, you should use the very latest versions of boto3 and botocore that officially support urllib3 2.0:

pip install --upgrade boto3 botocore

Note that as of the time of writing, fully compatible versions with urllib3 2.0+ might not be released yet, so Solution 1 is more reliable.

Solution 3: Use Version Constraints in Requirements.txt

For Docker containers and production deployments, specify the exact versions in your requirements.txt:

boto3>=1.34.55,<2.0.0
botocore>=1.34.55,<2.0.0
urllib3==1.26.18

This ensures your container uses compatible versions without unexpected upgrades.

Step-by-Step Fix Implementation

1. Check your current versions:

   python

   import boto3
   import botocore
   import urllib3
   print(f"boto3 version: {boto3.__version__}")
   print(f"botocore version: {botocore.__version__}")
   print(f"urllib3 version: {urllib3.__version__}")
   

2. Create a backup of your requirements.txt if you have one

3. Update your requirements.txt with the compatible versions:

   boto3>=1.34.55,<2.0.0
   botocore>=1.34.55,<2.0.0
   urllib3==1.26.18
   

4. Install the updated packages:

   bash

   pip install --upgrade -r requirements.txt
   

5. Test your boto3 implementation:

   python

   import boto3
   try:
       s3 = boto3.client('s3')
       print("Successfully created S3 client")
   except Exception as e:
       print(f"Error: {e}")
   

6. Verify your boto3 S3 operations work as expected

Verification and Testing

After implementing the fix, thoroughly test your boto3 implementation to ensure all functionality works correctly:

1. Test S3 client creation:

   python

   import boto3
   s3_client = boto3.client('s3')
   

2. Test S3 resource operations:

   python

   s3_resource = boto3.resource('s3')
   buckets = list(s3_resource.buckets.all())
   print(f"Found {len(buckets)} buckets")
   

3. Test file upload/download:

   python

   # Upload test
   s3_client.upload_file('test.txt', 'your-bucket-name', 'test.txt')
   
   # Download test
   s3_client.download_file('your-bucket-name', 'test.txt', 'downloaded.txt')
   

4. Test all your application’s boto3-related functionality

If you encounter any issues after the update, double-check that you’re using the exact versions specified in the compatibility matrix. For complex applications, consider testing in a staging environment before deploying to production.

Best Practices for Future boto3 Implementations

1. Use version constraints: Always specify version ranges in your requirements.txt to prevent unexpected upgrades

2. Regular dependency updates: Periodically check for updates to boto3 and related libraries, especially when upgrading Python versions

3. Monitor boto3 releases: Keep an eye on boto3 release notes for compatibility announcements

4. Test in isolated environments: Always test dependency changes in a staging environment before production deployment

5. Use virtual environments: Maintain separate virtual environments for different projects to avoid version conflicts

6. Document your dependencies: Keep clear documentation of the Python and library versions your application requires

7. Consider using Docker images: Use official Python Docker images that have compatible versions pre-installed

8. Implement error handling: Add proper error handling in your code to catch and gracefully handle any potential boto3-related errors

By following these practices, you can avoid compatibility issues with boto3 and Python 3.11 in the future and ensure your applications remain stable and reliable.

Sources

1. RecursionError / ImportError with boto3 on Python 3.11 - GitHub Issue #4061
2. ImportError: cannot import name ‘Mapping’ from ‘collections’ - GitHub Issue #4487
3. ImportError: cannot import name ‘Mapping’ from ‘collections’ - GitHub Issue #3766
4. botocore 1.34.63 requires outdated urllib3 with python 3.10 but python 3.11 - GitHub Issue #3143
5. Unable to import boto3 because of botocore dependancies on urllib3s removed DEFAULT_CIPHERS - GitHub Issue #2921
6. ImportError: cannot import name Mapping from ‘collections’ - Bobbyhadz

Conclusion

The RecursionError and ImportError issues you’re experiencing with boto3 and Python 3.11 are well-documented compatibility problems between the library versions. Rather than downgrading boto3—which introduces a different ImportError—the solution is to use the compatible version combination: boto3 1.34.55+, botocore 1.34.55+, and urllib3 1.26.18. This approach addresses both the SSL recursion issue and maintains compatibility with Python 3.11’s collection changes. By properly managing your dependencies and following the recommended version matrix, you can avoid these issues and ensure your boto3 S3 operations work reliably in your Python 3.11 environment.