Python Mouse Automation with QR Code Trigger

Creating a Python script to automate mouse clicks at specific screen coordinates triggered by QR code scanning can be accomplished using PyAutoGUI for mouse control and pyzbar for QR code detection. This automation solution will allow you to programmatically move the cursor to predetermined positions and perform left-clicks on specific application buttons like a ‘Deliver’ icon when activated by scanning a QR code, all within Python 3.13.

Contents

• Introduction to Python Mouse Automation with PyAutoGUI
• Setting Up Your Environment for Python 3.13
• Implementing QR Code Scanning with pyzbar
• Creating the Automated Clicking Workflow
• Best Practices and Safety Considerations

Introduction to Python Mouse Automation with PyAutoGUI

Python provides powerful libraries for automating mouse actions, with PyAutoGUI being the most comprehensive solution for cross-platform GUI automation. The pyautogui python library allows you to programmatically control mouse movements, clicks, and keyboard inputs, making it ideal for automating repetitive tasks like clicking specific application buttons. When combined with QR code scanning capabilities, this creates a powerful automation system that can trigger mouse actions based on visual input.

PyAutoGUI works by simulating user input at the operating system level, which means it can interact with virtually any application running on your computer. The library provides functions for moving the cursor to absolute screen coordinates, clicking with different mouse buttons, and even taking screenshots of the current display. For our QR code triggered automation, we’ll focus on the mouse control functions that allow precise positioning and clicking.

One of the most important aspects of using PyAutoGUI for python mouse automation is understanding the coordinate system. The screen origin (0,0) is at the top-left corner, with x-coordinates increasing to the right and y-coordinates increasing downward. To find the exact coordinates of the ‘Deliver’ icon or any other button you want to click, you can use PyAutoGUI’s pyautogui.displayMousePosition() function, which shows your current mouse position in real-time as you move it around the screen.

Setting Up Your Environment for Python 3.13

To begin creating your QR code triggered automation script, you’ll need to install several Python packages that provide the necessary functionality. First, ensure you have Python 3.13 installed on your system. Then, install the required libraries using pip:

pip install pyautogui opencv-python pyzbar imutils

The pyautogui python package provides the core mouse automation capabilities, opencv-python handles camera access and image processing, pyzbar enables QR code detection, and imutils offers convenience functions for OpenCV operations.

However, pyzbar may require additional system-level dependencies depending on your operating system:

• Ubuntu/Debian: sudo apt-get install libzbar0
• macOS: brew install zbar
• Windows: Download and install the Visual C++ Redistributable 2013

For Python 3.13 compatibility with pyzbar, you might need to install it without precompiled binaries:

pip install --no-binary :all: pyzbar

After installing the packages, you can verify they work correctly by running a simple test script:

import pyautogui
import cv2
from pyzbar.pyzbar import decode

# Test PyAutoGUI
print("PyAutoGUI version:", pyautogui.__version__)
print("Screen size:", pyautogui.size())

# Test camera access
cap = cv2.VideoCapture(0)
if cap.isOpened():
 ret, frame = cap.read()
 if ret:
 decoded = decode(frame)
 print("QR code scanning test successful:", len(decoded), "codes detected")
 cap.release()
else:
 print("Camera access failed")

This setup ensures all components for your python mouse automation are properly installed and functioning before implementing the complete solution.

Implementing QR Code Scanning with pyzbar

The QR code scanning component is crucial for triggering your automation when a specific code is detected. Using pyzbar, you can create a scanning system that continuously monitors your camera feed for QR codes. Here’s how to implement this functionality:

First, initialize your camera and create a function to scan for QR codes:

import cv2
from pyzbar.pyzbar import decode, ZBarSymbol

def scan_qr_codes():
 # Initialize camera
 cap = cv2.VideoCapture(0)
 cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
 cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
 
 print("QR code scanner started. Press 'q' to quit.")
 
 while True:
 # Read frame from camera
 ret, frame = cap.read()
 if not ret:
 break
 
 # Convert to grayscale for better QR detection
 gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
 
 # Decode QR codes
 decoded = decode(gray, symbols=[ZBarSymbol.QRCODE])
 
 # Process each detected QR code
 for code in decoded:
 # Get QR code data
 qr_data = code.data.decode('utf-8')
 print(f"QR Code Detected: {qr_data}")
 
 # Draw bounding box around QR code
 points = code.polygon
 if len(points) > 4:
 hull = cv2.convexHull(np.array([point for point in points], dtype=np.float32))
 hull = list(map(tuple, np.squeeze(hull)))
 else:
 hull = points
 
 n = len(hull)
 for j in range(n):
 cv2.line(frame, hull[j], hull[(j+1) % n], (255, 0, 0), 3)
 
 # Add text label
 cv2.putText(frame, qr_data, (hull[0][0], hull[0][1] - 10), 
 cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 0, 0), 2)
 
 # Return the detected QR code data
 return qr_data
 
 # Display the frame
 cv2.imshow('QR Code Scanner', frame)
 
 # Check for exit condition
 if cv2.waitKey(1) & 0xFF == ord('q'):
 break
 
 cap.release()
 cv2.destroyAllWindows()
 return None

This function continuously scans for QR codes and returns the data of the first detected code. The visual feedback helps you confirm that the scanner is working correctly and provides a way to exit the program by pressing ‘q’.

For your specific use case, you’ll want to create predefined QR codes that correspond to different screen coordinates. For example, you could create QR codes with data like “click:800,600” or “deliver” that your script will recognize and map to specific mouse actions.

The QR code scanning process works by capturing frames from your camera, converting them to grayscale for better detection accuracy, and then using the pyzbar library to identify and decode any QR codes present in the frame. Once a code is detected, the function returns the data contained within it, which your automation script can then use to determine what action to perform.

Creating the Clicking Workflow

Now that we have both the QR code scanning and mouse automation components, let’s integrate them into a complete workflow. This script will wait for a specific QR code to be scanned, then move the mouse to predetermined coordinates and perform a left click.

Here’s the complete implementation:

import pyautogui
import cv2
from pyzbar.pyzbar import decode, ZBarSymbol
import time
import sys

def setup_pyautogui():
 """Configure PyAutoGUI settings for safe operation"""
 # Enable fail-safe - move mouse to corner to abort
 pyautogui.FAILSAFE = True
 
 # Add pause between operations
 pyautogui.PAUSE = 0.1
 
 print("PyAutoGUI configured with fail-safe enabled")
 print("Move mouse to top-left corner to abort automation")

def scan_qr_codes():
 """Scan for QR codes and return detected data"""
 cap = cv2.VideoCapture(0)
 cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
 cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
 
 print("QR code scanner started. Press 'q' to quit.")
 
 while True:
 ret, frame = cap.read()
 if not ret:
 break
 
 gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
 decoded = decode(gray, symbols=[ZBarSymbol.QRCODE])
 
 for code in decoded:
 qr_data = code.data.decode('utf-8')
 print(f"QR Code Detected: {qr_data}")
 
 # Draw visual feedback
 points = code.polygon
 if len(points) > 4:
 hull = cv2.convexHull(np.array([point for point in points], dtype=np.float32))
 hull = list(map(tuple, np.squeeze(hull)))
 else:
 hull = points
 
 n = len(hull)
 for j in range(n):
 cv2.line(frame, hull[j], hull[(j+1) % n], (255, 0, 0), 3)
 
 cv2.putText(frame, qr_data, (hull[0][0], hull[0][1] - 10), 
 cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 0, 0), 2)
 
 return qr_data
 
 cv2.imshow('QR Code Scanner', frame)
 
 if cv2.waitKey(1) & 0xFF == ord('q'):
 break
 
 cap.release()
 cv2.destroyAllWindows()
 return None

def perform_click_action(qr_data):
 """Perform mouse action based on QR code data"""
 # Define your coordinate mappings here
 click_targets = {
 "deliver": (1200, 700), # Replace with your actual coordinates
 "submit": (800, 600), # Replace with your actual coordinates
 "confirm": (1000, 800) # Replace with your actual coordinates
 }
 
 # Check if QR data matches any target
 if qr_data in click_targets:
 x, y = click_targets[qr_data]
 print(f"Moving mouse to coordinates: ({x}, {y})")
 
 # Move mouse to position with smooth transition
 pyautogui.moveTo(x, y, duration=0.5)
 
 # Small pause to ensure position is reached
 time.sleep(0.2)
 
 # Perform left click
 pyautogui.click()
 
 print("Click action completed!")
 return True
 else:
 print(f"No action defined for QR code: {qr_data}")
 return False

def main():
 """Main automation loop"""
 # Setup PyAutoGUI
 setup_pyautogui()
 
 print("QR Code Mouse Automation Started")
 print("Scan a QR code to trigger mouse actions")
 
 try:
 while True:
 # Wait for QR code scan
 qr_data = scan_qr_codes()
 
 if qr_data:
 # Perform the corresponding action
 perform_click_action(qr_data)
 
 # Wait before scanning again
 time.sleep(2)
 
 except KeyboardInterrupt:
 print("\nAutomation stopped by user")
 except Exception as e:
 print(f"Error occurred: {e}")
 finally:
 cv2.destroyAllWindows()
 print("Program terminated")

if __name__ == "__main__":
 main()

This script creates a complete automation workflow that:

1. Sets up PyAutoGUI with safety features enabled
2. Continuously scans for QR codes using your camera
3. When a QR code is detected, it checks if the code matches any predefined action
4. If a match is found, it moves the mouse to the corresponding coordinates and performs a left click
5. The script includes error handling and a way to safely exit the program

To customize this for your specific needs, modify the click_targets dictionary in the perform_click_action function with your actual screen coordinates for the ‘Deliver’ icon or any other buttons you want to automate.

The script uses smooth mouse movements (duration=0.5) to make the automation less jarring and more natural-looking. The small pause between operations ensures the system has time to process each action before the next one begins.

Best Practices and Safety Considerations

When implementing a python mouse automation script triggered by QR code scanning, it’s essential to follow best practices to ensure reliability, safety, and maintainability. Here are key considerations:

Safety Mechanisms

Always enable PyAutoGUI’s fail-safe feature by setting pyautogui.FAILSAFE = True. This allows you to immediately stop any automation by quickly moving the mouse to the top-left corner of the screen. This is crucial for preventing runaway automation that might interfere with other applications or cause unintended actions.

Additionally, set pyautogui.PAUSE = 0.1 to add a small delay between PyAutoGUI operations. This prevents overwhelming your system with too many rapid commands and gives applications time to respond to each action.

Testing and Calibration

Before deploying your automation, thoroughly test your coordinate mappings. The screen resolution on different computers can vary significantly, so you may need to adjust your coordinates based on the specific display you’re using. Create a simple test script that moves the mouse to each coordinate position without clicking, allowing you to verify accuracy.

For QR code scanning, test with various lighting conditions and distances to ensure reliable detection. Position your QR codes where they’ll be easily visible to the camera and avoid reflective surfaces that might interfere with scanning.

Performance Optimization

For better performance, consider implementing threading to separate the QR scanning from the mouse control operations. This prevents the camera processing from blocking the mouse actions and vice versa. Here’s a basic example:

import threading
import queue

def qr_scanner(q):
 """QR scanning thread"""
 cap = cv2.VideoCapture(0)
 while True:
 ret, frame = cap.read()
 if ret:
 decoded = decode(frame)
 if decoded:
 q.put(decoded[0].data.decode('utf-8'))
 time.sleep(0.1)

def main():
 # Create thread-safe queue
 q = queue.Queue()
 
 # Start QR scanner thread
 scanner_thread = threading.Thread(target=qr_scanner, args=(q,))
 scanner_thread.daemon = True
 scanner_thread.start()
 
 # Main processing loop
 while True:
 try:
 qr_data = q.get(timeout=1) # Wait up to 1 second for QR code
 if qr_data:
 perform_click_action(qr_data)
 except queue.Empty:
 # No QR code detected, continue waiting
 continue

Error Handling and Logging

Implement comprehensive error handling to gracefully manage potential issues like camera access problems, QR decoding failures, or unexpected system behavior. Add logging to track when actions are performed and any errors that occur, which will help with debugging and monitoring the automation’s performance.

Documentation and Maintenance

Document your coordinate mappings and QR code data formats clearly. As you add more automation targets, maintain an organized system for managing these mappings. Consider implementing a configuration file or database to store your target coordinates and QR code triggers, making it easier to update and maintain your automation script.

By following these best practices, you’ll create a robust, safe, and maintainable python automation solution that reliably performs mouse clicks triggered by QR code scanning.

Sources

1. Automate the Boring Stuff with Python - Chapter 18 - Comprehensive guide to PyAutoGUI and GUI automation: https://automatetheboringstuff.com/chapter18/
2. pyzbar Documentation - Official documentation for QR code scanning with Python: https://pypi.org/project/pyzbar/
3. PyImageSearch - OpenCV Barcode and QR Code Scanner - Complete working example for QR code detection: https://pyimagesearch.com/2018/05/21/an-opencv-barcode-and-qr-code-scanner-with-zbar/
4. The Python Code - Control Mouse with Python - Alternative approaches to mouse automation: https://thepythoncode.com/article/control-mouse-python

Conclusion

Creating a Python script to automate mouse clicks triggered by QR code scanning is a powerful way to streamline repetitive tasks. By combining PyAutoGUI for precise mouse control with pyzbar for QR code detection, you can build a robust automation system that responds to visual cues with programmed actions. The complete solution outlined in this guide provides a solid foundation for your python automation project, with safety mechanisms, error handling, and performance optimizations already in place. Remember to calibrate your coordinates for your specific screen resolution and test thoroughly before deploying in production environments. With this approach, you can efficiently automate clicking on specific application buttons like a ‘Deliver’ icon whenever a QR code is scanned.