How to resolve Microsoft Graph API webhook multiple triggers despite sending 200 response within 2.5 seconds?
I’m experiencing an issue with Microsoft Graph API webhooks triggering multiple times despite my response handling. Even though I’m sending a 200 response within 2.5 seconds, the webhook continues to trigger repeatedly. Here’s my current response handling code:
var startTime = DateTime.UtcNow;
//return StatusCode(202);
try
{
var elapsed = DateTime.UtcNow - startTime;
if (elapsed.TotalSeconds < 2.5)
{
Console.WriteLine("Accepted");
return Ok("Ok");
}
else
{
Console.WriteLine("Ok");
return Accepted();
}
}
catch (Exception ex)
{
Console.WriteLine("internal server ok");
return StatusCode(500);
}
Has anyone encountered this issue before and found a solution? What could be causing the webhook to trigger multiple times even with proper response handling?
Brief Answer
Microsoft Graph API webhooks often trigger multiple times by design as a reliability mechanism, even when you return a 200 response quickly. The issue typically stems from not properly handling idempotency in your webhook processing logic rather than response timing. You need to implement deduplication mechanisms in your business logic to safely process the same notification multiple times without side effects.
Contents
- Understanding Microsoft Graph API Webhook Behavior
- Why Multiple Triggers Occur
- Proper Webhook Handling Pattern
- Implementing Idempotency and Deduplication
- Code Improvements for Your Solution
- Additional Troubleshooting Steps
Understanding Microsoft Graph API Webhook Behavior
Microsoft Graph API webhooks operate on an “at-least-once” delivery model, which means the service will attempt to deliver notifications until it receives a successful response (typically a 200 OK status code). This design ensures reliability in distributed systems where network issues or temporary server problems might cause notifications to be lost.
Important Note: The 2.5-second response requirement is only for acknowledging receipt of the notification, not for completing the processing. Your endpoint should respond quickly to acknowledge receipt, while the actual processing can happen asynchronously.
Why Multiple Triggers Occur
Multiple triggers can happen for several legitimate reasons:
- Network Retries: If Microsoft Graph doesn’t receive a timely response, it will retry the delivery
- Server Load Balancing: If your service has multiple instances, each might receive the same notification
- Microsoft Graph Internal Retries: The service may retry delivery if it suspects a failure
- Client-Side Processing Delays: If your processing takes longer than expected, the service might retry
In your current code, you’re checking response timing and returning different status codes based on elapsed time. This approach misunderstands the purpose of the webhook response:
var elapsed = DateTime.UtcNow - startTime;
if (elapsed.TotalSeconds < 2.5)
{
Console.WriteLine("Accepted");
return Ok("Ok");
}
else
{
Console.WriteLine("Ok");
return Accepted();
}
The Microsoft Graph documentation doesn’t require different status codes based on response time. A simple 200 OK is sufficient to acknowledge receipt, regardless of whether the processing completes quickly or takes longer.
Proper Webhook Handling Pattern
A robust webhook handler should follow this pattern:
- Acknowledge receipt immediately: Return 200 OK as soon as possible
- Validate the notification: Check if it’s a valid Microsoft Graph notification
- Extract notification data: Get the resource data and subscription information
- Check for duplicates: Use the notification ID to process each notification only once
- Process the notification: Handle the business logic
- Store state: Track processed notifications to prevent duplicate processing
Here’s a basic implementation pattern:
[HttpPost]
public async Task<IActionResult> WebhookHandler()
{
try
{
// 1. Acknowledge receipt immediately
return Ok();
}
catch (Exception ex)
{
// Log the exception
return StatusCode(500);
}
}
// Separate method for processing notifications
public async Task ProcessNotification(WebhookNotification notification)
{
// Check if we've already processed this notification
if (await IsNotificationProcessed(notification.Id))
{
return; // Already processed, skip
}
// Process the notification business logic
await HandleBusinessLogic(notification);
// Mark as processed
await MarkNotificationAsProcessed(notification.Id);
}
Implementing Idempotency and Deduplication
The key to preventing multiple trigger issues is implementing proper idempotency:
- Use Notification ID: Each notification includes a unique ID in the
validationTokens
orsubscriptionId
field - Store Processed Notifications: Keep track of processed notification IDs
- Set Expiration: Clean up old processed notification records to prevent database bloat
Here’s how to implement this in your handler:
[HttpPost]
public async Task<IActionResult> WebhookHandler()
{
try
{
// Read the notification content
var content = await new StreamReader(Request.Body).ReadToEndAsync();
// Parse the notification
var notification = JsonConvert.DeserializeObject<WebhookNotification>(content);
// Start processing asynchronously
_ = ProcessNotificationAsync(notification);
// Acknowledge receipt immediately
return Ok();
}
catch (Exception ex)
{
_logger.LogError(ex, "Error processing webhook");
return StatusCode(500);
}
}
private async Task ProcessNotificationAsync(WebhookNotification notification)
{
try
{
// Check if we've already processed this notification
if (await _notificationRepository.IsProcessedAsync(notification.Id))
{
_logger.LogInformation("Notification {Id} already processed", notification.Id);
return;
}
// Process the business logic
await _notificationProcessor.ProcessAsync(notification);
// Mark as processed
await _notificationRepository.MarkAsProcessedAsync(notification.Id);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error processing notification {Id}", notification.Id);
}
}
Code Improvements for Your Solution
Based on your provided code, here are the key improvements:
- Remove timing-based status codes: Always return 200 OK to acknowledge receipt
- Separate acknowledgment from processing: Acknowledge immediately, process asynchronously
- Implement proper error handling: Catch and log exceptions, but still return 200 OK for most cases
- Add idempotency checks: Use notification IDs to avoid duplicate processing
Here’s an improved version of your handler:
[HttpPost]
public async Task<IActionResult> WebhookHandler()
{
try
{
// Read the request body
var body = await new StreamReader(Request.Body).ReadToEndAsync();
// Parse the notification (adjust based on your actual notification structure)
var notification = JsonConvert.DeserializeObject<WebhookPayload>(body);
// Start processing asynchronously without waiting
_ = Task.Run(() => ProcessNotification(notification));
// Always return 200 OK to acknowledge receipt
return Ok();
}
catch (Exception ex)
{
// Log the error but still return 200 OK
// Microsoft Graph will retry if it gets a non-200 response
Console.WriteLine($"Error processing webhook: {ex.Message}");
return Ok(); // Still acknowledge receipt
}
}
private async Task ProcessNotification(WebhookPayload notification)
{
try
{
// Check if we've processed this notification before
if (await _notificationStore.IsProcessedAsync(notification.Id))
{
Console.WriteLine($"Notification {notification.Id} already processed");
return;
}
// Process the actual business logic
await _businessLogic.ProcessChangeAsync(notification);
// Mark as processed
await _notificationStore.MarkAsProcessedAsync(notification.Id);
}
catch (Exception ex)
{
Console.WriteLine($"Error processing notification {notification.Id}: {ex.Message}");
// Don't rethrow - we don't want to affect the HTTP response
}
}
Additional Troubleshooting Steps
If you’re still experiencing issues after implementing proper idempotency:
- Verify Subscription Configuration: Check your subscription settings for expiration and notification URL
- Review Logging: Add detailed logging to track notification processing
- Check for Throttling: Ensure your service isn’t being throttled by Microsoft Graph
- Validate Notification Format: Confirm the notification structure matches Microsoft Graph’s specifications
- Test with Microsoft Graph Explorer: Use the Microsoft Graph Explorer to create test subscriptions
- Monitor Endpoint Health: Ensure your endpoint is consistently available and responsive
Conclusion
Microsoft Graph API webhooks are designed to deliver notifications multiple times to ensure reliability. The key to handling this is:
- Always return a 200 OK response within 2.5 seconds to acknowledge receipt
- Implement proper idempotency handling in your business logic
- Use notification IDs to track and skip duplicate notifications
- Process notifications asynchronously after acknowledging receipt
- Store state to prevent duplicate processing of the same notification
By following these patterns, you can build a robust webhook handler that gracefully handles multiple notifications without causing duplicate business operations or side effects.