Stakater Reloader Not Detecting Secret Changes

Stakater Reloader is not detecting your secret changes despite proper configuration, which likely stems from namespace access restrictions or the specific way SecretProviderClass updates secrets. The default behavior shows Reloader should monitor all namespaces, but RBAC limitations or secret update mechanisms may be preventing proper detection.

Contents

• Understanding Reloader’s Namespace Monitoring
• Common Configuration Issues and Solutions
• SecretProviderClass Integration Challenges
• Troubleshooting Steps
• Advanced Configuration Options
• Verification Methods

Understanding Reloader’s Namespace Monitoring

Based on the research findings, Reloader’s default behavior is to monitor changes in all namespaces when deployed. The warning message in your logs “KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces” confirms this default configuration.

However, this default behavior doesn’t guarantee access to all namespaces. The key insight from the research is that:

“By default Reloader deploys in default namespace and monitors changes in all namespaces. however, this can be restricted if needed via the RBAC authorization, using the namespaces specified in the ClusterRole and ClusterRoleBinding.” [source]

In your case, even though Reloader is running in the reloader namespace, it should theoretically have access to monitor the test namespace if the RBAC permissions are properly configured.

Common Configuration Issues and Solutions

1. RBAC Namespace Restrictions

The most likely issue is that Reloader’s service account doesn’t have the necessary permissions to watch secrets in the test namespace. Since you deployed Reloader in the reloader namespace, you need to ensure the ClusterRoleBinding allows cross-namespace access.

Solution: Verify your RBAC configuration includes proper namespace permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: reloader-cluster-role-binding
  namespace: reloader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: reloader-cluster-role
subjects:
- kind: ServiceAccount
  name: reloader-sa
  namespace: reloader

2. Annotation Format Validation

Your deployment annotations appear correct, but let’s double-check the format:

secret.reloader.stakater.com/auto: "true"
secret.reloader.stakater.com/reload: test-keystore

These annotations should match the secret name exactly. The research shows that Reloader looks for these specific annotations to trigger reloads.

3. Secret Name Matching

Ensure the secret name in your annotation (test-keystore) exactly matches the secret name being updated. Even minor discrepancies can prevent detection.

SecretProviderClass Integration Challenges

The SecretProviderClass from Azure Key Vault presents a unique challenge for Reloader. According to the research, Reloader uses SHA1 hash computation to detect changes in secrets. However, SecretProviderClass updates may not trigger the same change detection mechanisms as direct secret updates.

Key Issues with SecretProviderClass:

1. Update Mechanism: SecretProviderClass updates secrets differently than manual kubectl commands, potentially not triggering the same change detection events.

2. Hash Comparison: Reloader compares SHA1 hashes to detect changes. If the SecretProviderClass update doesn’t change the secret’s content in a way that affects the hash, Reloader won’t detect it.

3. Event Generation: The SecretRotationComplete event indicates the secret was rotated, but this might not generate the same watch events that Reloader monitors.

Solutions for SecretProviderClass Integration:

1. Force Secret Content Changes: Ensure the SecretProviderClass actually changes the secret content, not just the rotation timestamp.

2. Manual Secret Update Test: Test with a manual secret update to verify if Reloader detects changes:

kubectl create secret generic test-keystore --from-literal=test=value -n test
kubectl create secret generic test-keystore --from-literal=test=newvalue -n test

3. Check SecretProviderClass Configuration: Review your SecretProviderClass to ensure it’s properly updating the secret content rather than just metadata.

Troubleshooting Steps

Step 1: Verify Reloader’s Access to the Test Namespace

Check if Reloader can actually access secrets in the test namespace:

# Check if reloader can list secrets in test namespace
kubectl auth can-i list secrets -n test --as=system:serviceaccount:reloader:reloader-sa

# Check if reloader can watch secrets in test namespace  
kubectl auth can-i watch secrets -n test --as=system:serviceaccount:reloader:reloader-sa

Step 2: Manual Secret Update Test

Perform a manual secret update to see if Reloader responds:

# Update a secret manually
kubectl create secret generic test-keystore --from-literal=test=value -n test --dry-run=client -o yaml | kubectl apply -f -

# Wait a moment, then update with new content
kubectl create secret generic test-keystore --from-literal=test=updatedvalue -n test --dry-run=client -o yaml | kubectl apply -f -

Step 3: Check Reloader Logs for Specific Errors

Enable debug mode and look for specific error messages related to the test namespace:

kubectl logs -f deployment/reloader -n reloader --tail=100

Look for messages like:

• Permission denied errors
• Watch errors for the test namespace
• Secret hash comparison results

Step 4: Verify Secret Hash Changes

According to the research, Reloader uses SHA1 to compute hash values to detect changes. You can manually verify if the secret hash is actually changing:

# Get the secret current state
kubectl get secret test-keystore -n test -o yaml

# Check the secret data content
kubectl get secret test-keystore -n test -o jsonpath='{.data}'

Advanced Configuration Options

1. Namespace-Specific Deployment

If you continue to have issues, consider deploying Reloader in the same namespace as your application:

helm install reloader stakater/reloader --namespace test --set watchGlobally=false

2. Configure Specific Namespace Monitoring

You can configure Reloader to monitor only specific namespaces by setting the environment variable:

kubectl set env deployment/reloader -n reloader KUBERNETES_NAMESPACE=test

3. Enable Additional Logging

Enable verbose logging to get more detailed information about Reloader’s operations:

env:
- name: LOG_LEVEL
  value: debug

Verification Methods

1. Event Verification

After making changes, check for Reloader-specific events:

kubectl get events -n test --field-selector involvedObject.kind=Secret

Look for events like:

• “Changes Detected in secret-name of type ‘SECRET’ in namespace: test”
• “Updated deployment-resource of type Deployment in namespace: test”

2. Pod Restart Verification

Check if your deployment pods are restarting after secret changes:

kubectl get pods -n test -l app=test-app
kubectl describe pod <pod-name> -n test | grep -i "reloading\|restarted\"

3. Manual Reloader Trigger

If all else fails, you can manually trigger Reloader by updating the ConfigMap that monitors the secret:

# Create a dummy configmap to trigger reloader
kubectl create configmap dummy-configmap -n test --from-literal=dummy=value
kubectl create configmap dummy-configmap -n test --from-literal=dummy=updatedvalue

Sources

1. GitHub - stakater/Reloader: A Kubernetes controller to watch changes in ConfigMap and Secrets
2. Reloader/docs/How-it-works.md at master · stakater/Reloader
3. Stakater Reloader
4. Effortless Configuration Reloading in Kubernetes with Stakater Reloader
5. Automating Configuration Updates in Kubernetes with Reloader - Collabnix
6. Verify Reloader’s Working - Stakater Reloader

Conclusion

The main issues preventing Reloader from detecting your secret changes are likely:

1. RBAC Namespace Access: Ensure Reloader’s service account has proper permissions to watch secrets in the test namespace
2. SecretProviderClass Update Behavior: The way SecretProviderClass updates secrets might not trigger Reloader’s SHA1-based change detection
3. Annotation Verification: Double-check that secret names in annotations exactly match the actual secret names

Recommended Actions:

• First, verify RBAC permissions for cross-namespace access
• Test with manual secret updates to confirm Reloader functionality
• Consider deploying Reloader in the same namespace as your application
• Enable debug logging to capture detailed diagnostic information

By systematically addressing these areas, you should be able to resolve the issue and ensure Reloader properly monitors and responds to secret changes in your environment.