How to Share State Between Rust Proc Macros

Currently, while using static variables to communicate between different proc macro invocations may work in tests, this approach is not guaranteed to work reliably due to incremental compilation, thread safety issues, and the lack of official support for state sharing between proc macros. The Rust compiler does not guarantee the execution order, thread placement, or even execution of individual macro invocations, making static variables an unreliable approach for inter-macro communication.

Contents

• How Static Variables Work in Proc Macros
• Incremental Compilation Issues
• Thread Safety and Execution Order
• Alternative Approaches
• Best Practices and Recommendations
• Working with Multiple Macro Types

How Static Variables Work in Proc Macros

Static variables can technically be used in procedural macros, but they come with significant limitations and risks. The example you provided demonstrates that static variables can work in simple test cases, but this doesn’t make them a reliable solution for production code.

static DATA: RwLock<bool> = RwLock::new(false);

#[proc_macro_attribute]
pub fn write_macro(_args: TokenStream, input: TokenStream) -> TokenStream {
    *DATA.write().unwrap() = true;
    input
}

#[proc_macro_attribute]
pub fn read_macro(_args: TokenStream, input: TokenStream) -> TokenStream {
    if *DATA.read().unwrap() {
        input
    } else {
        panic!("LoL")
    }
}

The problem is that there is nothing prevents you from using static in procedural macro [source], but the behavior is not guaranteed. Your test works because:

• Both macros are invoked during the same compilation session
• They run in the same process space
• The static variable persists across invocations

However, this approach breaks several key principles of procedural macro design and relies on implementation details rather than stable guarantees.

Incremental Compilation Issues

The most significant risk with using static variables in proc macros is incremental compilation. The Rust compiler may cache macro expansions and reuse them in subsequent compilations without re-executing the macro code.

According to the research findings:

• it is not guaranteed that all of your proc macro invocations are actually executed due to incremental compilation [source]
• The Rust compiler does not promise not to cache macro invocations, run each macro expansion [source]
• Macro execution order is not guaranteed, and macros may not be executed at all in a specific compilation if it could use a cached execution from a previous compilation [source]

This means that in a real-world project with incremental compilation enabled (which is the default in Cargo), your static variable approach could fail in unpredictable ways depending on:

• Which files have changed since the last compilation
• The order in which files are processed
• Whether the compiler decides to reuse cached macro expansions

Thread Safety and Execution Order

Another critical issue is thread safety. Procedural macros may be executed in different threads, which could lead to race conditions or other synchronization problems with static variables.

Key findings from the research:

• Ideally, we’d use CrossThread, which spawns a thread for each invocation, to prevent (and discourage) proc macros from using TLS for state between invocations [source]
• There is no guarantee on what order the macros will be executed in [source]
• To make it thread safe (and to avoid using unsafe) you could use AtomicUsize [source]

Even if you use thread-safe primitives like RwLock or AtomicUsize, the fundamental problem remains: the Rust compiler does not guarantee that all macro invocations will run in the same compilation session.

The RFC for procedural macros mentions that “A MacroContext is an object placed in thread-local storage when a macro is expanded” [source], which suggests that macros may indeed run in different threads or contexts.

Alternative Approaches

While static variables are unreliable, there are several alternative approaches to achieve your goal of collecting information from multiple proc macro expansions:

1. File-Based State Sharing

One reliable approach is to use temporary files to share state between macro invocations:

use std::fs;
use std::path::PathBuf;
use std::sync::Mutex;

static STATE_FILE: Mutex<Option<PathBuf>> = Mutex::new(None);

#[proc_macro_attribute]
pub fn collect_info(_args: TokenStream, input: TokenStream) -> TokenStream {
    let state_file = STATE_FILE.lock().unwrap();
    if let Some(path) = &*state_file {
        // Read existing state from file
        let existing = fs::read_to_string(path).unwrap_or_default();
        // Append new information
        let updated = format!("{}\n{}", existing, input.to_string());
        fs::write(path, updated).unwrap();
    }
    input
}

This approach works because file I/O is guaranteed to happen and isn’t affected by macro caching.

2. Environment Variables

You can also use environment variables to pass information:

#[proc_macro_attribute]
pub fn collect_info(_args: TokenStream, input: TokenStream) -> TokenStream {
    let existing = std::env::var("MY_MACRO_STATE").unwrap_or_default();
    let updated = format!("{}\n{}", existing, input.to_string());
    std::env::set_var("MY_MACRO_STATE", updated);
    input
}

3. Specialized Crates for Macro Coordination

Several crates are specifically designed to handle coordination between macro invocations:

• Macro Magic: “Among other things, the patterns introduced by Macro Magic can be used to implement safe and efficient coordination and communication between macro invocations in the same file, and even across different files and different crates” [source]

• Macro State: “designed to allow for building up and making use of state information across multiple macro invocations” [source]

These crates provide more robust and reliable solutions for inter-macro communication.

4. Module-Based Approach

Instead of trying to communicate between macros, you can restructure your approach to work within a module:

#[derive(MyDerive)]
pub struct MyType { }

#[collect_impls]
impl MyType {
  // ...
}

// Instead of mycrate::output!(), the output is generated by the derive macro

Best Practices and Recommendations

Based on the research findings and best practices for procedural macros, here are some recommendations:

1. Avoid Static Variables for Inter-Macro Communication

You should not use static variables in your macro crate to communicate information. It will work, right now, but the Rust compiler does not promise not to cache macro invocations [source].

2. Embrace Deterministic Approaches

Use approaches that are guaranteed to work regardless of compiler implementation details:

• File I/O
• Environment variables
• Dedicated coordination crates
• Module-based organization

3. Consider the Macro Ecosystem

If you’re building multiple related macros, think about how they’ll work together:

• Define clear interfaces between macros
• Document the expected usage patterns
• Provide examples and migration guides

4. Test for Incremental Compilation

When testing your macros, specifically test scenarios that exercise incremental compilation:

• Make small changes to files and verify macro behavior
• Test different compilation orders
• Verify that cached macro expansions don’t break your functionality

Working with Multiple Macro Types

Your use case involves different types of macros (derive macros and attribute macros), which adds complexity to the communication problem. Here are some strategies:

1. Separate Macro Crates

Consider splitting your macros into separate crates to reduce complexity:

// mytype-derive
#[derive(MyDerive)]
pub struct MyType { }

// mytype-impls
#[collect_impls]
impl MyType { }

// mytype-output
mytype_output::generate!()

2. Unified Macro System

Create a single macro system that handles both definition and implementation:

#[my_macro]
pub struct MyType { }

#[my_macro_impl]
impl MyType { }

// This is handled by the same macro system internally

3. Builder Pattern

Use a builder pattern where macros work together in a predictable sequence:

#[my_macro::define]
pub struct MyType { }

#[my_macro::impl]
impl MyType { }

#[my_macro::generate]
fn main() {
    // Generated code here
}

Sources

1. r/rust on Reddit: What is a good pattern to share state between procedural macros?
2. Is it possible to store state within Rust’s procedural macros? - Stack Overflow
3. rust - Proc macro execution order - Stack Overflow
4. r/rust on Reddit: Static modification from proc-macro context. Is it possible?
5. Run proc macro invocations in separate threads. · Issue #56058 · rust-lang/rust
6. mm_example_crate — Rust proc macro helper // Lib.rs
7. rust - Is there a consistent compilation context inside a proc_macro_attribute function? - Stack Overflow
8. Structuring, testing and debugging procedural macro crates - Ferrous Systems
9. 1566-proc-macros - The Rust RFC Book
10. Procedural Macros - The Rust Reference

Conclusion

While using static variables to communicate between proc macro invocations may work in simple tests, this approach is fundamentally unreliable and should be avoided in production code. The key takeaways are:

1. Static variables are not guaranteed to work due to incremental compilation, thread safety concerns, and lack of official support
2. Incremental compilation can cause macro invocations to be cached and not re-executed, breaking static variable communication
3. Thread safety issues exist as macros may run in different threads or contexts
4. Alternative approaches like file I/O, environment variables, or specialized crates (Macro Magic, Macro State) provide more reliable solutions
5. Restructure your macro design to avoid the need for inter-macro communication when possible

For your specific use case of collecting information from multiple macro expansions and outputting results all at once, consider using file-based state sharing, dedicated coordination crates, or restructuring your macros to work within a more deterministic pattern. These approaches will provide the reliability and maintainability needed for production code.