Crate syn

Source
Expand description

github crates-io docs-rs


Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.

Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally.

  • Data structures — Syn provides a complete syntax tree that can represent any valid Rust source code. The syntax tree is rooted at syn::File which represents a full source file, but there are other entry points that may be useful to procedural macros including syn::Item, syn::Expr and syn::Type.

  • Derives — Of particular interest to derive macros is syn::DeriveInput which is any of the three legal input items to a derive macro. An example below shows using this type in a library that can derive implementations of a user-defined trait.

  • Parsing — Parsing in Syn is built around parser functions with the signature fn(ParseStream) -> Result<T>. Every syntax tree node defined by Syn is individually parsable and may be used as a building block for custom syntaxes, or you may dream up your own brand new syntax without involving any of our syntax tree types.

  • Location information — Every token parsed by Syn is associated with a Span that tracks line and column information back to the source of that token. These spans allow a procedural macro to display detailed error messages pointing to all the right places in the user’s code. There is an example of this below.

  • Feature flags — Functionality is aggressively feature gated so your procedural macros enable only what they need, and do not pay in compile time for all the rest.


§Example of a derive macro

The canonical derive macro using Syn looks like this. We write an ordinary Rust function tagged with a proc_macro_derive attribute and the name of the trait we are deriving. Any time that derive appears in the user’s code, the Rust compiler passes their data structure as tokens into our macro. We get to execute arbitrary Rust code to figure out what to do with those tokens, then hand some tokens back to the compiler to compile into the user’s crate.

[dependencies]
syn = "2.0"
quote = "1.0"

[lib]
proc-macro = true
use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};

#[proc_macro_derive(MyMacro)]
pub fn my_macro(input: TokenStream) -> TokenStream {
    // Parse the input tokens into a syntax tree
    let input = parse_macro_input!(input as DeriveInput);

    // Build the output, possibly using quasi-quotation
    let expanded = quote! {
        // ...
    };

    // Hand the output tokens back to the compiler
    TokenStream::from(expanded)
}

The heapsize example directory shows a complete working implementation of a derive macro. The example derives a HeapSize trait which computes an estimate of the amount of heap memory owned by a value.

pub trait HeapSize {
    /// Total number of bytes of heap memory owned by `self`.
    fn heap_size_of_children(&self) -> usize;
}

The derive macro allows users to write #[derive(HeapSize)] on data structures in their program.

#[derive(HeapSize)]
struct Demo<'a, T: ?Sized> {
    a: Box<T>,
    b: u8,
    c: &'a str,
    d: String,
}


§Spans and error reporting

The token-based procedural macro API provides great control over where the compiler’s error messages are displayed in user code. Consider the error the user sees if one of their field types does not implement HeapSize.

#[derive(HeapSize)]
struct Broken {
    ok: String,
    bad: std::thread::Thread,
}

By tracking span information all the way through the expansion of a procedural macro as shown in the heapsize example, token-based macros in Syn are able to trigger errors that directly pinpoint the source of the problem.

error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
 --> src/main.rs:7:5
  |
7 |     bad: std::thread::Thread,
  |     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`

§Parsing a custom syntax

The lazy-static example directory shows the implementation of a functionlike!(...) procedural macro in which the input tokens are parsed using Syn’s parsing API.

The example reimplements the popular lazy_static crate from crates.io as a procedural macro.

lazy_static! {
    static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}

The implementation shows how to trigger custom warnings and error messages on the macro input.

warning: come on, pick a more creative name
  --> src/main.rs:10:16
   |
10 |     static ref FOO: String = "lazy_static".to_owned();
   |                ^^^

§Testing

When testing macros, we often care not just that the macro can be used successfully but also that when the macro is provided with invalid input it produces maximally helpful error messages. Consider using the trybuild crate to write tests for errors that are emitted by your macro or errors detected by the Rust compiler in the expanded code following misuse of the macro. Such tests help avoid regressions from later refactors that mistakenly make an error no longer trigger or be less helpful than it used to be.


§Debugging

When developing a procedural macro it can be helpful to look at what the generated code looks like. Use cargo rustc -- -Zunstable-options --pretty=expanded or the cargo expand subcommand.

To show the expanded code for some crate that uses your procedural macro, run cargo expand from that crate. To show the expanded code for one of your own test cases, run cargo expand --test the_test_case where the last argument is the name of the test file without the .rs extension.

This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust’s new Custom Derive system.


§Optional features

Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.

  • derive (enabled by default) — Data structures for representing the possible input to a derive macro, including structs and enums and types.
  • full — Data structures for representing the syntax tree of all valid Rust source code, including items and expressions.
  • parsing (enabled by default) — Ability to parse input tokens into a syntax tree node of a chosen type.
  • printing (enabled by default) — Ability to print a syntax tree node as tokens of Rust source code.
  • visit — Trait for traversing a syntax tree.
  • visit-mut — Trait for traversing and mutating in place a syntax tree.
  • fold — Trait for transforming an owned syntax tree.
  • clone-impls (enabled by default) — Clone impls for all syntax tree types.
  • extra-traits — Debug, Eq, PartialEq, Hash impls for all syntax tree types.
  • proc-macro (enabled by default) — Runtime dependency on the dynamic library libproc_macro from rustc toolchain.

Modules§

  • bufferparsing
    A stably addressed token buffer supporting efficient traversal based on a cheaply copyable cursor.
  • extparsing
    Extension traits to provide parsing methods on foreign types.
  • foldfold
    Syntax tree traversal to transform the nodes of an owned syntax tree.
  • metaparsing and (full or derive)
    Facility for interpreting structured content inside of an Attribute.
  • parseparsing
    Parsing interface for parsing a token stream into a syntax tree node.
  • punctuated
    A punctuated sequence of syntax tree nodes separated by punctuation.
  • spannedparsing and printing
    A trait that can provide the Span of the complete contents of a syntax tree node.
  • token
    Tokens representing Rust punctuation, keywords, and delimiters.
  • visitvisit
    Syntax tree traversal to walk a shared borrow of a syntax tree.
  • visit_mutvisit-mut
    Syntax tree traversal to mutate an exclusive borrow of a syntax tree in place.

Macros§

  • Token
    A type-macro that expands to the name of the Rust type representation of a given token.
  • bracedparsing
    Parse a set of curly braces and expose their content to subsequent parsers.
  • bracketedparsing
    Parse a set of square brackets and expose their content to subsequent parsers.
  • custom_keyword
    Define a type that supports parsing and printing a given identifier as if it were a keyword.
  • custom_punctuation
    Define a type that supports parsing and printing a multi-character symbol as if it were a punctuation token.
  • parenthesizedparsing
    Parse a set of parentheses and expose their content to subsequent parsers.
  • parse_macro_inputparsing and proc-macro
    Parse the input TokenStream of a macro, triggering a compile error if the tokens fail to parse.
  • parse_quoteparsing and printing
    Quasi-quotation macro that accepts input like the quote! macro but uses type inference to figure out a return type for those tokens.
  • parse_quote_spannedparsing and printing
    This macro is parse_quote! + quote_spanned!.

Structs§

  • Abifull or derive
    The binary interface of a function: extern "C".
  • AngleBracketedGenericArgumentsfull or derive
    Angle bracketed arguments of a path segment: the <K, V> in HashMap<K, V>.
  • Armfull
    One arm of a match expression: 0..=10 => { return true; }.
  • AssocConstfull or derive
    An equality constraint on an associated constant: the PANIC = false in Trait<PANIC = false>.
  • AssocTypefull or derive
    A binding (equality constraint) on an associated type: the Item = u8 in Iterator<Item = u8>.
  • Attributefull or derive
    An attribute, like #[repr(transparent)].
  • BareFnArgfull or derive
    An argument in a function type: the usize in fn(usize) -> bool.
  • BareVariadicfull or derive
    The variadic argument of a function pointer like fn(usize, ...).
  • Blockfull
    A braced block containing Rust statements.
  • BoundLifetimesfull or derive
    A set of bound lifetimes: for<'a, 'b, 'c>.
  • ConstParamfull or derive
    A const generic parameter: const LENGTH: usize.
  • Constraintfull or derive
    An associated type bound: Iterator<Item: Display>.
  • DataEnumderive
    An enum input to a proc_macro_derive macro.
  • DataStructderive
    A struct input to a proc_macro_derive macro.
  • DataUnionderive
    An untagged union input to a proc_macro_derive macro.
  • DeriveInputderive
    Data structure sent to a proc_macro_derive macro.
  • Error
    Error returned when a Syn parser cannot parse the input tokens.
  • ExprArrayfull
    A slice literal expression: [a, b, c, d].
  • ExprAssignfull
    An assignment expression: a = compute().
  • ExprAsyncfull
    An async block: async { ... }.
  • ExprAwaitfull
    An await expression: fut.await.
  • ExprBinaryfull or derive
    A binary operation: a + b, a += b.
  • ExprBlockfull
    A blocked scope: { ... }.
  • ExprBreakfull
    A break, with an optional label to break and an optional expression.
  • ExprCallfull or derive
    A function call expression: invoke(a, b).
  • ExprCastfull or derive
    A cast expression: foo as f64.
  • ExprClosurefull
    A closure expression: |a, b| a + b.
  • ExprConstfull
    A const block: const { ... }.
  • ExprContinuefull
    A continue, with an optional label.
  • ExprFieldfull or derive
    Access of a named struct field (obj.k) or unnamed tuple struct field (obj.0).
  • ExprForLoopfull
    A for loop: for pat in expr { ... }.
  • ExprGroupfull
    An expression contained within invisible delimiters.
  • ExprIffull
    An if expression with an optional else block: if expr { ... } else { ... }.
  • ExprIndexfull or derive
    A square bracketed indexing expression: vector[2].
  • ExprInferfull
    The inferred value of a const generic argument, denoted _.
  • ExprLetfull
    A let guard: let Some(x) = opt.
  • ExprLitfull or derive
    A literal in place of an expression: 1, "foo".
  • ExprLoopfull
    Conditionless loop: loop { ... }.
  • ExprMacrofull or derive
    A macro invocation expression: format!("{}", q).
  • ExprMatchfull
    A match expression: match n { Some(n) => {}, None => {} }.
  • ExprMethodCallfull or derive
    A method call expression: x.foo::<T>(a, b).
  • ExprParenfull or derive
    A parenthesized expression: (a + b).
  • ExprPathfull or derive
    A path like std::mem::replace possibly containing generic parameters and a qualified self-type.
  • ExprRangefull
    A range expression: 1..2, 1.., ..2, 1..=2, ..=2.
  • ExprRawAddrfull
    Address-of operation: &raw const place or &raw mut place.
  • ExprReferencefull or derive
    A referencing operation: &a or &mut a.
  • ExprRepeatfull
    An array literal constructed from one repeated element: [0u8; N].
  • ExprReturnfull
    A return, with an optional value to be returned.
  • ExprStructfull or derive
    A struct literal expression: Point { x: 1, y: 1 }.
  • ExprTryfull
    A try-expression: expr?.
  • ExprTryBlockfull
    A try block: try { ... }.
  • ExprTuplefull
    A tuple expression: (a, b, c, d).
  • ExprUnaryfull or derive
    A unary operation: !x, *x.
  • ExprUnsafefull
    An unsafe block: unsafe { ... }.
  • ExprWhilefull
    A while loop: while expr { ... }.
  • ExprYieldfull
    A yield expression: yield expr.
  • Fieldfull or derive
    A field of a struct or enum variant.
  • FieldPatfull
    A single field in a struct pattern.
  • FieldValuefull or derive
    A field-value pair in a struct literal.
  • FieldsNamedfull or derive
    Named fields of a struct or struct variant such as Point { x: f64, y: f64 }.
  • FieldsUnnamedfull or derive
    Unnamed fields of a tuple struct or tuple variant such as Some(T).
  • Filefull
    A complete file of Rust source code.
  • ForeignItemFnfull
    A foreign function in an extern block.
  • ForeignItemMacrofull
    A macro invocation within an extern block.
  • ForeignItemStaticfull
    A foreign static item in an extern block: static ext: u8.
  • ForeignItemTypefull
    A foreign type in an extern block: type void.
  • Genericsfull or derive
    Lifetimes and type parameters attached to a declaration of a function, enum, trait, etc.
  • Ident
    A word of Rust code, which may be a keyword or legal variable name.
  • ImplGenerics(full or derive) and printing
    Returned by Generics::split_for_impl.
  • ImplItemConstfull
    An associated constant within an impl block.
  • ImplItemFnfull
    An associated function within an impl block.
  • ImplItemMacrofull
    A macro invocation within an impl block.
  • ImplItemTypefull
    An associated type within an impl block.
  • Indexfull or derive
    The index of an unnamed tuple struct field.
  • ItemConstfull
    A constant item: const MAX: u16 = 65535.
  • ItemEnumfull
    An enum definition: enum Foo<A, B> { A(A), B(B) }.
  • ItemExternCratefull
    An extern crate item: extern crate serde.
  • ItemFnfull
    A free-standing function: fn process(n: usize) -> Result<()> { ... }.
  • ItemForeignModfull
    A block of foreign items: extern "C" { ... }.
  • ItemImplfull
    An impl block providing trait or associated items: impl<A> Trait for Data<A> { ... }.
  • ItemMacrofull
    A macro invocation, which includes macro_rules! definitions.
  • ItemModfull
    A module or module declaration: mod m or mod m { ... }.
  • ItemStaticfull
    A static item: static BIKE: Shed = Shed(42).
  • ItemStructfull
    A struct definition: struct Foo<A> { x: A }.
  • ItemTraitfull
    A trait definition: pub trait Iterator { ... }.
  • ItemTraitAliasfull
    A trait alias: pub trait SharableIterator = Iterator + Sync.
  • ItemTypefull
    A type alias: type Result<T> = std::result::Result<T, MyError>.
  • ItemUnionfull
    A union definition: union Foo<A, B> { x: A, y: B }.
  • ItemUsefull
    A use declaration: use std::collections::HashMap.
  • Labelfull
    A lifetime labeling a for, while, or loop.
  • Lifetime
    A Rust lifetime: 'a.
  • LifetimeParamfull or derive
    A lifetime definition: 'a: 'b + 'c + 'd.
  • LitBool
    A boolean literal: true or false.
  • LitByte
    A byte literal: b'f'.
  • LitByteStr
    A byte string literal: b"foo".
  • LitCStr
    A nul-terminated C-string literal: c"foo".
  • LitChar
    A character literal: 'a'.
  • LitFloat
    A floating point literal: 1f64 or 1.0e10f64.
  • LitInt
    An integer literal: 1 or 1u16.
  • LitStr
    A UTF-8 string literal: "foo".
  • Localfull
    A local let binding: let x: u64 = s.parse()?.
  • LocalInitfull
    The expression assigned in a local let binding, including optional diverging else block.
  • Macrofull or derive
    A macro invocation: println!("{}", mac).
  • MetaListfull or derive
    A structured list within an attribute, like derive(Copy, Clone).
  • MetaNameValuefull or derive
    A name-value pair within an attribute, like feature = "nightly".
  • ParenthesizedGenericArgumentsfull or derive
    Arguments of a function path segment: the (A, B) -> C in Fn(A,B) -> C.
  • PatConstfull
    A const block: const { ... }.
  • PatIdentfull
    A pattern that binds a new variable: ref mut binding @ SUBPATTERN.
  • PatLitfull
    A literal in place of an expression: 1, "foo".
  • PatMacrofull
    A macro invocation expression: format!("{}", q).
  • PatOrfull
    A pattern that matches any one of a set of cases.
  • PatParenfull
    A parenthesized pattern: (A | B).
  • PatPathfull
    A path like std::mem::replace possibly containing generic parameters and a qualified self-type.
  • PatRangefull
    A range expression: 1..2, 1.., ..2, 1..=2, ..=2.
  • PatReferencefull
    A reference pattern: &mut var.
  • PatRestfull
    The dots in a tuple or slice pattern: [0, 1, ..].
  • PatSlicefull
    A dynamically sized slice pattern: [a, b, ref i @ .., y, z].
  • PatStructfull
    A struct or struct variant pattern: Variant { x, y, .. }.
  • PatTuplefull
    A tuple pattern: (a, b).
  • PatTupleStructfull
    A tuple struct or tuple variant pattern: Variant(x, y, .., z).
  • PatTypefull
    A type ascription pattern: foo: f64.
  • PatWildfull
    A pattern that matches any value: _.
  • Pathfull or derive
    A path at which a named item is exported (e.g. std::collections::HashMap).
  • PathSegmentfull or derive
    A segment of a path together with any path arguments on that segment.
  • PreciseCapturefull
    Precise capturing bound: the ‘use<…>’ in impl Trait + use<'a, T>.
  • PredicateLifetimefull or derive
    A lifetime predicate in a where clause: 'a: 'b + 'c.
  • PredicateTypefull or derive
    A type predicate in a where clause: for<'c> Foo<'c>: Trait<'c>.
  • QSelffull or derive
    The explicit Self type in a qualified path: the T in <T as Display>::fmt.
  • Receiverfull
    The self argument of an associated method.
  • Signaturefull
    A function signature in a trait or implementation: unsafe fn initialize(&self).
  • StmtMacrofull
    A macro invocation in statement position.
  • TraitBoundfull or derive
    A trait used as a bound on a type parameter.
  • TraitItemConstfull
    An associated constant within the definition of a trait.
  • TraitItemFnfull
    An associated function within the definition of a trait.
  • TraitItemMacrofull
    A macro invocation within the definition of a trait.
  • TraitItemTypefull
    An associated type within the definition of a trait.
  • Turbofish(full or derive) and printing
    Returned by TypeGenerics::as_turbofish.
  • TypeArrayfull or derive
    A fixed size array type: [T; n].
  • TypeBareFnfull or derive
    A bare function type: fn(usize) -> bool.
  • TypeGenerics(full or derive) and printing
    Returned by Generics::split_for_impl.
  • TypeGroupfull or derive
    A type contained within invisible delimiters.
  • TypeImplTraitfull or derive
    An impl Bound1 + Bound2 + Bound3 type where Bound is a trait or a lifetime.
  • TypeInferfull or derive
    Indication that a type should be inferred by the compiler: _.
  • TypeMacrofull or derive
    A macro in the type position.
  • TypeNeverfull or derive
    The never type: !.
  • TypeParamfull or derive
    A generic type parameter: T: Into<String>.
  • TypeParenfull or derive
    A parenthesized type equivalent to the inner type.
  • TypePathfull or derive
    A path like std::slice::Iter, optionally qualified with a self-type as in <Vec<T> as SomeTrait>::Associated.
  • TypePtrfull or derive
    A raw pointer type: *const T or *mut T.
  • TypeReferencefull or derive
    A reference type: &'a T or &'a mut T.
  • TypeSlicefull or derive
    A dynamically sized slice type: [T].
  • TypeTraitObjectfull or derive
    A trait object type dyn Bound1 + Bound2 + Bound3 where Bound is a trait or a lifetime.
  • TypeTuplefull or derive
    A tuple type: (A, B, C, String).
  • UseGlobfull
    A glob import in a use item: *.
  • UseGroupfull
    A braced group of imports in a use item: {A, B, C}.
  • UseNamefull
    An identifier imported by a use item: HashMap.
  • UsePathfull
    A path prefix of imports in a use item: std::....
  • UseRenamefull
    An renamed identifier imported by a use item: HashMap as Map.
  • Variadicfull
    The variadic argument of a foreign function.
  • Variantfull or derive
    An enum variant.
  • VisRestrictedfull or derive
    A visibility level restricted to some path: pub(self) or pub(super) or pub(crate) or pub(in some::module).
  • WhereClausefull or derive
    A where clause in a definition: where T: Deserialize<'de>, D: 'static.

Enums§

  • AttrStylefull or derive
    Distinguishes between attributes that decorate an item and attributes that are contained within an item.
  • BinOpfull or derive
    A binary operator: +, +=, &.
  • CapturedParamfull
    Single parameter in a precise capturing bound.
  • Dataderive
    The storage of a struct, enum or union data structure.
  • Exprfull or derive
    A Rust expression.
  • FieldMutabilityfull or derive
    Unused, but reserved for RFC 3323 restrictions.
  • Fieldsfull or derive
    Data stored within an enum variant or struct.
  • FnArgfull
    An argument in a function signature: the n: usize in fn f(n: usize).
  • ForeignItemfull
    An item within an extern block.
  • GenericArgumentfull or derive
    An individual generic argument, like 'a, T, or Item = T.
  • GenericParamfull or derive
    A generic type parameter, lifetime, or const generic: T: Into<String>, 'a: 'b, const LEN: usize.
  • ImplItemfull
    An item within an impl block.
  • ImplRestrictionfull
    Unused, but reserved for RFC 3323 restrictions.
  • Itemfull
    Things that can appear directly inside of a module or scope.
  • Lit
    A Rust literal such as a string or integer or boolean.
  • MacroDelimiterfull or derive
    A grouping token that surrounds a macro body: m!(...) or m!{...} or m![...].
  • Memberfull or derive
    A struct or tuple struct field accessed in a struct literal or field expression.
  • Metafull or derive
    Content of a compile-time structured attribute.
  • Patfull
    A pattern in a local binding, function signature, match expression, or various other places.
  • PathArgumentsfull or derive
    Angle bracketed or parenthesized arguments of a path segment.
  • PointerMutabilityfull
    Mutability of a raw pointer (*const T, *mut T), in which non-mutable isn’t the implicit default.
  • RangeLimitsfull
    Limit types of a range, inclusive or exclusive.
  • ReturnTypefull or derive
    Return type of a function signature.
  • StaticMutabilityfull
    The mutability of an Item::Static or ForeignItem::Static.
  • Stmtfull
    A statement, usually ending in a semicolon.
  • TraitBoundModifierfull or derive
    A modifier on a trait bound, currently only used for the ? in ?Sized.
  • TraitItemfull
    An item declaration within the definition of a trait.
  • Typefull or derive
    The possible types that a Rust value could have.
  • TypeParamBoundfull or derive
    A trait or lifetime used as a bound on a type parameter.
  • UnOpfull or derive
    A unary operator: *, !, -.
  • UseTreefull
    A suffix of an import tree in a use item: Type as Renamed or *.
  • Visibilityfull or derive
    The visibility level of an item: inherited or pub or pub(restricted).
  • WherePredicatefull or derive
    A single predicate in a where clause: T: Deserialize<'de>.

Functions§

  • parseparsing and proc-macro
    Parse tokens of source code into the chosen syntax tree node.
  • parse2parsing
    Parse a proc-macro2 token stream into the chosen syntax tree node.
  • parse_fileparsing and full
    Parse the content of a file of Rust code.
  • parse_strparsing
    Parse a string of Rust code into the chosen syntax tree node.

Type Aliases§

  • Result
    The result of a Syn parser.