Annotations vs. Metadata: Making Traces Searchable

Attaching Custom Data to Traces

Out of the box, X-Ray records timing, HTTP status codes, and service names. But you often need to correlate traces with your business domain: "Show me all traces for Order #98765" or "Find every trace where the user was on the free tier."

X-Ray gives you two mechanisms for attaching custom data, and choosing the wrong one is a common mistake.

Annotations (Indexed — Searchable)

• Type: Key-value pairs. Keys are strings. Values must be String, Number, or Boolean.
• Indexed: Yes. X-Ray builds an index on annotations, making them filterable.
• Query syntax: annotation.UserID = "user-123" or annotation.OrderTotal > 500
• Use cases: Any identifier you will use to find traces — UserID, OrderID, TenantID, FeatureFlag, Region, PlanTier.
• Limit: Up to 50 annotations per trace.

from aws_xray_sdk.core import xray_recorder

segment = xray_recorder.current_segment()
segment.put_annotation('UserID', 'user-123')
segment.put_annotation('PremiumUser', True)
segment.put_annotation('OrderTotal', 149.99)

Metadata (Not Indexed — Readable Only)

• Type: Key-value pairs. Values can be any JSON-serializable object — arrays, nested objects, large strings.
• Indexed: No. You cannot filter or search by metadata values.
• Use cases: Debugging payloads you want to inspect after you find the trace — the full request body, the raw API response, a stack trace, a list of items in a cart.
• Limit: No strict size limit per field, but total segment size must stay under 64 KB.

segment.put_metadata('request_payload', {
    'items': ['item-1', 'item-2'],
    'shipping_address': '123 Main St',
    'raw_response': full_stripe_response_object
})

The Mental Model

Think of a filing cabinet. Annotations are the labels on the outside of each folder — you can scan the cabinet and pull out every folder labeled "PremiumUser=True" without opening any of them. Metadata is the documents inside the folder — rich, detailed, but you can only read them after you have already found and opened the right folder.

Senior Insight: A common mistake is storing large objects as annotations. Annotations are indexed, which means X-Ray must parse and store them in a search index. Large annotation values increase costs and can hit size limits. If you do not need to filter by a value, always use metadata. If you need to find traces by a value, use annotations — but keep the values small (IDs, flags, short strings).