Event Source Mappings: The Poller Fleet

Push vs. Pull Invocation Models

Lambda can be invoked in two fundamentally different ways, and the distinction matters for error handling, scaling, and cost:

Push model: The event source directly invokes Lambda. API Gateway, SNS, S3, and EventBridge all work this way. When an HTTP request hits API Gateway, API Gateway calls lambda:InvokeFunction immediately. Lambda scales instantly to match the incoming request rate.

Pull model: The event source holds data in a queue or stream, and something must actively read from it. SQS queues and Kinesis streams don't call Lambda — they just accumulate records. Lambda must poll them.

The Event Source Mapping

An Event Source Mapping (ESM) is an AWS-managed polling process that bridges the gap between pull-based sources and Lambda. You configure it once, and AWS runs the poller infrastructure on your behalf — you never see it, you don't pay for it directly, and you don't manage it.

The ESM lifecycle for SQS:

1. Poll: The ESM continuously long-polls your SQS queue using ReceiveMessage API calls.
2. Batch: It collects up to BatchSize messages (configurable from 1 to 10,000 for standard queues).
3. Invoke: It calls your Lambda function synchronously with a batch of messages as the event payload.
4. Acknowledge or retry: If your function returns successfully, the ESM deletes the messages from the queue. If your function throws an error, the messages remain in the queue and become visible again after the visibility timeout — triggering a retry.

// Example event payload delivered to your handler by the ESM
{
  "Records": [
    {
      "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
      "receiptHandle": "AQEBwJnKyrHigUMZj6reyasLE54BvyOiEETs...",
      "body": "{\"order_id\": \"12345\", \"amount\": 99.99}",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1545082650636"
      },
      "messageAttributes": {},
      "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:orders-queue",
      "awsRegion": "us-east-1"
    }
  ]
}

Creating one is a single configuration call linking a function to a queue ARN, along with the batch size and batching window you want the poller to use.

Partial Batch Failure — The Critical Detail

By default, if your Lambda function throws an error while processing a batch of 10 messages, all 10 messages are retried — even if 9 of them were processed successfully. This causes duplicate processing of the successful messages.

The fix is ReportBatchItemFailures. Enable this on your ESM, and return a structured response from your handler identifying which specific message IDs failed:

def handler(event, context):
    failed_message_ids = []

    for record in event['Records']:
        try:
            process_order(record['body'])
        except Exception as e:
            print(f"Failed to process message {record['messageId']}: {e}")
            failed_message_ids.append({"itemIdentifier": record['messageId']})

    # Only the failed messages are retried; successful ones are deleted
    return {
        "batchItemFailures": failed_message_ids
    }

Enabling it is a single update to the mapping's configuration, turning on the ReportBatchItemFailures function-response type.

Senior depth — Kinesis vs. SQS ESM behavior: For Kinesis streams, the ESM processes records from each shard sequentially (to preserve ordering). A single failed record blocks the entire shard until it succeeds or expires. This is why ReportBatchItemFailures and a properly configured bisectBatchOnFunctionError (which splits the failing batch in half to isolate the poison pill) are critical for Kinesis-based pipelines. SQS standard queues have no ordering guarantee, so failed messages simply return to the queue without blocking others.