Ray Architecture: The Object Store and Cluster Internals

How Ray Is Structured

Understanding Ray's internals helps you debug performance issues and design efficient distributed systems. A Ray cluster has two types of nodes:

• Head Node: Runs the Global Control Store (GCS), which tracks cluster state (which nodes are alive, which actors exist, resource availability). Also runs the driver script.
• Worker Nodes: Run the actual tasks and actors. Each node runs a Raylet — a local scheduler and object manager daemon.

Zero-Copy Performance with Plasma

Passing large data between processes is normally expensive. Serializing (pickling) a 1 GB NumPy array, sending it over a socket, and deserializing it on the other side can take seconds and doubles your memory usage.

Ray uses the Plasma Object Store, a shared-memory system embedded in each node:

1. Same-node access: When Task A produces a large NumPy array and Task B (on the same node) needs it, Task B reads the memory directly via a memory-mapped file. No copying, no serialization. This is called zero-copy deserialization and is enabled by the Apache Arrow columnar format that Ray uses internally.

2. Cross-node access: If Task B is on a different node, Ray's object manager detects this, transfers the serialized object over the network to the destination node's Plasma store, and caches it there. Subsequent tasks on that node reuse the cached copy.

3. Spilling to disk: If the object store fills up (default is 30% of available RAM), Ray automatically spills objects to disk and reloads them on demand — similar to virtual memory, but for distributed objects.

This design makes Ray exceptionally efficient for ML workloads where you're constantly passing large tensors between preprocessing, training, and evaluation steps.

The Scheduling Pipeline

When you call square.remote(5), here is what happens:

1. The driver submits the task to the local Raylet on the head node.
2. The local Raylet checks if it has capacity. If yes, it runs the task locally. If not, it forwards to the GCS.
3. The GCS finds a worker node with sufficient resources (CPUs, GPUs, memory).
4. The task is sent to that node's Raylet, which assigns it to a worker process.
5. The worker executes the function, writes the result to its local Plasma store, and registers the ObjectRef with the GCS.
6. When you call ray.get(ref), Ray fetches the object from wherever it lives.