Custom Backend Metrics

Overview

Simple load balancing decisions can be made by taking into account local or global knowledge of a backend’s load, for example CPU. More sophisticated load balancing decisions are possible with application specific knowledge, e.g. queue depth, or by combining multiple metrics.

The custom backend metrics feature exposes APIs to allow users to implement the metrics feedback in their LB policies.

Use Cases

The feature is mainly for advanced use cases where a custom LB policy is used to route traffic more intelligently to a list of backend servers to improve the routing performance, e.g. a weighted round robin LB policy.

gRPC traditionally allows users to plug in their own load balancing policies, see guide. For xDS users, custom load balancer can be configured to select the custom LB policy.

Metrics Reporting

Open Request Cost Aggregation (ORCA) is an open standard for conveying backend metrics information. gRPC uses ORCA service and metrics standards and supports two metrics reporting mechanisms:

• Per-query metrics reporting: the backend server attaches the injected custom metrics in the trailing metadata when the corresponding RPC finishes. This is typically useful for short RPCs like unary calls.

• Out-of-band metrics reporting: the backend server periodically pushes metrics data, e.g. cpu and memory utilization, to the client. This is useful for all situations: unary calls, long RPCs in streaming calls, or no RPCs. However, out-of-band metrics reporting does not send query cost metrics. The metrics emission frequency is user-configurable, and this configuration resides in the custom load balancing policy.

The diagram shows the architecture where a user creates their own LB policy that implements backend metrics feedback.

Implementation

For more details, please see gRPC proposal A51.

Language Support