Ray Cluster Observability: Logs Setup via Helm Chart. Part 2

This guide offers a detailed walkthrough for implementing observability in Ray clusters deployed through Helm charts, specifically focusing on log collection using Fluent Bit sidecars.

Overview

Ray clusters don’t store logs as container logs, which prevents standard DaemonSet Fluent Bit configurations from collecting them.

This creates a need for proper observability setup in Ray clusters to monitor performance, troubleshoot issues, and maintain operational health. The setup includes:

• Logging: Using Fluent Bit as a sidecar container to collect and forward Ray logs

If you haven’t read Ray Cluster Observability: Logs Setup via Helm Chart. Part 1, please review it first: https://medium.com/@jun.seo/ray-cluster-observability-logs-setup-via-helm-chart-part-1-78113f15c479

Prerequisites

• kuberay-operator installed in your Kubernetes environment
• Ray-cluster Helm chart downloaded

You can find these charts at the link below. Note that you must install kuberay-operator before installing ray-cluster.

https://github.com/ray-project/kuberay-helm

Update Ray Cluster Values Files

Using the ray-cluster values file, we’ll implement the logging configuration:

We previously created a fluent-bit-config as described in Ray Cluster Observability: Logs Setup via Helm Chart. Part 1.

First, let’s apply this fluent-bit-config to the worker section.

We’ll also add the log storage path /tmp/ray to the volumes and volumeMounts sections:

# <https://github.com/ray-project/kuberay-helm/blob/51711514ae948c9bea2c2440bdee3708f5625ee5/helm-chart/ray-cluster/values.yaml#L107>
worker:
  # sidecarContainers specifies additional containers to attach to the Ray pod.
  # Follows standard K8s container spec.
  sidecarContainers: []
  volumes:
    - name: fluent-bit-config
      configMap:
        name: fluent-bit-config
    - name: ray-logs
        emptyDir: {}
  volumeMounts:
    - name: ray-logs
      mountPath: /tmp/ray

Let’s create a Fluent Bit sidecar using the anchor and alias technique for easy reuse at the top of the values.yaml file.

Remember that our fluent-bit-config contains both fluent-bit.conf and custom_parsers.conf

fluent_bit_anchor: &fluent_bit_anchor
  name: fluent-bit
  image: fluent/fluent-bit:4.0
  volumeMounts:
    - name: ray-logs
      mountPath: tmp/ray/
    - mountPath: /fluent-bit/etc/fluent-bit.conf
      name: fluent-bit-config
      subPath: fluent-bit.conf
    - mountPath: /fluent-bit/etc/custom_parsers.conf
      name: fluent-bit-config
      subPath: custom_parsers.conf

The final implementation in the ray-cluster worker section will look like this. We can apply the same configuration to the head section as well.

# <https://github.com/ray-project/kuberay-helm/blob/51711514ae948c9bea2c2440bdee3708f5625ee5/helm-chart/ray-cluster/values.yaml#L107>
worker:
  # sidecarContainers specifies additional containers to attach to the Ray pod.
  # Follows standard K8s container spec.
  sidecarContainers:
    - <<: *fluentBitCommon
  volumes:
    - name: fluent-bit-config
      configMap:
        name: fluent-bit-config
    - name: ray-logs
        emptyDir: {}
  volumeMounts:
    - name: ray-logs
      mountPath: /tmp/ray

Referece

• https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/persist-kuberay-custom-resource-logs.html
• https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/