TensorFlow Serving by Example: Part 3

Beginning to explore monitoring models deployed to a Kubernetes cluster.

This is a continuation of a series of articles that start with *TensorFlow Serving by Example: Part 1*.

Basic GPU Workload Metrics

From *Monitor your GPU node workload performance the section in the document Run GPUs in GKE Standard node pools*.

If your GKE cluster has system metrics enabled, then the following metrics are available in Cloud Monitoring to monitor your GPU workload performance:

Duty Cycle (container/accelerator/duty_cycle): Percentage of time over the past sample period (10 seconds) during which the accelerator was actively processing. Between 1 and 100.

Memory Usage (container/accelerator/memory_used): Amount of accelerator memory allocated in bytes.

Memory Capacity (container/accelerator/memory_total): Total accelerator memory in bytes.

These metrics apply at the container level (container/accelerator) and are not collected for containers scheduled on a GPU that uses GPU time-sharing or NVIDIA MPS.

While the memory metrics are fairly easy to interpret, the duty cycle is bit more challenging; courtesy of Google AI overview.

At a low level, a GPU’s processing units (such as NVIDIA’s Streaming Multiprocessors or AMD’s Compute Units) are constantly running on clock cycles. The GPU duty cycle measures how many of those clock cycles are actually being used for computation.

note: The NVIDIA Tesla T4 GPU operates at 585 MHz (or 585,000,000 clock cycles per second).

Google AI continues.

The duty cycle is not a measure of how efficiently the GPU is running or how much computational work is being done. A GPU can be at 100% duty cycle while performing very little work if its cores are being inefficiently utilized. It also does not measure memory bandwidth usage or the amount of work in the processing queue.

The Load Test

In order to meaningfully observe the basic GPU workload metrics in action, we need to put a load on our inference workload; i.e. the linear-regression deployment from the last article.

Here we can start with a basic pod with sufficient CPU and memory.

apiVersion: v1
kind: Pod
metadata:
  name: debug
  namespace: default
spec:
  containers:
  - name: debug
    image: ubuntu:latest
    command: ["/bin/bash"]
    args: ["-c", "while true; do sleep 30; done"]
    resources:
      requests:
        memory: "1G"
        cpu: "500m"
      limits:
        memory: "1G"
        cpu: "500m"

note: During the tests, did ensure that this pod was sufficiently resourced to perform the load tests; CPU request utilization was less than 5% and memory usage was negligible.

We exec into the pod and install the *wrk apt package. We have to write a tiny Lua script, post_script.lua, *to use the workload’s inference API.

wrk.method = "POST"
wrk.body = "{ \"instances\": [ [2.093837833] ] }"
wrk.headers["Content-Type"] = "application/json"

The following command will use the Lua script and a single thread that opens up 10 simultaneous connections, waits for all the responses, and then repeats the process for 5 minutes.

$ wrk -t1 -c10 -d300s -s post_script.lua http://linear-regression:8501/v1/models/linear-regression:predict

Testing the linear-regression Deployment

The results of the load test on the linear-regression deployment gave us an average of 740 RPS (requests per second) with and average of 18 ms latency.

$ wrk -t1 -c10 -d300s -s post_script.lua http://linear-regression:8501/v1/models/linear-regression:predict
Running 5m test @ http://linear-regression:8501/v1/models/linear-regression:predict
  1 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    17.61ms   17.16ms  87.62ms   81.29%
    Req/Sec   744.85     65.64     1.24k    88.06%
  222431 requests in 5.00m, 31.82MB read
Requests/sec:    741.39
Transfer/sec:    108.60KB

Here we look at both the GPU duty cycle (blue) and the CPU request (green) utilization.

Observations:

• Here we see GPU duty cycle (blue) is flatlined at 0%; did look a the logs of the container to confirm that it was indeed using the GPU
• Here we see the CPU was hitting 100% utilization; while we only requested 300m CPU, between the allocatable (940m) CPU and the required daemonsets, the n1-standard-1 (single CPU) node was already at 83% allocation

The conclusion here is that because our linear regression model is so simple, it does not require much effort by the GPU where-as the CPU is fully loaded handling the overhead work, i.e., accepting the HTTP request, sending the work to the GPU, getting the results from the GPU, and sending the HTTP response.

For comparison, we can run the same model without the GPU (here running on the e2-medium node with the same 300m CPU request).

note: Even with 2 CPU, the e2-medium node interestingly only shows 940m allocatable; we really could not give it much more CPU than this.

While we again fully loaded the CPU, we see we are getting an average of 645 RPS (requests per second) and average of 30 ms latency.

$ wrk -t1 -c10 -d300s -s post_script.lua http://linear-regression:8501/v1/models/linear-regression:predict
Running 5m test @ http://linear-regression:8501/v1/models/linear-regression:predict
  1 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.04ms   30.88ms 106.85ms   80.91%
    Req/Sec   648.61     55.27     1.18k    75.85%
  193689 requests in 5.00m, 27.71MB read
Requests/sec:    645.61
Transfer/sec:     94.57KB

This compares fairly well to the average of 740 RPS (requests per second) with and average of 18 ms latency when we used the GPU.

Sidebar into Pricing

Let us look at the price comparison between the two scenarios.

Running on the N1 (1 CPU) + NVIDIA Tesla T4 (1 GPU) is about $0.40 / hour.

• n1-standard-1: $0.04749975 / hour
• NVIDIA Tesla T4 GPU $0.35 / hour

Where as the e2-medium is $0.03350571 / hour.

Conclusion with roughly the same performance between the two scenarios but 10x the price difference, it makes sense to run this particular model using CPUs only.

A Different Model

Because we want to see the duty cycle metric using more than 0%, we switch to using a more complex model; a multi-class classification model described in *Keras by Example (Part 5).*

The exercise we are doing here is the classic “hello world” machine learning exercise consisting of training on images of numbers between 0 and 9 and then predicting the number using the Modified National Institute of Standards and Technology database (MNIST) dataset.

The model’s parameters can be downloaded here&project=skillful-figure-459619-t4); the multi-class-classification.tar file.

Using the model’s parameters and an updated models.config file, we can follow the steps done in the previous article to create and store a custom image for this model.

model_config_list {
  config {
    name: 'multi-class-classification'
    base_path: '/models/multi-class-classification'
    model_platform: 'tensorflow'
    model_version_policy {
      specific {
        versions: 1
      }
    }
  }
}

The deployment and service are essentially the same as before; just changed up the container image and the names.

The Updated Load Test

Here we use the same approach as before, we just need to change up the request body for test; here passing in a 26 x 26 x 1 array of floats (this one is an image of a 2).

wrk.method = "POST"
wrk.body = "{ \"instances\": [[[[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.09803921568627451], [0.16862745098039217], [0.15294117647058825], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.1411764705882353], [0.8901960784313725], [0.9882352941176471], [0.9686274509803922], [0.5568627450980392], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.24313725490196078], [0.9137254901960784], [0.9098039215686274], [0.5294117647058824], [0.7725490196078432], [0.9450980392156862], [0.9921568627450981], [0.9882352941176471], [0.9882352941176471], [0.9686274509803922], [0.7372549019607844], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.06274509803921569], [0.8666666666666667], [0.9921568627450981], [0.9882352941176471], [0.9882352941176471], [0.9882352941176471], [0.8862745098039215], [0.32941176470588235], [0.32941176470588235], [0.8313725490196079], [0.9882352941176471], [0.7137254901960784], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.21176470588235294], [0.7294117647058823], [0.9882352941176471], [0.9921568627450981], [0.9882352941176471], [0.6078431372549019], [0.4745098039215686], [0.09411764705882353], [0.0], [0.0], [0.8431372549019608], [0.9882352941176471], [0.5764705882352941], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.5019607843137255], [0.9921568627450981], [0.9921568627450981], [0.7607843137254902], [0.10588235294117647], [0.0], [0.0], [0.0], [0.0], [0.0], [0.9137254901960784], [0.9921568627450981], [0.5058823529411764], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.3333333333333333], [0.6588235294117647], [0.48627450980392156], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.39215686274509803], [0.9686274509803922], [0.9882352941176471], [0.16470588235294117], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.7450980392156863], [0.9882352941176471], [0.7254901960784313], [0.01568627450980392], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.21568627450980393], [0.9098039215686274], [0.9882352941176471], [0.32941176470588235], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.17647058823529413], [0.9254901960784314], [0.9882352941176471], [0.7450980392156863], [0.054901960784313725], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.16862745098039217], [0.4823529411764706], [0.788235294117647], [0.9921568627450981], [0.9921568627450981], [1.0], [0.9921568627450981], [0.7843137254901961], [0.5803921568627451], [0.9254901960784314], [1.0], [0.9372549019607843], [0.24313725490196078], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.32941176470588235], [0.7450980392156863], [0.9921568627450981], [0.9882352941176471], [0.9882352941176471], [0.9882352941176471], [0.9882352941176471], [0.9921568627450981], [0.9882352941176471], [0.9882352941176471], [0.9882352941176471], [0.9882352941176471], [0.9921568627450981], [0.34901960784313724], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.3215686274509804], [0.9647058823529412], [0.9882352941176471], [0.9098039215686274], [0.9058823529411765], [0.5254901960784314], [0.49411764705882355], [0.14901960784313725], [0.08235294117647059], [0.592156862745098], [0.9882352941176471], [0.9882352941176471], [0.9882352941176471], [0.9921568627450981], [0.24705882352941178], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.32941176470588235], [0.9647058823529412], [0.9882352941176471], [0.6078431372549019], [0.0], [0.0], [0.0], [0.0], [0.0], [0.49411764705882355], [0.9098039215686274], [0.9882352941176471], [0.9882352941176471], [0.9882352941176471], [0.9921568627450981], [0.6039215686274509], [0.023529411764705882], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.7450980392156863], [0.9882352941176471], [0.6078431372549019], [0.027450980392156862], [0.0], [0.0], [0.0], [0.10588235294117647], [0.5882352941176471], [0.9921568627450981], [0.9882352941176471], [0.7803921568627451], [0.30196078431372547], [0.6823529411764706], [0.9921568627450981], [0.9882352941176471], [0.42745098039215684], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.23921568627450981], [0.9921568627450981], [0.8392156862745098], [0.0], [0.0], [0.0], [0.12549019607843137], [0.3764705882352941], [0.8549019607843137], [0.9921568627450981], [0.9686274509803922], [0.5137254901960784], [0.20784313725490197], [0.0], [0.0], [0.9294117647058824], [0.9921568627450981], [0.7490196078431373], [0.054901960784313725], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.7176470588235294], [0.9882352941176471], [0.49411764705882355], [0.0], [0.1411764705882353], [0.403921568627451], [0.9294117647058824], [0.9882352941176471], [0.9882352941176471], [0.7137254901960784], [0.20784313725490197], [0.0], [0.0], [0.0], [0.0], [0.21568627450980393], [0.9058823529411765], [0.9882352941176471], [0.6392156862745098], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.9921568627450981], [0.9882352941176471], [0.7411764705882353], [0.7725490196078432], [0.9450980392156862], [0.9921568627450981], [0.9607843137254902], [0.8705882352941177], [0.2196078431372549], [0.01568627450980392], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.6823529411764706], [0.984313725490196], [0.8666666666666667], [0.2901960784313726], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.611764705882353], [0.9882352941176471], [0.9882352941176471], [0.8235294117647058], [0.5686274509803921], [0.32941176470588235], [0.2196078431372549], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.6313725490196078], [0.9882352941176471], [0.4745098039215686], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.027450980392156862], [0.16470588235294117], [0.16470588235294117], [0.054901960784313725], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.08235294117647059], [0.16470588235294117], [0.027450980392156862], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]], [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]]]] }"
wrk.headers["Content-Type"] = "application/json"

Testing the multi-class-classification Deployment

We perform the same test as we did earlier; gave us an average of 488 RPS (requests per second) with and average of 22 ms latency.

$ wrk -t1 -c10 -d300s -s post_script.lua http://multi-class-classification:8501/v1/models/multi-class-classification:predict
Running 5m test @ http://multi-class-classification:8501/v1/models/multi-class-classification:predict
  1 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    22.85ms   36.61ms   1.06s    99.45%
    Req/Sec   488.43     46.33   797.00     86.68%
  145413 requests in 5.00m, 38.14MB read
Requests/sec:    484.65
Transfer/sec:    130.16KB

Much like the previous example, it is the CPU that is the limiting factor; but we at least see the GPU duty cycle metric reaching 1% (still barely used).

Here go a bit further and look at the GPU memory usage and capacity (in the form of a ratio) and see that this workload is using 90% of the GPU memory (this seems a bit unusual to me). As expected, the memory use of the container itself is negligible (like 5%).

Again, it turns out that it makes sense to run this particular model using CPUs only.

Next Steps

In the next article, *TensorFlow Serving by Example: Part 4*, we continue to explore monitoring models deployed to a Kubernetes cluster.