Efficiently Running TensorFlow Serving on Dedicated Servers

Machine-learning models are powerful, but their real value comes when they’re reliably served in production. By running TensorFlow Serving on a dedicated server, you get full control over performance, scalability, and security, enabling your AI-enabled applications to handle real-time predictions with confidence.

Here is a brief and simple guide to the deployment process, tailored for servers running Ubuntu 22.04 or 24.04 (CPU or NVIDIA GPU).

1. Prepare Your Server

Make sure your system is up to date and secure, install essential packages and configure the firewall:

sudo apt update
sudo apt install ca-certificates curl jq unzip ufw -y
sudo ufw allow OpenSSH
sudo ufw enable

Create a central directory for your models (for example: /srv/tfmodels) and make sure it’s owned by your user.

Export your model in the correct format (for example a Keras SavedModel) and place it in a versioned sub-folder under your model's directory.

2. Install TensorFlow Serving

You have two main options:

Docker (recommended):

Using Docker gives flexibility and simplifies updates and GPU usage. Install Docker, pull the tensorflow/serving:latest image (or latest-gpu for GPU support) and run a container, mapping your model directory and exposing the correct ports (8500 for gRPC, 8501 for REST).

If you have an NVIDIA GPU, you’ll also need to set up the NVIDIA driver and container toolkit.

Native APT install:

If you prefer not to use Docker, you can install via APT. Add the official TensorFlow Serving repository, install tensorflow-model-server, and you’re ready.

3. Serve Multiple Models and Versions

In a production setting you’ll often want to serve several models or multiple versions of the same model.

Configure a models.config file that lists each model’s base path and version policy (for example “serve all versions”).

Then start TF Serving using that config, and it will monitor for new versions automatically.

4. Improve Throughput with Batching

For high traffic, enable server-side batching.

Create a batching.config file that defines parameters like max_batch_size, batch_timeout_micros, and num_batch_threads.

Then start TF Serving with --enable_batching=true --batching_parameters_file=/path/to/batching.config.

This improves throughput by grouping requests.

5. Tune Resource Usage

Adjust flags for your server’s CPU or GPU resources. For example:

--rest_api_num_threads=48 \
--tensorflow_intra_op_parallelism=8 \
--tensorflow_inter_op_parallelism=4

These settings help optimize latency and throughput depending on your hardware.

6. Add Monitoring (Prometheus Endpoint)

Monitoring is critical for production ML services. Enable the Prometheus endpoint so you can track request counts, latencies and errors.

Create a monitoring.config with:

prometheus_config {
  enable: true
  path: "/monitoring/prometheus/metrics"
}

Start TF Serving with --monitoring_config_file=/path/to/monitoring.config and you’ll expose the metrics for scraping.

7. Client Access: REST and gRPC

Once your server is up, clients can send prediction requests via REST or gRPC:

• REST: Send a POST request with JSON to http://your-server:8501/v1/models/your_model:predict.
• gRPC: Use the TensorFlow Serving gRPC API for lower latency and more efficient transport. Example code uses prediction_service_pb2_grpc and constructs a PredictRequest.

8. Secure the Setup

Don’t expose the internal serving ports publicly. Use Nginx as a reverse proxy bound to localhost and enable TLS via Certbot.

Also restrict the /monitoring/prometheus/metrics endpoint to internal network ranges and use firewall rules to block ports 8500/8501 externally.

9. Run as a Service (Optional)

Ensure your service starts on boot by using a systemd unit file.

For Docker-based installations, create a unit (for example tfs@.service) that runs the Docker container and enables automatic restarts.

Then enable the service for your model target.

10. Health Checks and Troubleshooting

Make sure everything is working properly before serving clients. You can query the model status using REST endpoints:

curl -s http://127.0.0.1:8501/v1/models/your_model | jq
curl -s http://127.0.0.1:8501/v1/models/your_model/versions/1 | jq

Common issues include model version errors, GPU detection problems, request timeouts, input/output mismatches, check logs (docker logs if using Docker) and verify your SavedModel structure.

Conclusion

Deploying TensorFlow Serving on a dedicated server gives you the flexibility and performance needed for production-grade machine-learning services.

With the steps above, from preparation and installation to monitoring and security, you can set up a reliable, scalable environment for serving models.

Tip: For a detailed setup and information, you can check this guide on **Serve Machine Learning Models with TensorFlow Serving**.