Introduction to Benchmarking, Performance Measurement, and Benchmark Types

The term benchmark is heavily overloaded in computer architecture and can be used as both a noun and a verb. The professor defined it in a few distinct ways:

• As a Noun (Code): A specific body of code or program used to determine performance.
• As a Noun (Value): The actual numerical value, metric, or measure produced by running that code (e.g., 1 million operations per second).
• As a Verb: The actual process of running the code or analyzing the numbers it produces (“benchmarking”).

Introduction to Benchmarking, Performance Measurement, and Benchmark Types

The Basis of Measurement: Execution Time

The fundamental base metric for computational performance is execution time. Because performance is defined as 1 / Execution Time, a larger performance number equates to a smaller (faster) execution time. However, there is an inherent tension in how execution time is viewed and measured:

• The Consumer’s Perspective: Users want to know exactly how long their specific, realistic workloads (like a CAD package or spreadsheet) will take. They want the most realistic number possible.
• The Vendor’s Perspective: Hardware producers want to report the absolute best, most optimal performance number to drive sales, which can sometimes lead to disabling operating system features or cherry-picking scenarios to artificially inflate the score.

Choices for Measuring Execution Time:

• Wall Clock (Elapsed) Time: Measures the total time from start to finish. This is heavily viewed as unreliable for architectural measurement because it includes overhead for background OS tasks, other running apps, and I/O wait times.
• CPU Time: The time the CPU spends actively working on our task alone. This is preferred and consists of two subcomponents:
• User CPU Time: The time spent executing the actual instructions you wrote.
• System CPU Time: The time the CPU spends executing OS or library tasks directly on behalf of your program (e.g., dynamic heap allocations, math libraries computing sine/cosine, or printing to the console).

Ground Rule for Measurement: To isolate true CPU performance and prevent the OS from corrupting the timing (e.g., via hard drive defragmentation or virtual memory swaps), benchmarks must be measured on an unloaded system.

Best Practice: Performance Multiple: When reporting comparative performance, best practice dictates reporting it as a performance multiple (e.g., “Machine X is 2 times as fast as Machine Y”) rather than confusing percentage increases. This means Performance of X / Performance of Y = n.

Benchmark Methods and Suites

To evaluate hardware, architects use various levels of program complexity. Each method has different levels of manipulability:

1. Real Program Execution: Running complete, well-known programs (e.g., compiling the GNU C Compiler). Scripted Execution of a Real Program: Using a real program but controlling it with a fixed, scripted input, so testing is entirely uniform across different testers.
2. Program Kernel Execution: Extracting and running just the core computational section of a real program (e.g., the LINPACK benchmark for matrix operations in linear algebra).
3. Synthetic Benchmarks: Artificial programs designed to mimic the instruction mix of real applications without actually performing any recognizable or useful task (e.g., Dhrystone and Whetstone).
4. Toy/Novelty Benchmarks: Recognizable, functional tasks that are not typically done professionally or represent realistic standalone workloads (e.g., POV-Ray graphic rendering, Quicksort, or Quake frames-per-second).
5. Suite of Programs: Because single benchmarks can be easily “gamed” by vendors (e.g., hardcoding a special compiler flag for a specific divide operation), best practice relies on a benchmark suite — a collection of diverse programs that average out weaknesses and limit manipulability. Example: SPEC CPU (Standard Performance Evaluation Corporation). SPEC suites (like SPEC89, SPEC2006, SPEC2017) contain separate modules for integer (SPEC CPU int) and floating-point (SPEC CPU FP) applications.

The Importance of Reproducibility: Organizations like SPEC enforce high standards by making results entirely reproducible. Submissions must document the exact memory size, cache, OS version, compiler flags, and clock speeds so anyone can verify the results.

Combination Techniques and Normalization

When using a benchmark suite, we end up with dozens of performance numbers that must be boiled down to a single value.

1. Arithmetic Mean: (Sum of times / n). Advantage: Smooths out runtime variations/noise. Disadvantage: Heavily skewed by the longest-running task in the suite.
2. Weighted Arithmetic Mean: Applies a specific weight (W_i) to each program’s time. Advantage: Can normalize tasks of vastly different magnitudes. Disadvantage: The weights are somewhat arbitrary/empirical, and standard weights can quickly become biased or obsolete as computer architectures evolve.
3. Geometric Mean and Normalization: The most useful combination method for the benchmarking community. A geometric mean is the nth root of the product of the performances.

• Advantage: It is self-normalizing. No single factor or execution time can overemphasize the result, bypassing the need for arbitrary weights.
• Normalization Factor: Results are typically normalized against a well-known reference machine (like an older VAX or Sun SPARC workstation).
• Simple Example: Suppose Program 1 takes 5s on CPU A and 20s on a VAX. Program 2 takes 50s on CPU A and 150s on a VAX. First, we compute the normalized performance multiples: Program 1 is 4x faster (20/5), and Program 2 is 3x faster (150/50). We combine them with a geometric mean: sqrt(4 * 3) = sqrt(12) ≈ 3.46. We can confidently say CPU A is 3.46 times faster than the VAX.

Speedup and Amdahl’s Law

Speedup is the dimensionless ratio of Time Old (without enhancement) / Time New (with enhancement).

Amdahl’s Law, coined by Gene Amdahl, acts as our “speed limit”. The professor defined it strictly as: “Speed up is limited by the amount of the task that is improvable.” It reminds architects that enhancing a single feature will only speed up the fraction of the code that actively uses that feature.

• Example: A program takes 10 seconds to run on an old CPU, and 4 seconds of that time is spent doing divides. If we introduce a massive enhancement that makes divides 5x faster, the new divide time is 0.8 seconds. The overall new time is 6s (un-enhanced time) + 0.8s = 6.8s. The overall speedup is 10 / 6.8 ≈ 1.47 (or roughly 1.5x).
• The Speed Limit: Even if our new divide instruction executed instantaneously (0 seconds), the best possible overall time is 6s. Thus, our theoretical maximum speedup limit is 10 / 6 ≈ 1.67x.

The CPU Performance Equation (The Iron Law)

CPU time is determined by three equally important parameters. **CPU Time = Instruction Count (IC) * Clocks per Instruction (CPI) * Clock Cycle Time (CCT)**.

1. CCT (Clock Cycle Time): The time between clock pulses (the inverse of clock rate/speed). Regulates how fast the hardware physically switches.
2. IC (Instruction Count): The exact number of instructions that actually started and finished execution (not just lines of source code). Crucial for loops where a single line of code might execute thousands of times.
3. CPI (Clocks per Instruction): The average number of clock cycles it takes for instructions to execute.

Because these parameters are multiplicative, a 10% improvement in any of the three yields an identical 10% improvement in CPU time.

CISC vs. RISC Focus:

• CISC Focus (e.g., VAX): CISC architectures attempted to optimize almost entirely by minimizing the Instruction Count (IC). They built overly complex instructions to try to do massive amounts of work in a single line of assembly.
• RISC Focus: RISC architectures realized that overly complex instructions severely penalized the other two metrics. By heavily restricting instructions (e.g., forcing ALU operations to only use register operands), RISC dramatically lowers CPI (simplifying pipelines/hazards) and enables a much faster CCT (simplified circuitry allows the clock to pulse faster). Thus, RISC successfully attacks two out of the three metrics simultaneously.

Measuring Instruction Count (IC): Since IC evaluates runtime execution, it cannot simply be read from the source code. It is measured via:

1. Hardware Counters: Special, often hidden, performance registers built directly into the CPU that tally executed instructions in real-time.
2. Simulation: Writing software programs that mock up the CPU’s behavior, feeding the code into it, and keeping a software count of every instruction fetched and executed

🔙 Back to all notes 𝕏 Let’s Connect

Backlinks: *Computer Architecture Notes: A Quantitative Guide to Modern Computing*