next up previous
Next: About this document

An Overview of Computer System Performance Evaluation
- CSCI 6175 Seminar in Computer Science

Xiannong Meng
Department of Computer Science
The University of Texas - Pan American
Edinburg, TX 78539-2999

Fall 1997

Opening Questions

The answer to all three questions are ``not necessarily''. The first two depend on issues such as what kind of instructions are in question, how many instructions can be in pipeline, etc.

The answer to last question depends on what kind of network protocol is in use. For a bus-based protocol such as Ethernet, faster speed may actually send less data through.

Computer systems performance evaluation is a collection of theories, practices, sometimes ``arts'' to be used in analyzing computer systems which include processors, primary memory, secondary memory, system bus, networks, and others.

Outline of the Presentation

The Question of Workload

Table: Gibson Instruction Mix (quoted from Raj Jain's book page 49)

Performance Metrics

Typical performance metrics include the following.

Delay :
e.g. how long would it take to transfer data from disk to memory?
Response time :
e.g. how long would it take for a job to get started?
Throughput :
e.g. how many jobs can be processed in a given period of time?
Utilization :
e.g. how much time was the CPU used? How much time was it idle?

Three Types Techniques

Typically, three different types of performance evaluation techniques can be used (or a mix of these).

Using software or hardware actually measure predefined performance metrics.


Common benchmark programs

Find all prime numbers in a given range, computation intensive
Ackermann's function:
Calculate a particular numeric value, computation intensive


Some values of A(x,y): tex2html_wrap_inline120 , tex2html_wrap_inline122

Whetstone consists of a set of 11 modules designed to match observed dynamic frequency of operations used in 949 ALGOL programs. (later it was translated into other programming languages). The measurement unit is KWIPS (Kilo Whetstone Instructions Per Second).

Developed at Argonne National Lab in 1983, this benchmark program consists of a number of programs that solve dense systems of linear equations. The measurement unit is MFLOPS (Million Floating Point Instructions per Second).

Developed at Siemens in 1984, this set contains many procedure calls and is considered to represent systems programming environment. Available in C, Pascal, and Ada. The measurement unit is Dhrystone Instructions Per Second (DIPS).

Lawrence Livermore Loops:
This contains a set of 24 separate tests dominated by large scientific calculations that are largely vectorizable. The measurement unit is MFLOPS.

Debit-Credit Benchmark:
While above mentioned benchmarks are computation-oriented, this is a transaction process oriented. It represents a distributed banking network. The measurement unit is Transactions Per Second (TPS).

SPEC Benchmark Suite:
Most recent, more widely used benchmark program. The recent version include SPEC CPU95 (including integer, floating etc.) and SPEC WEB96. The measurement unit is SPECmark.

Here we are mainly concerned with statistical simulation. A simulation is the process of designing a mathematical or logical model of a real system and then conducting computer-based experiments with the model to describe, explain, and predict the behavior of the real system.


Types of simulation:

Monte Carlo Simulation:
A simulation without time axis involved. For example, Buffon's needle experiment.
Trace-Driven simulation:
A simulation using a trace as its input. A trace is a time-ordered record of events on a real system.
Discrete-Event Simulation:
A simulation using a discrete-state model of the system.

Analytical Study:
Analytical performance study uses mathematical model (often a queueing model) to represent the system under study. The solution for the queueing model represents the characteristics of a real system.


Ethernet example: maximum throughput of Ethernet can be represented as follows (see Lam80).




and tex2html_wrap_inline124 is the propagation delay (time for signal to travel in wire, often taken as tex2html_wrap_inline126 per 1,000 meters) and T is the packet transmitting time.


For a Ethernet of 500 m length, 1500 bytes packet length


What Others Are Doing

From recent SIGMETRICS conference

My Work

Any takers?

next up previous
Next: About this document

Xiannong Meng
Sat Sep 27 15:13:58 CDT 1997