Next: About this document
An Overview of Computer System Performance Evaluation
- CSCI 6175 Seminar in Computer Science
Department of Computer Science
The University of Texas - Pan American
Edinburg, TX 78539-2999
- Is a 200 M Hz processor necessarily ``faster'' than a 133 M
- Or is a 1 BIPS processor necessarily ``faster'' than a 700
MIPS (0.7 BIPS) processor?
- Will a faster network necessarily send data through easier?
(for example 100 M bps Ethernet vs. 10 M bps Ethernet)
The answer to all three questions are ``not necessarily''. The first
on issues such as what kind of instructions are in question, how many
instructions can be in pipeline, etc.
The answer to last question depends on what kind of network protocol
is in use. For a bus-based protocol such as Ethernet, faster speed may
actually send less data through.
Computer systems performance evaluation is a collection of theories,
practices, sometimes ``arts'' to be used in analyzing computer systems
which include processors, primary memory, secondary memory, system
bus, networks, and others.
Outline of the Presentation
- Basic concepts
- The question of workload
- Performance metrics
- Three types of techniques
- Analytical Study
- What others are doing
- My work
The Question of Workload
- The term test workload denotes any workload used in
- To make any performance analysis valid, one has to consider
what kind of workload to be used in the study.
- A real workload is one observed on a system being used
for normal operations. Problem: can't repeat, thus not suitable for
- A synthetic workload is one whose characteristics are
similar to those of real workload and can be applied repeatedly in a
Table: Gibson Instruction Mix (quoted from Raj Jain's book page 49)
Typical performance metrics include the following.
- Delay :
- e.g. how long would it take to transfer
data from disk to memory?
- Response time :
- e.g. how long would it take for a job to get
- Throughput :
- e.g. how many jobs can be processed in a
given period of time?
- Utilization :
- e.g. how much time was the CPU used? How
much time was it idle?
Three Types Techniques
Typically, three different types of performance evaluation techniques
can be used (or a mix of these).
- Using software or hardware actually measure
predefined performance metrics.
- Statistical significance
- Results tend to be more accurate, more meaningful,
more directly applicable
- Expensive, difficult to repeat
Common benchmark programs
- Find all prime numbers in a given range,
- Ackermann's function:
- Calculate a particular numeric
value, computation intensive
Some values of A(x,y): ,
- Whetstone consists of a set of 11 modules
designed to match observed dynamic frequency of operations used in 949
ALGOL programs. (later it was translated into other programming
languages). The measurement unit is KWIPS (Kilo Whetstone Instructions
- Developed at Argonne National Lab in 1983, this
benchmark program consists of a number of programs that solve dense
systems of linear equations. The measurement unit is MFLOPS (Million
Floating Point Instructions per Second).
- Developed at Siemens in 1984, this set
contains many procedure calls and is considered to represent systems
programming environment. Available in C, Pascal, and Ada. The
measurement unit is Dhrystone Instructions Per Second (DIPS).
- Lawrence Livermore Loops:
- This contains a set of 24
separate tests dominated by large scientific calculations that are
largely vectorizable. The measurement unit is MFLOPS.
- Debit-Credit Benchmark:
- While above mentioned
benchmarks are computation-oriented, this is a transaction process
oriented. It represents a distributed banking network. The measurement
unit is Transactions Per Second (TPS).
- SPEC Benchmark Suite:
- Most recent, more widely used
benchmark program. The recent version include SPEC CPU95 (including
integer, floating etc.) and SPEC WEB96. The measurement unit is SPECmark.
- Here we are mainly concerned with statistical
simulation. A simulation is the process of designing a mathematical or
logical model of a real system and then conducting computer-based
experiments with the model to describe, explain, and predict the
behavior of the real system.
- Easy to vary the parameters
- Easy to repeat
- How to model the real system accurately
- Statistical significance
- How to interpret results (analysis of simulation results)
Types of simulation:
- Monte Carlo Simulation:
- A simulation without time
axis involved. For example, Buffon's needle experiment.
- Trace-Driven simulation:
- A simulation using a trace
as its input. A trace is a time-ordered record of events on a real
- Discrete-Event Simulation:
- A simulation using a
discrete-state model of the system.
- Analytical Study:
- Analytical performance study uses
mathematical model (often a queueing model) to represent the system
under study. The solution for the queueing model represents the
characteristics of a real system.
- Simple to perform (paper and pencil)
- Less accurate
- Sometimes a solution to the queueing model is
difficult to come up
Ethernet example: maximum throughput of Ethernet can be represented as
follows (see Lam80).
and is the propagation delay (time for signal to travel in
wire, often taken as per 1,000 meters) and T is the packet
For a Ethernet of 500 m length, 1500 bytes packet length
What Others Are Doing
From recent SIGMETRICS conference
- Network measurement and modeling: wide-area network, WWW
- Network resource management: queue management for rate based
congestion control, TCP over ATM
- Parallel computer systems: performance debugging shared
memory parallel programs
- Network support for video
- Network performance modeling
- Memory management: informed multiprocess prefetching and
caching, adaptive page replacement based on memory reference behavior
- Queueing model and simulation of metropolitan area network
with multimedia traffic (remote past)
- Queueing model and simulation of load sharing and
fault-tolerance computing systems (remote past)
- Measured performance of distributed simulations (recent past)
- Measured performance comparison of TCP/IP communications in
C++ and Java (recent)
- Simulation and (hopefully) queueing model of telephone center
where customers can take vacation (David Lung)
- Web page access pattern (to do)
- Web caching pattern (to do)
- Queuing model for WWW (to do)
Next: About this document
Sat Sep 27 15:13:58 CDT 1997