CSE 221 Fall 2004 Programming Project: System Measurement

Due at the beginning of the last class meeting.

Overview

In building an operating system, it is important to be able to determine the performance characteristics of underlying hardware (CPU, RAM, disk, network, etc.), and to understand how their performance influences or constrains operating system services. Likewise, in building an application, one should understand the performance of the underlying hardware and operating system, and how they relate to the user's subjective sense of that application's "responsiveness". While some of the relevant quantities can be found in specs and documentation, many must be determined experimentally. While some values may be used to predict others, the relations between lower- and higher-level performance are often subtle and non-obvious.

In this project, you will create, justify, and apply a set of experiments to a system to characterize and understand its performance. In addition, you may explore the relations between some of these quantities. In doing so, you will study how to use benchmarks to usefully characterize a complex system. You should also gain an intuitive feel for the relative speeds of different basic operations, which is invaluable in identifying performance bottlenecks.

You may work either alone or in two-person groups. If you work alone, you will be responsible only for the system measurement part of the project. If you work with a partner, you will be required to do both parts. The group project is intended to require about as much time as two individual projects. However, since some of the operations are serialized, it may take somewhat longer to finish, so you should start early.

Machine Description

Your report should contain a reasonably detailed description of the test machine(s). The relevant information should be available either from the system (e.g. sysctl on BSD, /proc on Linux, System Profiler on Mac OS X), or online. You will not be graded on this part, and it should not require much work, but in explaining and analyzing your results, you will find these numbers useful; for example, you might compare advertised and observed RAM bandwidths. You should report at least the following quantities:
  1. Processor: model, cycle time, cache sizes (L1, L2, instruction, data, etc.).
  2. Bus speed.
  3. RAM size.
  4. Disk: size, RPM, controller cache size.
  5. Network card speed.
  6. Operating system (including version/release)

Properties to Measure and/or Report

Both individuals and groups should do this part. Individuals may omit some parts, noted below.

For each property, you should clearly explain the experiment you perform, describe how you intend it to measure the quantity of interest, and evaluate its success. In all cases, you should run your experiment multiple times, for long enough to obtain repeatable measurements, and average the results. Often, the property of interest is an interesting function data size; in such cases (noted below), you should plot a curve showing time as a function of size, and explain any interesting features of this curve.

  1. Scheduling and OS services
    1. System call overhead
      i.e. what is the cost of a minimal system call?
    2. Task creation time
      If available, compare processes to system threads.
    3. Time to write-protecting a segment
      Report as a function of segment size.
  2. Virtual Memory
    1. RAM access time
      Note: you will need to account for L1 and L2 caches; explain how.
    2. RAM bandwidth (both read and write)
    3. Page fault service time
  3. Network
    For each quantity, you should compare remote and loopback interfaces, and compare TCP (stream) and (groups only) UDP (packet) protocols. For your remote tests, either provide a machine description for the second machine (as above), or use two identical machines.
    1. Connection setup and tear-down.
    2. Latency
    3. Peak bandwidth
  4. File System
    1. Size of file cache
      Note: this may be very sensitive to other load on the machine.
    2. Sequential and random-access read time vs. size.
      Note: Accurate local measurement may require root access to a Unix machine. If you do not have this, use an ordinary file as in the remote case, and explain the sense in which your "sequential" access might not be sequential.
    3. (groups only) Synchronous and asynchronous (sequential) write time vs. size.
    4. Time to create, fill, and delete a file vs. size

Application Performance

Only groups should do this part.

From the point of the user of a system, a key question is how the individual operations measured above relate to application-level performance. Given the above measurements, one wants to predict how user tasks will scale with hardware size. To answer this question, you will measure the performance of three different application-level operations as a function of size. Before running each test, predict which of the quantities measured above will have a significant effect. Afterwards, state whether or not your predictions were correct, and if they were not, explain which factors turned out to dominate.

  1. Application startup (interactive responsiveness)
    Choose several applications of varying size and construction (e.g. echo, vi, Emacs, Netscape). For each application, measure the time it takes to start and immediately quit, distinguishing between the first and subsequent launches.
  2. Numerical computation
    In Matlab (or equivalent), measure the time to multiply two matrices, to add two vectors, and to add two sparse matrices (i.e. matrices created with sparse()), as a function of size.
  3. Web server performance
    Set up a web server on your benchmark machine serving a file statically, and serving that same file through a CGI script that simply reads and outputs the file. Create a client program that makes repeated requests for one method or the other. Report the steady-state response rate for both methods as a function of the file size and of the number of simultaneous clients. Make sure your client machine(s) is (are) not a bottleneck.

References