Selection Criteria for Metrics

There are many things that can be measured. What should be measured depends on your goals. The appropriate metric or metrics to use depend on the user, client, and workgroup, and can vary from company to company or product to product. Ultimately, what you measure should pertain to what the overall organization is interested in. This is usually time, money, and resources.

Metrics come from users, clients, and internal standards. These metrics can be derived from basic research, CSFs, user satisfaction surveys, or process analysis. In addition, existing published metrics can be used. Metrics can measure products (how many pages, typos per page) or processes (elapsed days to complete, project billable hours).

A project management axiom is that what you cannot control what you cannot measure, so measure what you want to control.

Consider whether the metric is worth the difficulty of obtaining it. Metrics that measure quality attributes of importance to users, clients, and internal standards as well (which tend also to be CSFs) are the most important to measure. Two examples are timeliness and accuracy, which are universally understood to be critical. However, this is not to say that an easily obtained metric is more important than one difficult to obtain.

Metrics are most accurate and effective in combination (multiple metrics measuring one attribute), or if they are multivariate (one metric measuring multiple attributes). For example, the simple process metric pages per day could be circumvented if the writer increases the font size, page margins, or line spacing of a document; but by combining that metric with adherence to a workgroup style guide that specifies font size, page margins, and line spacing, the metric remains meaningful. In practical usage, given independent variables, only a small number of multivariate metrics is sufficient to measure any given quality attribute.

Some metrics are easier to obtain than others. Direct measurements are easier than indirect ones; for example, the cost of creating something is easier to obtain than the cost of not creating something.

Prefer objective to subjective metrics, because objective metrics do not depend on the person making the measurement and are consistently repeatable. For example, an objective measurement is the number of graphics in a document, because given a definition of “graphic,” everyone who takes the measurement will get the same result; a subjective measurement is readers’ opinion of whether the number of graphics is appropriate, because different readers will likely have different opinions. (This is not to say that subjective metrics have no value.)

Wherever possible, record and retain raw data. This lets you combine data over people, projects, and time.