The Nature of Measurement: Part 7: Significant Figures in Measurements

How to express a measurement is a complex matter. The number of digits observed and recorded is only the initial perception and expression of the quantity. What happens after that, as the data are corrected, adjusted and subjected to geometric manipulations and unit conversions, affects the precision of the measurement and, therefore, the expression of that measurement.

The number of digits recorded or reported should be related only to the precision of the observation and the geometric strength of the survey. Perceived significance, or that resulting from arbitrariness, may be quite different from actual significance. A displayed or recorded measurement viewed as 456.874 feet may be significant to other than the six digits displayed. A precision index, such as standard deviation, 95% error or a good estimate can override the convention that the number is precise to plus or minus half the last decimal place shown. If an analysis of the procedures showed an uncertainty of plus or minus 0.058 feet, the value has four significant figures and would be correctly shown as 456.9 feet. However, to avoid round-off errors in the future, it should be shown as 456.87 feet (Rule 4 below).

If the decimal places shown in a measurement are a result of proper analysis of the precision of the observation and subsequent attention to computational rules on significant figures and round-off errors, then the number has been correctly represented. If it is merely what the scales of the instruments read, what the computer or calculator screen displayed or based on wild estimates, it has probably been incorrectly represented.

The Meaning of Significant Figures

The number of digits expressed in a measurement is a statement of someone's perception of the significant figures of that quantity. Significant figures have often been defined in simple terms such as "the number of digits that have meaning." Some authors define them as "those that are read directly, plus the first doubtful one." However, such definitions are too simple to explain the full meaning of this abstract and sometimes confusing concept. For this reason, I define the term as follows:

The number of significant figures in a measurement is the number of figures that have meaning as estimated from the precision with which the quantity was observed and the data was reduced, said meaning being limited by the judgment of the person evaluating the precision of the observation and the computations.

The method and the analysis of its precision are what gives "meaning" or "significance" to the digits. When a user accepts this, he or she realizes that one cannot decide significant figures just by examining the digits or decimal places shown. Figures do not attain significance merely because they have been displayed. The above definition also says that different observers can conceivably have different opinions as to the "meaning," as perceived from an analysis of the precision. Thus, significance is at the mercy of the varying judgment of different observers and is not absolute. This definition also goes beyond a consideration of just the "raw readings" because the precision of a measured quantity is affected by many variables, most of them usually having greater effects on precision than the "least count" or refinement of instrument scale displays. For this reason, everyone working with the data must employ measurement theory, including random error analysis, error propagation and rules concerning significant figures in computations; otherwise measurements will probably be misrepresented.

Rule 1: The recorder of a measurement should report the value to the appropriate number of figures as reflected by the measuring and computational precision, showing one extra figure to avoid future round-off errors.

Whenever a person records a number in a field book, on a computation form, on a plat or elsewhere, the number of digits expressed is important because they affect the interpretation of the precision of the measurement, the time required to manipulate the number during future calculations and the precision of calculations using the number. The precision of any computed quantity obviously starts with the precision of the measurements used in the calculations. A user of the number often may have no means to evaluate the precision other than through the digits expressed by someone else. To an expert in measurement, a distance expressed as 125.8 feet does not mean the same thing as 125.80 feet. The non-professional or the uninformed surveyor or engineer may not see or care about the difference, but the difference is very real, and misunderstandings about it can affect the interpretation and use of measured data. For the remainder of this discussion, it must be assumed that numbers have been assigned an appropriate number of significant figures.

Rule 2: In adding or subtracting, the number of significant figures in the sum or difference is determined by the fewest decimal places in the numbers involved.

The sum 12.574 + 8,894.3 - 0.25860 = 8,906.615 as added directly with a calculator whose "FIX" is set to three decimal places but is properly rounded to 8,906.6 because the number with the "fewest decimal places" is the one to tenths. Some call this the "left-most decimal place" rule. If I measured three distances in segments, adding them to get the total, it is logical that the precision of the sum is determined by the one with the lowest precision. It would be a waste of time and effort to have some to higher precision than the least precise one.

Rule 3: In multiplication or division, the number of significant figures in the product or quotient is the same as the fewest number of significant figures in the measured values used, conversion factors or constants having no influence on the significant figures in results.

The computation of 34,780.07 x 56.999 ÷ 21.33 = 92,940.89123, but the correct answer is 92,940. This is because the 21.33 had only four significant figures. That the calculator was arbitrarily set to "FIX 5" has no bearing on the precision of the result. When conversion factors, functions or constants are involved, only the measured numbers dictate the significant figures. A length of 1,345.87 inches, converted to feet, is 112.156 feet because the conversion 12 inches/foot is an exact number. There are six significant figures in the measured value before and after conversion. Precision to hundredths of an inch is comparable to precision to thousandths of a foot. When the units are changed the decimal places often change in the converted number.

Rule 4: To avoid round-off errors in calculations:

• Use at least one digit more in conversion factors or constants than is in the measured value with the least number of significant figures

• Carry one extra digit in computed quantities to avoid round-off errors in calculating other quantities

• Record values appropriately as in Rule 1

In the example demonstrating multiplication and division, the answer was 92,940 (to four significant figures). Using this rule, I would record and retain 92,941, which rounds the calculated answer to five figures, not four.

Note: hereafter, substitute Pi for p and Delta for D

The following example dramatically illustrates the effect of round-off error and the importance of this rule. The area of a fraction of a circle is determined using pR2D÷360º where R = 348.56 feet, and D = 40º. I have assumed that the 40º is exact. If I did otherwise, we would be forced to deal with error propagation involving two variables, which is beyond the scope of what I am trying to illustrate here. As we have contrived the example, the round-off error will be affected by the significant figures used in the radius and the chosen value of p. Using five significant figures in the radius as expressed, the following results, using various popular values of p, are:

p = 3.14, Area = 42,387.932345 (rounds to 42,388 — large error)

p = 3.1416, Area = 42,409.5312913 (rounds to 42,410 — small error)

p = 3.14159, Area = 42,409.3962979 (rounds to 42,409 — no error)

p = 3.141592654, Area = 42,409.43212 (rounds to 42,409 — no error)

Note that it does not matter how many figures are used in p beyond six digits. The answer, to five places, will always be the same, and thus round-off error is avoided by applying the "one extra figure" rule. Anyone who uses approximations or those misleading tables of conversion factors often found in professi

onal publications, textbooks, and other references will probably have round-off errors.

Because p, trigonometric functions and other such numbers are usually derived to more than sufficient precision in modern electronic calculators or computers, this rule may often be unimportant. But anytime such values are not generated within the programmed systems to sufficient precision, round-off errors can occur. A conversion using 2.47 acres per hectare is good only to three significant figures as is a feet-to-meters conversion using 3.28 feet per meter. The computed answer changes in the decimal place to the right of the last digit of the constant used. If this place is not at least one digit beyond the analyzed precision warranted by the data, round-off error occurs. However, if the conversion factor is an exact number, such as 12 inches per foot, the answer retains the significant figures of the measurements, as explained in Rule 3.

Dealing With Zeroes

Zeroes have always been a source of confusion to anyone using measured data. It is best to think of zeroes in terms of how they affect the significant figures of computed quantities. The number 0.000465 has the same effect as 0.0465 or 46.5. Assuming the originator of such numbers analyzed their precision properly, there are only three significant figures in each because only these figures would affect any answer involving multiplication or division with this number. Zeroes at the beginning (either to left or right of the decimal point) are not significant. However, 125.8 has different significant figures than 125.80. Normally, a zero at the end is just as likely as the other nine random possibilities. However, there are exceptions when we know otherwise. In the example used for Rule 3, the correctly rounded number of 92,940 had four significant figures, not five. It can get more confusing when there are two or more zeroes. If the moon is 186,000 miles away, is that significant to units, tens, hundreds, or thousands? Most of the zeros are probably just there to give us the size of the number. The most likely value is either three or four significant figures. Knowing more about the measuring method would help resolve it.

Significant Figures an Indicator of Precision

As mentioned previously, a measured number is understood to be precise to plus or minus half the last decimal place shown unless some other precision is stated. For example, a distance to thousandths of a foot is understood to be plus or minus 0.0005 feet. A bearing shown to seconds is understood to be precise to plus or minus 0.5 seconds. An area shown to hundredths of an acre is understood to be precise to plus or minus 0.005 acres. All of these statements assume that the rules on significant figures were observed by those manipulating the data and that no "extra" digits were shown.

Let us put each of the above common expressions to a test. Suppose a surveyor used a total station and a prism pole to measure a distance and expressed it as 2,004.782 feet. At this distance, I think we could easily show that errors caused by instrument and target centering, failing to keep the prism pole plumb, observing temperature and pressure, observing the vertical angle used to convert to horizontal distance and various instrumental and other errors, would preclude having precision to plus or minus 0.0005 feet. The precision is probably plus or minus a few hundredths. A more honest expression of the distance is probably 2,004.8 feet, or 2,004.78 in view of the "one extra digit" rule.

A bearing is generally computed using an angle with respect to some fixed line. That angle contains random errors owing to the type of target; pointings to both targets; instrument reading precision; target and instrument centering; bubble centering and several geometric factors including sensitivity of the plate bubble, size of the vertical angle, sight distances to both targets and even the size of the angle itself. For sight distances under 100 feet, the surveyor is lucky to have plus or minus 1-minute precision in the angles. Expressing bearings to the nearest second hardly seems appropriate.

An area expressed to 0.01 acres implies a precision of plus or minus about 218 square feet (0.005 acres). If the tract happened to be square, it can be shown (using error propagation) that this implies the errors in the sides are about 0.74 feet each. That is only 1 part in 280 for distance precision! Something is clearly wrong here. If one assumed the precision in the distances to be more like 1 part in 2800, the area should at least be shown to thousandths of an acre. Using the "one extra digit" rule to avoid round-off errors in the future, it should probably be expressed to ten-thousandths. If the precise area was 1.0042 acres, showing 1.00 acres on the plat is a misrepresentation of the precision. The area error here is about 183 square feet. This not only cheats someone of an area the size of an average room in a house (worth perhaps several hundred or even thousands of dollars), but if a future surveyor took the 1.00 acre expression and made computations to divide the tract in half, the dividing line would be about 0.44 feet off of its correct position.

"More or Less" Not Enough

As a related point concerning area or any measured quantity, the commonly used term "more or less" is an unfortunate abomination of measurement science. Any plus or minus cited or implied ought to be based on measurement analysis, precision and significant figures. It should be quantified accordingly, not left hanging as if the surveyor were hiding from something or so unsure of his measurements that he had to end the statement with an embarrassing "I don't know what I have here—your guess is as good as mine." The whole thing is rooted in ignorance of the nature of measurement.

Unwarranted Extra Digits

Lack of attention to significant figures comes in three forms, and there are consequences for all three types of oversight. The first misuse is when unwarranted digits are reported in measured values. This falsifies the precision, misleads the user and usually results in false precision expressed in future numbers calculated. Unsuspecting users of the data may think they are precise to the extra decimal places shown, based on methods better than those actually used. Expressing an angle to seconds when it may only be good to minutes is an example of this misuse. Digitizing or scaling coordinates from maps and showing any numbers to the right of the decimal place is another example. The consequences of taking the measured number out of the context of the methods employed to derive it could be serious. This oversight does not contribute to round-off error, but is a misrepresentation of results. Although fraud may be too strong of a word ( in law, fraud usually carries with it the intent to deceive and gain advantage), reporting excess figures (beyond the "one extra") is misrepresentation.

Insufficient Significant Figures

The second problem is when insufficient significant figures are shown somewhere in the measuring or computing process, owing either to faulty initial recording of data or subsequent expressions of values computed. Showing acreage to too low of a precision is an example. The person using the data often has no way to retrieve the true precision resulting from the field procedures and thus precision of any calculations made with the faulty expressions of the numbers is affected. Sophisticated computer programs having high internal computational precision and fancy mathematical procedures cannot create precision in imprecise data. They can only retain the inherent precision of the data. If values have been reported to insufficient precision, the loss of precision in overall results can exceed that caused by the three error sources associated with the observations (nature, instruments, people). In other words, thousands of dollars spent on instruments, computer hardware and software, personnel training and field work could easily be wasted if someone neglects the importance of round-off errors in calculations. Remember, "garbage in, garbage out."

Accumulated Round-off Errors

The third problem occurs when the computational personnel violate rules regarding round-off errors or otherwise misuse reported data. This problem is different from merely reporting numbers incorrectly, as above. This problem represents the deterioration of precision as a result of accumulated round-off errors. What may have been good data originally can evolve to numerical "garbage." The usual faults here are not using sufficient digits in conversion factors and failing to follow the "one extra digit" rule throughout the computational procedures.

Violation of the rules of expression and computation of measured values to be published in legal documents, relied on by others as the beginning place to derive other quantities, and often used as a basis of determining the precise monetary value of land, should be considered a violation of minimum acceptable standards for professional surveyors. As experts in measurement, surveyors should report measured quantities to the appropriate precision and follow the basic rules of computation to avoid unnecessary round-off errors in derived measurements. Of all of the errors associated with measurement, round-off error is the only one that can be avoided.

Dr. Ben Buckner is an educator, author, and seminar presenter with Surveyors' Educational Seminars and a Contributing Editor for the magazine.

» Back to our March 1998 Issue