Chapter 01

Medium

Floating-Point Arithmetic and Rounding Error

00 · Symbol Glossary

$\beta$beta — base

The integer base of the digit expansion, with $\beta\geq 2$ . Decimal examples in this chapter use $\beta=10$ . The binary model uses $\beta=2$ . Read " $\beta$ " as "base beta."

$p$p — precision

The number of significant digits kept in a normal floating-point number. Precision is a count of digits, not a statement about a particular device. With $\beta=10$ and $p=4$ , every normal number has the shape $\pm d_0.d_1 d_2 d_3\times 10^{e}$ .

$F$F — floating-point set

The finite set of normal floating-point numbers determined by $\beta$ , $p$ , and the exponent bounds. Rounding is a map from a real number in range into $F$ . Read " $x\in F$ " as "x is a floating-point number."

$\varepsilon_{\mathrm{mach}}$epsilon mach — machine epsilon

The gap $\beta^{1-p}$ between $1$ and the next larger number in $F$ . It is a property of the set $F$ , not of an algorithm. Linear Algebra 20 uses this same gap.

$u$u — unit roundoff

Half a machine epsilon under round-to-nearest: $u=\tfrac12\beta^{1-p}$ . A single rounding of a real in the normal range changes it by a relative amount of at most $u$ .

$\mathrm{fl}(x)$fl of x — rounded value

The floating-point number obtained by rounding the real $x$ into $F$ . Read "fl of x." For an operation $\star$ , $\mathrm{fl}(x\star y)$ means the exact real result is formed and then rounded.

$\delta$delta — relative rounding factor

The real number, depending on the operands, for which the rounded result equals the exact result times $1+\delta$ , with $|\delta|\leq u$ . It is not a constant defect: each operation has its own $\delta$ .

$E_{\mathrm{abs}},\ E_{\mathrm{rel}}$absolute and relative error

$E_{\mathrm{abs}}=|\hat x-x|$ and, when $x\neq 0$ , $E_{\mathrm{rel}}=|\hat x-x|/|x|$ . Unit roundoff controls the second of these for one rounding. Cancellation inflates the second while leaving the first small.

$\kappa_{\mathrm{sub}}$kappa sub — condition of subtraction

The amplification $(|x|+|y|)/|x-y|$ of relative perturbations through the map $(x,y)\mapsto x-y$ . A large value means the subtraction is sensitive. The matrix condition number in Linear Algebra 20 is the same idea for a linear map.

01 · A Finite Subset of the Reals

Exact real arithmetic has infinite precision. A calculation that keeps a fixed number of digits cannot. The object of this chapter is that restriction, written as a set of real numbers and a rounding map, together with the error bounds the map satisfies. Nothing below depends on a programming language. A four-digit decimal calculation done by hand is the same mathematical model as a binary system with a larger $p$ .

The practical distinction the bounds explain is this. Rounding $123.448$ to four significant digits moves it by a few hundredths, a small relative change. Subtracting two such rounded prices can move the difference by a large relative amount, because the digits that survived rounding are precisely the digits the two prices share. Later sections turn that observation into a theorem.

Definition — Normal floating-point numbers

Fix integers $\beta\geq 2$ , $p\geq 1$ , and exponent bounds $e_{\min}\leq e_{\max}$ . A normal floating-point number is a real number

x=\pm\, m\times \beta^{\,e-p},\qquad m\in\mathbb{Z},\quad \beta^{p-1}\leq m\leq \beta^{p}-1,\quad e_{\min}\leq e\leq e_{\max}.

The integer $m$ is the integer significand. Equivalently, in scientific form,

x=\pm\,(d_0.d_1\ldots d_{p-1})_{\beta}\times \beta^{\,e-1},\qquad d_0\in\{1,\ldots,\beta-1\},

so the leading digit is nonzero. The factor $(d_0.d_1\ldots d_{p-1})_{\beta}$ is the significand (or mantissa), and $e-1$ in this second writing is the exponent of the scientific form. The set of all such $x$ , together with $0$ , is denoted $F$ .

Plain language: keep a fixed number of digits, require the first digit to be nonzero, and attach a power of the base. Those rules name a finite list of real numbers, written $F$ , together with zero. A real that is not on the list is not a floating-point number, however close it sits to one that is.

The two displays are the same set. For $\beta=10$ and $p=4$ , the integer form $1234\times 10^{-2}$ is the scientific form $1.234\times 10^{1}$ . Normalization — the demand that $d_0\neq 0$ , or equivalently $\beta^{p-1}\leq m$ — prevents the same real from being written with two different exponents inside $F$ . Without it, $1.234\times 10^{1}$ and $0.1234\times 10^{2}$ would be two names for one number, and "the" gap between adjacent elements would be undefined.

In the binade $\beta^{e-1}\leq |x|<\beta^{e}$ the admissible $m$ step by $1$ , so adjacent elements of $F$ are spaced $\beta^{e-p}$ apart. The spacing is proportional to the magnitude. That is the structural fact behind every relative-error bound in this chapter.

Example — Four decimal digits, two binades

Take $\beta=10$ , $p=4$ , and exponents wide enough that every number below is normal.

At $1=1.000\times 10^{0}$ the successor in $F$ is $1.001$ , so the forward gap is $0.001=\beta^{1-p}$ . The predecessor is $0.9999$ , one binade lower, where the gap is $0.0001$ .
At $1000=1.000\times 10^{3}$ the successor is $1001$ and the forward gap is $1$ . The predecessor is $999.9$ . Throughout $[1000,9999]$ the gap stays $1$ .
The real $1000.4$ lies strictly between $1000$ and $1001$ , so it is not an element of $F$ .

A price quoted near $1000$ and a rate quoted near $1$ are not being recorded to the same absolute tolerance, even though both keep four digits.

Binary parameters, including the IEEE set

For $\beta=2$ the leading digit of a normal number is necessarily $1$ . Describing $m$ then requires only the remaining $p-1$ bits; the set $F$ is unchanged. The binary64 normal numbers are exactly this model with $\beta=2$ , $p=53$ , $e_{\min}=-1022$ , and $e_{\max}=1023$ . Then $\varepsilon_{\mathrm{mach}}=2^{-52}$ . The specification also fixes a tie-breaking rule, recorded in the next section. Subnormal numbers, which fill the gap below the smallest normal with constant absolute spacing, are not needed for the identities below; the bounds assume a normal result.

❌ What Breaks — Treating spacing as an absolute constant

Suppose four decimal digits are in use, and a spread of $0.4$ is treated as "safely larger than the last digit" because the last digit near $1$ is $0.001$ .

Why it breaks: the gap at magnitude $10^{3}$ is $1$ , not $0.001$ . The real $1000.4$ is not in $F$ , and both $1000$ and $1001$ are admissible roundings of nearby reals. A tolerance copied from the binade of $1$ does not apply in the binade of $1000$ .

Consequence: an absolute error budget with no factor of $|x|$ is not a floating-point error budget. The quantity that is uniform across binades is the relative gap, computed next.

02 · Rounding and the Standard Model

$F$ is discrete. A real $x$ in the normal range is replaced by a nearby element of $F$ .

Definition — Rounding, machine epsilon, unit roundoff

Let $x\neq 0$ lie between the least and greatest positive elements of $F$ , and let $x_{-}<x_{+}$ be the adjacent elements of $F$ with $x_{-}\leq x\leq x_{+}$ . Round-to-nearest sets $\mathrm{fl}(x)$ equal to whichever of $x_{-}$ , $x_{+}$ is closer to $x$ . On an exact tie either neighbor may be chosen; the binary64 rule chooses the neighbor whose integer significand is even. Chopping always chooses the neighbor toward $0$ .

The machine epsilon is the gap just above $1$ ,

\varepsilon_{\mathrm{mach}}=\beta^{1-p}.

For round-to-nearest the unit roundoff is half that gap,

u=\tfrac12\,\varepsilon_{\mathrm{mach}}=\tfrac12\,\beta^{1-p}.

For the running decimal system, $\varepsilon_{\mathrm{mach}}=10^{-3}$ and $u=5\times 10^{-4}$ . For binary64, $\varepsilon_{\mathrm{mach}}=2^{-52}$ and $u=2^{-53}$ .

Plain language: rounding sends a real to a neighboring entry of $F$ . Machine epsilon is the distance from $1$ to the next larger entry. Unit roundoff is half that distance, and it is the largest relative change one round-to-nearest step is allowed to make.

Linear Algebra 20 quotes $\varepsilon_{\mathrm{mach}}\approx 2.2\times 10^{-16}$ for binary64. That number is $2^{-52}$ , the gap. The rounding inequality in this chapter is stated with $u$ , which is half the gap when ties are broken by nearness. For chopping, the sharp constant is $\varepsilon_{\mathrm{mach}}$ rather than $u$ ; every theorem below is the round-to-nearest case.

Theorem — Relative form of a single rounding

If $x$ lies in the open normal range, then there exists a real $\delta$ with

\mathrm{fl}(x)=x(1+\delta),\qquad |\delta|\leq u.

Equivalently, $|\mathrm{fl}(x)-x|\leq u\,|x|$ .

Plain language: a single rounding multiplies the true value by $1+\delta$ , and $|\delta|$ is at most the unit roundoff. The absolute error may grow with $|x|$ ; the relative error stays within $u$ .

Step-by-step — Why the relative error is at most u

Place $x$ in its binade. For some exponent $e$ , $\beta^{e-1}\leq |x|<\beta^{e}$ . The proof for $x<0$ is the same after changing sign, so take $x>0$ .

Read the gap. Adjacent elements of $F$ in this binade differ by $\beta^{e-p}$ . Round-to-nearest moves $x$ by at most half a gap, including on a tie: $|\mathrm{fl}(x)-x|\leq \tfrac12\beta^{e-p}$ .

Divide by the smallest magnitude in the binade. The largest relative error occurs at the left endpoint, where $|x|$ may be as small as $\beta^{e-1}$ . Therefore

\frac{|\mathrm{fl}(x)-x|}{|x|}\leq \frac{\tfrac12\beta^{e-p}}{\beta^{e-1}}=\tfrac12\beta^{1-p}=u.

Higher in the same binade the denominator is larger and the relative error is strictly smaller.

Rewrite as a factor. Set $\delta=(\mathrm{fl}(x)-x)/x$ . The previous step is $|\delta|\leq u$ , and $\mathrm{fl}(x)=x(1+\delta)$ .

Compare

Unit roundoff is the smallest relative nudge near $1$ : half the step from $1$ to the next number in $F$ . The absolute step grows with the magnitude, and that relative nudge stays the size of $u$ .

Rounding each input, then operating, then rounding the result, is the whole of floating-point arithmetic. For inputs already in $F$ the first rounding is idle.

Theorem — Standard model of arithmetic

Let $\star$ be one of $+$ , $-$ , $\times$ , $\div$ . Let $x,y\in F$ , with $y\neq 0$ in the division case, and assume $x\star y$ lies in the normal range (no overflow and no underflow). Then there exists $\delta$ , depending on $x$ , $y$ , and $\star$ , such that

\mathrm{fl}(x\star y)=(x\star y)\,(1+\delta),\qquad |\delta|\leq u.

In words: the computed result is the exact result, multiplied by a relative factor within one unit roundoff.

Plain language: the computed result of one addition, subtraction, multiplication, or division is the exact real result, then one rounding. That rounding is a relative factor within one unit roundoff, as long as the exact result stays inside the normal range.

The identity is not an approximation slogan. It is the previous theorem applied to the real number $x\star y$ . Overflow (an exponent above $e_{\max}$ ) and underflow (a result below the smallest normal) are excluded from the hypothesis because the rounded result then leaves the model: there is no $\delta$ of size at most $u$ for which the identity holds.

Example — A product in four decimal digits

Compute $1.248\times 3.162$ with $\beta=10$ , $p=4$ , round-to-nearest.

The exact product splits as $1.248\times 3=3.744$ , $1.248\times 0.16=0.19968$ , and $1.248\times 0.002=0.002496$ . Adding these gives $3.744+0.19968=3.94368$ , then $3.94368+0.002496=3.946176$ .

The fifth significant digit of $3.946176$ is $1$ , so $\mathrm{fl}(3.946176)=3.946$ . The absolute discrepancy is $3.946176-3.946=0.000176$ . Hence

\delta=\frac{3.946-3.946176}{3.946176},\qquad |\delta|=\frac{1.76\times 10^{-4}}{3.946176}.

Since $3.946176\times 5\times 10^{-5}=1.973\times 10^{-4}>1.76\times 10^{-4}$ , one has $|\delta|<5\times 10^{-5}$ , which is well below $u=5\times 10^{-4}$ . The standard model guarantees only the coarser bound; this particular product lands closer.

The model does not say that a formula of several operations has relative error $u$ . Each operation contributes its own factor $1+\delta_i$ . Nor does it say that the operations remain associative.

❌ What Breaks — Associativity of addition

Work in $\beta=10$ , $p=3$ , so the gap at $10^{3}$ is $10$ and $u=5\times 10^{-3}$ . Let $a=1.00\times 10^{3}$ , $b=-1.00\times 10^{3}$ , and $c=1.00\times 10^{0}$ , all three already in $F$ .

The two groupings. First, $\mathrm{fl}(a+b)=0$ and $\mathrm{fl}(0+c)=1$ . Second, the exact sum $b+c=-999$ lies at distance $1$ from $-1000$ and at distance $9$ from $-990$ , so $\mathrm{fl}(b+c)=-1.00\times 10^{3}$ . Then $\mathrm{fl}(a+\mathrm{fl}(b+c))=0$ .

Why it breaks: each individual addition satisfies the standard model. The two expressions insert the rounding at different places. After $\mathrm{fl}(b+c)$ has discarded the unit, no later addition can recover it.

Consequence: $a+(b+c)$ and $(a+b)+c$ are not interchangeable once results are required to lie in $F$ . A rearrangement is a different algorithm and needs its own error bound.

03 · Absolute and Relative Error

Definition — Absolute and relative error

If $\hat x$ is an approximation to a real $x$ , the absolute error is $E_{\mathrm{abs}}=|\hat x-x|$ . When $x\neq 0$ , the relative error is $E_{\mathrm{rel}}=|\hat x-x|/|x|$ . Relative error is unchanged if $x$ and $\hat x$ are scaled by the same nonzero constant. Absolute error is not.

Plain language: absolute error is the distance between the approximation and the true value. Relative error is that distance divided by the size of the true value, so scaling both numbers by the same nonzero constant leaves it unchanged.

The theorem of the previous section is a relative-error statement: one rounding produces $E_{\mathrm{rel}}\leq u$ , and therefore $E_{\mathrm{abs}}\leq u\,|x|$ . An absolute tolerance copied from one magnitude to another has no justification in $F$ .

Example — One absolute error, two relative errors

Suppose a computed value is $0.010$ above the true value.

On a price $x=250$ , one has $E_{\mathrm{rel}}=0.010/250=4\times 10^{-5}$ . In the four-digit decimal system this is below $u=5\times 10^{-4}$ : the price has survived to better than a unit roundoff.
On a one-day rate difference $x=0.0004$ , the same absolute error gives $E_{\mathrm{rel}}=0.010/0.0004=25$ . The computed difference has no correct digit. The absolute defect did not change; the denominator did.

Reporting " $0.010$ of error" without $|x|$ does not say whether the result is usable.

❌ What Breaks — Reading u as an absolute tolerance

In binary64, $u=2^{-53}\approx 1.11\times 10^{-16}$ . A notional of size $10^{9}$ is rounded.

Why it breaks: the absolute bound from the rounding theorem is $u\,|x|\approx 1.11\times 10^{-7}$ , not $10^{-16}$ . The number $u$ is dimensionless. Multiplying a cash amount by $u$ changes the unit from "relative" back to "currency."

Consequence: a control that treats $10^{-16}$ as a currency tolerance accepts absolute errors a billion times larger than that control was meant to catch, as soon as the notional is $10^{9}$ .

04 · Catastrophic Cancellation

Subtraction of nearly equal floating-point numbers is often exact. The damage is older than the subtraction.

Lemma — Exact difference at a common exponent

Let $x,y\in F$ be nonzero and have the same scientific exponent, and assume $x-y$ does not underflow. Then $x-y\in F$ , so $\mathrm{fl}(x-y)=x-y$ .

Plain language: if two floating-point numbers share an exponent, their difference is already an element of $F$ , provided it does not underflow. The subtraction returns the exact difference, and the rounding factor on that operation is $1$ .

Step-by-step — Why a same-exponent difference needs no extra digits

Write both numbers on the integer lattice of that binade. For a common exponent parameter $e$ , $x=m_x\,\beta^{e-p}$ and $y=m_y\,\beta^{e-p}$ with integers $m_x,m_y$ between $\beta^{p-1}$ and $\beta^{p}-1$ .

Subtract the integers. $x-y=(m_x-m_y)\,\beta^{e-p}$ . The coefficient $m_x-m_y$ is an integer of absolute value at most $\beta^{p}-\beta^{p-1}-1<\beta^{p}$ , so it does not need a $(p+1)$ st digit.

Normalize if the leading digit vanished. If $|m_x-m_y|\geq \beta^{p-1}$ , the difference is already normal at this exponent. If the coefficient is smaller, lower the exponent until the leading digit is nonzero. Lowering the exponent shifts a shorter integer; it does not invent digits. The hypothesis that the result is still above the underflow threshold keeps the new exponent inside the range.

Conclude. $x-y$ equals an element of $F$ , so rounding changes nothing and the $\delta$ in the standard model is $0$ .

The binary statement extends past a single binade. If $x,y$ are binary floating-point numbers, $y/2\leq x\leq 2y$ , and $x-y$ does not underflow, then again $x-y\in F$ . The extra room of a factor two is special to $\beta=2$ : the unit in the last place at most doubles when the exponent increases by one, and $x\leq 2y$ keeps $x$ on the same lattice as $y$ . This is Sterbenz's lemma. The decimal examples below need only the same-exponent form already proved.

Definition — Catastrophic cancellation

Let $x,y$ be reals and $\hat x=\mathrm{fl}(x)$ , $\hat y=\mathrm{fl}(y)$ . Catastrophic cancellation is the situation in which $|\hat x-\hat y|$ is comparable to the absolute errors $|\hat x-x|$ and $|\hat y-y|$ , so that the relative error of $\hat x-\hat y$ as an approximation to $x-y$ is large, even though $\mathrm{fl}(\hat x-\hat y)=\hat x-\hat y$ exactly.

Plain language: the stored difference can equal $\hat x-\hat y$ exactly and still be a poor stand-in for $x-y$ . The digits that separated the true values were dropped when each value was rounded, so the difference is assembled from what rounding discarded.

The condition number of the map makes the inflation quantitative. Perturb $x$ by a relative amount $\delta_x$ and $y$ by $\delta_y$ . The relative change in the difference satisfies

\frac{|(x(1+\delta_x)-y(1+\delta_y))-(x-y)|}{|x-y|}\leq \frac{|x|\,|\delta_x|+|y|\,|\delta_y|}{|x-y|}.

If $|\delta_x|,|\delta_y|\leq u$ , the right-hand side is at most $\kappa_{\mathrm{sub}}\,u$ with

\kappa_{\mathrm{sub}}(x,y)=\frac{|x|+|y|}{|x-y|}.

When $x\approx y$ the denominator is small and $\kappa_{\mathrm{sub}}$ is large. The subtraction did not become an inaccurate operation. The function $(x,y)\mapsto x-y$ became a sensitive function.

Compare

Cancellation here is shared leading digits wiping the difference. The stored values agree on the digits that were kept, so $\hat x-\hat y$ is built from the digits the two roundings already threw away.

Example — Leading digits that were never stored

Take $\beta=10$ , $p=4$ , $u=5\times 10^{-4}$ . Let the true values be $x=1.23448$ and $y=1.23052$ , so $x-y=0.00396$ .

The fifth digit of $1.23448$ is $4$ , so $\hat x=1.234$ . The fifth digit of $1.23052$ is $5$ followed by a nonzero remainder, so $\hat y=1.231$ under round-half-away-from-zero. Both inputs have absolute error $0.00048$ , and both relative errors are about $3.9\times 10^{-4}\leq u$ .

The stored numbers share an exponent. By the lemma, $\mathrm{fl}(\hat x-\hat y)=1.234-1.231=0.003$ exactly. Against the true difference,

E_{\mathrm{abs}}=|0.003-0.00396|=0.00096,\qquad E_{\mathrm{rel}}=\frac{0.00096}{0.00396}=\frac{8}{33}\approx 0.242.

The relative error is several hundred times $u$ . The condition number explains the size: $(|x|+|y|)/|x-y|\approx 2.465/0.00396\approx 622$ , and $622\times 3.9\times 10^{-4}\approx 0.24$ .

The same arithmetic is a spread of two quoted levels. The stored level keeps four digits; the spread lives in the digits the rounding already discarded. Measuring the spread's relative error against $u$ alone, with no factor $\kappa_{\mathrm{sub}}$ , will call a bad spread acceptable.

A standard cure is to rewrite the expression so that the small result is not a difference of two rounded quantities. The quadratic formula is the classical case, and every step below is four-digit decimal arithmetic.

Step-by-step — The smaller root of x squared minus 1000 x plus 1

Record the exact roots. For $x^{2}-1000x+1=0$ , the discriminant is $10^{6}-4=999996$ and its square root is just under $1000$ . The smaller root is $(1000-\sqrt{999996})/2=0.001000001000\ldots$ , and the product of the roots is $1$ .

Round the discriminant in four digits. $999996=9.99996\times 10^{5}$ . The fifth significant digit is $6$ , so $\mathrm{fl}(999996)=1.000\times 10^{6}$ . The subtraction $10^{6}-4$ has been lost: the gap at $10^{6}$ is $100$ , and $4$ is far below half a gap.

Evaluate the cancelling formula. $\mathrm{fl}(\sqrt{1.000\times 10^{6}})=1.000\times 10^{3}$ . Then $\mathrm{fl}(1000-1000)=0$ , so the computed smaller root is $0$ . The relative error against $0.001$ is $1$ .

Use the conjugate identity instead. For $x^{2}-Bx+C=0$ with $B>0$ , the smaller root equals $2C/(B+\sqrt{B^{2}-4C})$ . In four digits, $B+\sqrt{\cdot}$ is computed as $1000+1000=2000$ and $2/2000=0.001=1.000\times 10^{-3}$ , which lies in $F$ . The absolute error is about $10^{-9}$ , and the relative error is about $10^{-6}$ , far below $u$ .

The algebra of the rewrite is worth one line, because it is the reason the second algorithm never subtracts. If $s=\sqrt{B^{2}-4C}$ , then

\frac{B-s}{2}=\frac{(B-s)(B+s)}{2(B+s)}=\frac{B^{2}-s^{2}}{2(B+s)}=\frac{2C}{B+s}.

The left side subtracts two quantities that agree to six figures. The right side divides two quantities of moderate size. They are equal in real arithmetic and unequal in $F$ .

❌ What Breaks — Blaming the subtraction operation

In the quadratic calculation, $1000-1000$ was exact, and in the two-level example $1.234-1.231$ was exact.

Why it breaks: the standard model applied to those subtractions gives $\delta=0$ . The zeros that appeared in the leading digits were already zeros in $F$ before the subtraction was performed. Sterbenz's setting, and the same-exponent lemma, describe an operation that returns its exact mathematical value.

Consequence: "carrying more guard digits through the subtraction" does not repair a difference whose error was committed when the operands were rounded. The repair is an algebraically equivalent expression whose condition number is moderate, as with $2C/(B+s)$ .

05 · Truncation Error and Roundoff

Rounding is not the only discrepancy between a mathematical expression and the number that replaces it. Stopping a limit, a series, or a difference quotient introduces a second discrepancy that would be present even if every arithmetic operation were exact in $\mathbb{R}$ .

Definition — Truncation error and roundoff error

Suppose a target value $L$ is replaced by a mathematical approximation $T(h)$ , and $T(h)$ is then evaluated in $F$ to produce $\hat T(h)$ . The truncation error is $|T(h)-L|$ . The roundoff error is $|\hat T(h)-T(h)|$ . The error that is actually observed splits as

|\hat T(h)-L|\leq |T(h)-L|+|\hat T(h)-T(h)|.

The first summand is a statement about analysis in $\mathbb{R}$ . The second is a statement about $\mathrm{fl}$ and is bounded using $u$ .

Plain language: truncation error is the distance between the mathematical shortcut $T(h)$ and the target $L$ . It would remain if every arithmetic step were exact. Roundoff error is the further distance between the digit-limited value $\hat T(h)$ and that same shortcut.

Compare

Truncation is the formula cut short of the limit, series, or quotient that defines $L$ . Roundoff is the effect of a finite digit budget on the formula that was actually evaluated. The observed error adds the two.

The two summands typically move in opposite directions as a step $h$ shrinks. A first-order difference quotient has truncation of order $h$ and a roundoff contribution of order $u/h$ , because a difference of size $h$ is divided by $h$ after absolute noise of size about $u$ has entered the function values. Shrinking $h$ improves the first and degrades the second.

Step-by-step — (e to the h, minus 1), divided by h, in four digits

Name the target and the remainder. The target is $e^{0}=1$ , estimated by $T(h)=(e^{h}-1)/h$ . Taylor's theorem with remainder gives $e^{h}=1+h+\tfrac12 h^{2}e^{\xi}$ for some $\xi$ between $0$ and $h$ , so

\big|T(h)-1\big|=\tfrac12 h\,e^{\xi}\leq \tfrac12 h\,e^{|h|}.

The right-hand side is pure truncation: no rounding has been used.

Take $h=10^{-1}$ , where truncation dominates. The series $1+0.1+0.005+0.0001667+0.000004167$ sums to $1.1051708\ldots$ , and the unused remainder is smaller than $10^{-7}$ . Four-digit rounding gives $\mathrm{fl}(e^{0.1})=1.105$ . Then $(1.105-1)/0.1=1.05$ . The observed absolute error is $|1.05-1|=0.05$ , matching $\tfrac12 h=0.05$ from the truncation bound. Replacing $1.1051708$ by $1.105$ changed the quotient by only $0.0017$ .

Take $h=10^{-4}$ , where roundoff dominates. Here $e^{h}=1.000100005\ldots$ . The successor of $1.000$ is $1.001$ . Distance $0.0001$ against distance $0.0009$ puts $1.0001$ closer to $1.000$ , so $\mathrm{fl}(e^{h})=1.000$ . The computed quotient is $(1.000-1)/10^{-4}=0$ . Truncation is at most $\tfrac12 h\,e^{h}<6\times 10^{-5}$ . The observed absolute error is $1$ , from rounding the numerator to zero in $F$ .

Check the roundoff budget. An absolute perturbation of size at most $u\,e^{|h|}$ in the numerator, followed by division by $h$ , moves $T(h)$ by at most about $u/|h|$ . For $h=10^{-4}$ and $u=5\times 10^{-4}$ this ceiling is $5$ , which is large enough to permit the computed value $0$ . For $h=10^{-1}$ the same ceiling is $5\times 10^{-3}$ , smaller than the truncation $5\times 10^{-2}$ . The bound tells which term is in charge before the quotient is formed.

What later chapters inherit

Differentiation, quadrature, and one-step methods for differential equations each produce a $T(h)$ whose truncation tends to $0$ with $h$ , and a computed $\hat T(h)$ whose roundoff does not. Those chapters bound truncation by a Taylor expansion in $\mathbb{R}$ and roundoff by the standard model above. The useful step balances the two terms in the split.

❌ What Breaks — Taking a smaller step to buy accuracy

The four-digit quotient at $h=10^{-1}$ has absolute error $0.05$ . Replacing the step by $h=10^{-4}$ is proposed in order to "make the Taylor remainder negligible."

Why it breaks: the truncation bound does fall, from $5\times 10^{-2}$ to under $6\times 10^{-5}$ . The computed numerator $e^{h}-1$ falls below half a unit in the last place of $1$ and rounds to $0$ . The observed error rises from $0.05$ to $1$ .

Consequence: once $|\hat T(h)-T(h)|$ exceeds $|T(h)-L|$ , further reduction of $h$ is not a refinement of the approximation. The limiting value in exact arithmetic and the limiting value in $F$ are different objects.

06 · The Scalar Model Beside a Linear System

Every bound above concerns a scalar map. A linear system $A\mathbf{x}=\mathbf{b}$ is a map on matrices and vectors. If $\kappa(A)=\|A\|\,\|A^{-1}\|$ and the data move by small relative amounts, the relative change in $\mathbf{x}$ is controlled by $\kappa(A)$ times those amounts. Linear Algebra 20 states that theorem; the argument is not repeated here.

The scalar standard model supplies only the size of the perturbation. One rounding multiplies by $1+\delta$ with $|\delta|\leq u$ . A backward-stable elimination returns the exact solution of a nearby system whose relative perturbation is a modest multiple of $u$ , and passing to the forward error multiplies by $\kappa(A)$ :

\text{relative forward error}\ \lesssim\ \kappa(A)\cdot u,

up to the dimensional factors kept explicit in Linear Algebra 20. Moderate $\kappa(A)$ with small $u$ leaves digits intact. A condition number as large as $1/u$ can remove every digit while each arithmetic step still satisfies $|\delta|\leq u$ .

Section 04 is the same pattern with $\kappa_{\mathrm{sub}}$ in place of $\kappa(A)$ . An exact subtraction and a large $\kappa_{\mathrm{sub}}$ match a backward-stable factorization and a large $\kappa(A)$ : the operation met the model, and the function amplified it. The conjugate form of the quadratic changes the condition number. It leaves $u$ as it is.

07 · Exercises

EXERCISE 1.1

Write $2/3$ as $6.666\ldots\times 10^{-1}$ and look at the fifth significant digit. The absolute error is a difference of two fractions with denominator $30000$ .

The expansion is $2/3=0.6666\ldots=6.666\ldots\times 10^{-1}$ . Five leading digits begin $66666$ , so the fifth digit is $6$ and $\mathrm{fl}(2/3)=6.667\times 10^{-1}=0.6667$ . Then $0.6667-2/3=6667/10000-2/3=(20001-20000)/30000=1/30000$ , so $E_{\mathrm{abs}}=1/30000$ . The relative error is $(1/30000)/(2/3)=1/20000=5\times 10^{-5}$ . Since $u=5\times 10^{-4}$ , this $\delta$ satisfies $|\delta|\leq u$ . For this system $\varepsilon_{\mathrm{mach}}=10^{1-4}=10^{-3}$ , and $u=\varepsilon_{\mathrm{mach}}/2$ . The predecessor of $1$ in $F$ is $0.9999$ and the successor is $1.001$ .

In the system $\beta=10$ , $p=4$ , with round-to-nearest (half away from zero when the discarded part is at least half a unit), compute $\mathrm{fl}(2/3)$ . Give the absolute error and the relative error as exact fractions, and check them against $u$ . State $\varepsilon_{\mathrm{mach}}$ and the two neighbors of $1$ in $F$ .

EXERCISE 1.2

Expand $2.375\times 8$ and $2.375\times 0.124$ separately, add, then keep four significant digits. Compare the absolute discrepancy with $u$ times the exact product.

$2.375\times 8=19$ and $2.375\times 0.124=2.375\times 0.1+2.375\times 0.02+2.375\times 0.004=0.2375+0.0475+0.0095=0.2945$ . The exact product is $19.2945$ . The fifth significant digit is $4$ , so $\mathrm{fl}(19.2945)=19.29$ . The absolute discrepancy is $0.0045$ . Then $|\delta|=0.0045/19.2945$ . Because $19.2945\times 3\times 10^{-4}=0.005788>0.0045$ , one has $|\delta|<3\times 10^{-4}\leq u=5\times 10^{-4}$ , in agreement with the standard model. The model asserts existence of some such $\delta$ ; this product exhibits one.

In four-digit decimal round-to-nearest arithmetic, compute $\mathrm{fl}(2.375\times 8.124)$ from the exact product. Exhibit a $\delta$ satisfying the standard model and show $|\delta|\leq u$ by a direct comparison, without appealing to the general proof.

EXERCISE 1.3

With $B=400$ and $C=1$ , form $B^{2}-4C=159996$ and round to four significant digits before taking the square root. For the stable value use $2C/(B+s)$ with that same rounded square root.

$B^{2}-4C=160000-4=159996=1.59996\times 10^{5}$ , which rounds to $1.600\times 10^{5}$ because the fifth digit is $9$ . The square root in four digits is $400.0$ . The cancelling formula returns $(400-400)/2=0$ . The true smaller root is $(400-\sqrt{159996})/2\approx 0.0025000156$ , so the computed root $0$ has relative error $1$ . The conjugate form uses the same rounded square root: $2/(400+400)=2/800=0.0025$ , which equals $2.500\times 10^{-3}$ and lies in $F$ . Its absolute error is about $1.6\times 10^{-8}$ and its relative error is about $6\times 10^{-6}$ , below $u=5\times 10^{-4}$ . The subtraction $400-400$ was exact; the information was lost when $159996$ was replaced by $160000$ .

For $x^{2}-400x+1=0$ , compute the smaller root in four-digit decimal arithmetic by the cancelling quadratic formula, and again by the conjugate form $2C/(B+\sqrt{B^{2}-4C})$ . Compare both with the true smaller root, and state whether the subtraction of the two four-digit terms was itself inexact.

EXERCISE 1.4

Bound the truncation by $\tfrac12 |h|\,e^{|h|}$ . Build $e^{h}$ from its Taylor polynomial far enough to round to four digits, then form $( \mathrm{fl}(e^{h})-1)/h$ . At $h=10^{-4}$ the series $1+h+h^{2}/2$ already shows where $e^{h}$ sits between $1.000$ and $1.001$ .

At $h=0.2$ the truncation bound is $0.1\,e^{0.2}$ . The partial sum $1+0.2+0.02+0.00133333+0.000066667=1.221400$ . The tail is smaller than $0.2^{5}/120$ divided by $1-0.2/6$ , which is under $3\times 10^{-6}$ , so $e^{0.2}<1.221403$ and the truncation is under $0.1222$ . Four-digit rounding gives $1.221$ because the fifth digit is $4$ . The quotient is $(1.221-1)/0.2=1.105$ , absolute error $0.105$ against the derivative $1$ . Rounding the exponential moved the quotient by about $0.00040/0.2=0.002$ , so truncation dominates. At $h=10^{-4}$ the truncation is at most $\tfrac12\cdot 10^{-4}\cdot e^{0.0001}<5.1\times 10^{-5}$ . The series gives $e^{h}=1.000100005\ldots$ . The successor of $1.000$ is $1.001$ , and $1.0001$ is closer to $1.000$ , so $\mathrm{fl}(e^{h})=1.000$ and the quotient is $0$ . The absolute error against the derivative is $1$ . Roundoff dominates.

Let $T(h)=(e^{h}-1)/h$ estimate the derivative of $e^{t}$ at $t=0$ . In the four-digit decimal system, bound the truncation for $h=0.2$ and for $h=10^{-4}$ , compute both rounded quotients, and say which of truncation or roundoff dominates at each step. Use the Taylor expansion with remainder, evaluated by hand.

08 · Summary

Object	Statement
Normal set $F$	$\pm m\,\beta^{e-p}$ with $\beta^{p-1}\leq m\leq \beta^{p}-1$
Machine epsilon	gap $\varepsilon_{\mathrm{mach}}=\beta^{1-p}$ just above $1$
Unit roundoff	$u=\tfrac12\beta^{1-p}$ for round-to-nearest
One rounding	$\mathrm{fl}(x)=x(1+\delta)$ with $\delta$ at most $u$ in magnitude
Standard model	$\mathrm{fl}(x\star y)=(x\star y)(1+\delta)$ under the same bound, no over/underflow
Absolute vs relative	$u$ bounds relative error; absolute error scales with the magnitude
Exact cancellation	same-exponent differences in $F$ are exact if they do not underflow
Catastrophic cancellation	relative error inflated by $\kappa_{\mathrm{sub}}$ , magnitudes over the difference
Truncation vs roundoff	$T(h)-L$ and $\hat T(h)-T(h)$ scale oppositely in $h$
Link to linear systems	forward error carries a factor $\kappa(A)\cdot u$ ; $\kappa(A)$ is proved in Linear Algebra 20

Next: Chapter 02 — Root-Finding Algorithms, which uses this model when a bracketed root is refined by bisection, Newton, and secant updates: each update is exact arithmetic plus a rounding factor $1+\delta$ , and the truncation of the iteration is a separate term.