From: Thomas Walker Lynch Date: Sun, 9 Aug 2026 17:22:35 +0000 (+0000) Subject: . X-Git-Url: https://git.reasoningtechnology.com/%5B%5E?a=commitdiff_plain;h=fc4fbc1f831331c27a22030f2d793a6070ebc676;p=TM-2026 . --- diff --git a/document/book/TM-2026.html b/document/book/TM-2026.html index a0b95d9..8c40e5c 100644 --- a/document/book/TM-2026.html +++ b/document/book/TM-2026.html @@ -112,19 +112,35 @@ Preface -

One afternoon I was fortunate to have lunch in Berkeley with Bill Kahan, a pioneer in applied number theory, and our conversation turned to errors in numeric computation. I observed that an error is merely an opportunity to do more computing. Bill seemed to like the remark; he repeated it in a Dr. Dobb's interview, where it was credited to him.TODO·cite: the Dr. Dobb's interview, issue and date. When we met again, I asked him about it. He apologized, explaining that the interviewer had created something rather than reporting it, and that he was himself surprised by what he read. The apology was unnecessary. I had intended to summarize Kahan's own work. He was instrumental in creating IEEE Std 754, where an exceptional condition, more informally an 'error', need not end computation but can serve as a gateway to a higher symbolic algebra. I found that architecture inspiring, and my comment had been meant to acknowledge the debt.

+

Bill Kahan had been instrumental in the specification of IEEE Std 754. Kahan had added some very insightful features. For example the representation switches to denormalized numbers so as to approach zero as closely as possible, with nothing wasted. Kahan also added a substitution flag for Infinity, which when the reciprocal was taken would produce zero. His stated reasoning was to approximate limiting arithmetic. What I saw in that was not limiting arithmetic per sé but rather an escape from a first order computation to something higher order. I wondered if there was not an analogy, perhaps a shadowing, with non-standard analysis.

+ +

I went to Berkeley to talk to Bill about this, and we had lunch. We discussed errors and what do to with them. Bill was thinking in terms of interrupts and pipelines. I suggested a different path, that an error was merely an opportunity to do more computing. That the code design should be aware of the operands that caused errors, and thus they were not something to interrupt from, or to insert a symbolic value, but rather a branch, like any other. I was hoping that Bill would be interested in discussing it further.

-

In practical computing the symbol algebra of IEEE Std 754 quickly devolves into the catch-all indeterminate value, the so-called NaN, which tends to leave programmers wishing to extend the system. Kulisch and Miranker attempted to formalize computer arithmetic, and started by reasoning about a screen that lay over the real number space.Ulrich W. Kulisch and Willard L. Miranker, Computer Arithmetic in Theory and Practice (New York: Academic Press, 1981). Boehm and Cartwright introduced a concept of Constructive Real Numbers, where precision of computation was extended upon demand. And through Stephen Wolfram's vision, Mathematica provides a complete symbolic math manipulation package, where a symbolic expression can be reduced before being evaluated. Mathematica then tracks loss of precision through significance arithmetic. In my own work, I have suggested an analysis step first, where required precision is derived, followed by computation. However, among these there is still missing a formalism of passing through the gateway upon an error, so to speak. Mathematica comes the closest. Each of these is taken up in its turn later in this book, and the citations are given there.

+

No long after Bill gave an interview with Dr. Dobb's Jack Woehr, "A Conversation with William Kahan," Dr. Dobb's Journal, November 1997, archived at https://jacobfilipp.com/DrDobbs/articles/DDJ/1997/9711/9711a/9711a.htm. The published piece opens with Kahan giving as his own thesis that exceptions are not errors unless they are handled badly, and that they are opportunities for extra computation. A related interview, "An Interview with the Old Man of Floating-Point," February 20, 1998, is sometimes cited alongside it. and there at the top was the exact quote that exceptions are opportunities for further processing.

-

John Gustafson also brought up the topic of the nature of error shortly before publishing The End of Error. He now has another proposal, the posit, which abandons the variable width operands of his earlier unums in favour of a fixed width whose internal field boundaries move, so that accuracy is greatest for magnitudes near one and tapers toward the extremes of the range. John grilled me over the method I had used, and what he was pressing on was the question of how a computation is to know what precision it needs. I answered with High Radix Online Arithmetic and with the importance of variable precision, but I could not formulate the rest of the thoughts. I had learned a great deal while doing formal analysis of the algorithms I had put on the Am29050, Am486, and K5 processors at AMD, yet when I tried to articulate this, I only stammered. I feel that I fell short, as all I could do was point at my already published work at that time on High Radix Online Arithmetic. So this volume is my answer to John's question. It is a little late, well years late, but I hope it will still have some value.

+

So of course I called Bill to ask about this. I had him on the spot, it was going to be an interesting conversation. Bill explained that the interviewer had edited the interview to make it dramatic, and that he was himself surprised upon reading it. I accepted that explanation, but I still wondered when was I going to get an opportunity to discuss my concepts? And I also know what it is like as I once had done a bad job discussing a Balgarian student's work at a conference in France, something I still feel embarrassed about when thinking about it.

-

There is a first edition of this book.TODO·cite: the first edition, and the Quicklisp release of the Lisp library. The same concepts are described in those pages, but the story is not as well told, and no one has come full circle to say that it made sense. Well, one reader did come back. Kelvin Goveas, a colleague and a dear friend, reviewed the first edition, admitted he was bewildered by it, and asked whether I was proposing that we build Turing Machines. After all, we are the music makers,Arthur O'Shaughnessy, "Ode", in Music and Moonlight (London: Chatto and Windus, 1874). in the music business, so to speak; only instead of being written in notes, our music is written in transistors and layout. So then, he rightly asked, what is the proposal here? I have yet to give him an answer. The best I can do is point him at Mitch Alsup's architecture book,TODO·cite: Mitch Alsup's architecture book. Title and publication details to be confirmed; his My 66000 architecture is the subject. where the proposals are, and go on pondering how I can tell this story differently.

+

John Gustafson also brought up the topic of the nature of error shortly before publishing The End of Error. He now has another proposal, the Posit, which abandons the variable width operands of his earlier unums in favour of a fixed width whose internal field boundaries move, so that accuracy is greatest for magnitudes near one and tapers toward the extremes of the range. This makes small numbers work even better than denorms did. John was polite and asked me about the method I had used to achieve accuracy in microcode. I answered with High Radix Online Arithmetic and with the importance of variable precision, but I could not formulate the rest of the thoughts. I had learned a great deal while doing formal analysis of the algorithms I had put on the Am29050, Am486, and K5 processors at AMD, yet when I tried to articulate this, I only stammered. So here passed a second opportunity to discuss the concepts of levels of analysis.

-

There is no hardware specification for a new machine in this book, and no proposal to build a Turing Machine in hardware. This is an absolute. The RT Machine is an improved computation theoretic model, and what distinguishes it from the Turing Machine is that it can be realized, which is what saves the Naturalist thesis. Its organization is described in these pages, and building one might be interesting, but there is no suggestion here that microprocessors should be made that way. This volume is the second edition of the first of four that were envisioned, leading in the end to the TTCA Machine, which is a real machine, and which I proposed building in 2005. What this book does is lay the foundation for that proposal, and explain and justify the features such a machine will need. The architecture itself is not found in these pages.

+

Others have also addressed the problem of errors in computation. Moore originally introduce interval methods, where two numbers would bound a true value. Kulisch and Miranker attempted to formalize computer arithmetic, and started by reasoning about a screen that lay over the real number space, which is a route to the interval methods they embedded in their XSC languages. Jean-Michel Muller found a way to produce correctly rounded to full precision transcendental functions. Boehm and Cartwright introduced a concept of Constructive Real Numbers, where precision of computation was extended upon demand. And through Stephen Wolfram's vision, Mathematica provides a complete symbolic math manipulation package, where a symbolic expression can be reduced before being evaluated. Mathematica then tracks loss of precision through significance arithmetic. In my own work at AMD, employed an analysis step first, where required precision was derived, followed by variable precision computation.

-

There is a practical aspect to this work. A software library that is of general usefulness can be found in these pages, and downloaded from GitHub. It was first articulated in the Lisp language in the first edition of the book and released on Quicklisp. Lisp is in a sense pure, and formal statements can be made with it, so writing the library there kept it a small and natural step away from the formal discussion that precedes it. Since that time I have written parts of the library in other languages and found it useful. This volume presents the Python API.

+

However, all of these methods are missing a key insight. That the further computation that the error invites us to engage in, is part of a richer second order system. The various systems mentioned above remain in the first order, though perhaps with a hint of the entering a second order analysis.

-

A Peano Number is a computational structure, as its name records. It starts with emptiness, a zero, and is then built up by repeated application of the successor function. Because a number is a computational structure, number theory was already computation theory before Church, Turing, and their contemporaries gave the subject its name. It is a computation theory whose discussion is confined to the relationships among Peano structures, which is to say, to Peano Numbers. Then the activity of writing algorithms and designing circuits that relate those structures to one another, in the service of solving real problems, is applied number theory.

+

In practical computing the error algebra of IEEE Std 754 error can quickly mount due to the substitution flags, and the system tends towards the catch-all indeterminate value, the so-called NaN. This effect leaves some computer scientists wishing to extend the system. However this is not a failure of the system. The flags were intended to assist a numerical analysis expert in performing limiting arithmetic, not to make computations that have errors to work without being specially designed to do so. Extensions to the system do not usually fair much better, because error tag algebra can not serve as computation.

+ +

Intervals will encompass error causing singularities that otherwise would not have manifested. However, the intervals tend to pessimistically widen and then pull in extraneous singularities. Exceptions can happen at run time without the code designer knowing of the potential for them. And when an except does occur, it tends to be handled ad hoc in the manner all exceptions are, for example by exiting the program. The interval system does not provide further guidance.

+ +

The Constructive Real is reminiscent of a limit, it will force the computation to provide sufficient precision. A limit concept is indeed key when moving from first order computation to a second order, but such an second order system is not built in. As for interval methods, an error could surprise an algorithm design when it appears at run time, and thus the code might not be prepared to handle it. However, there is an interesting aspect here, the expression syntax trees are available to the exception handler, or at least could be, so a layer could be built on top.

+ +

Mathematica uses symbolic manipulation of expressions combined with significance arithmetic, so numeric resolution can fail, and then the user is left holding the error as a result. However, similarly to how the Constructive Real system has syntax trees, the exception handler can have access to the unresolved symbolic expression, which he could perform more computation with if he knew what to do with it. This is interesting.

+ +

There is a first edition of this book. The same concepts are described in those pages, but the story is not as well told, and no one has come full circle to say it made any sense to them. Well, one colleague, after reviewing the first edition, admitted he was bewildered by it and asked if I was proposing that we build Turing Machines. After all we are the music makers, in the music business, so to speak; only instead of being written in notes, our music is written in transistors and layout. So then, he rightfully asks, what is the proposal here? I can only point him at Mitch Alsup's book, and ponder how I can tell this story differently.

+ +

There is a first edition of this book.TODO·cite: the first edition, and the Quicklisp release of the Lisp library. The same concepts are described in those pages, but the story is not as well told, and no one has come full circle to say that it made sense. One colleague who reviewed it admitted he was bewildered by it, and asked whether I was proposing that we build Turing Machines. After all, we are the music makers,Arthur O'Shaughnessy, "Ode", in Music and Moonlight (London: Chatto and Windus, 1874)., so to speak; and we write notes in transistors and layout. So then, he rightly asked, "What is the proposal here?" I have yet to give him an answer. Perhaps this second edition will help.

+ +

There is no hardware specification for a new proposed commercial machine in this book. This book is not proposing to build in hardware a Turing Machine. This is an absolute. Building a RT machine might be interesting. It is true that the organization for it is described in this book. Yet this is certainly not the proposal here. Rather this book explains and justifies a set of features that are planned to be incorporated into a commercial machine, one which I proposed to model and build in 2005. There was some interest, but ultimately we did not raise money for it.

+ +

However, there is a practical aspect to this work. A software library that is of general usefulness can be found in these pages, and downloaded from GitHub. It was first articulated in the Lisp language in the first edition of the book and released on Quicklisp. It is described in an appendix. Lisp is in a sense pure, and formal statements can be made with it, so writing the library there kept it a small and natural step away from the formal discussion that precedes it. Since that time I have written parts of the library in other languages and found it useful. This volume develops the Python API, and the Python version is to be made available on Github.

 @@ -3547,7 +3563,7 @@ second-order Arithmetic

- In their 1990 paper, "Exact Real Arithmetic: Formulating Real Numbers as Functions," Hans-Juergen Boehm and Robert Cartwright presented a system using Turing Machines to represent 'constructive real numbers', or exact real arithmetic. This directly implements a core concept from Alan Turing's 1936 paper of the "computable number." Hans-J. Boehm and Robert Cartwright, "Exact Real Arithmetic: Formulating Real Numbers as Functions," in Design and Implementation of Symbolic Computation Systems (Berlin: Springer, 1990), 43 52. Boehm translated this computation theory into a practical software architecture. Instead of storing a real number as a fixed-width floating-point approximation, Boehm's system represents a real number as a lazy evaluation function. + In their 1990 paper, "Exact Real Arithmetic: Formulating Real Numbers as Functions," Hans-Juergen Boehm and Robert Cartwright presented a system using Turing Machines to represent 'constructive real numbers', or exact real arithmetic. This directly implements a core concept from Alan Turing's 1936 paper of the "computable number." Hans-J. Boehm and Robert Cartwright, "Exact Real Arithmetic: Formulating Real Numbers as Functions," in D. A. Turner, ed., Research Topics in Functional Programming (Reading: Addison-Wesley, 1990), 43–64. The original presentation is Hans-J. Boehm, Robert Cartwright, Mark Riggle, and Michael J. O'Donnell, "Exact Real Arithmetic: A Case Study in Higher Order Programming," in Proceedings of the 1986 ACM Conference on LISP and Functional Programming (New York: ACM Press, 1986), 162–173. Boehm translated this computation theory into a practical software architecture. Instead of storing a real number as a fixed-width floating-point approximation, Boehm's system represents a real number as a lazy evaluation function.

@@ -4499,6 +4515,14 @@

The point of this chapter is to state that computational analysis exists, rather than to present a comprehensive computational analysis system. If a person is looking for practical systems for extending functions that expand precision, or for second-order computation, there are options.

+

The oldest of the options carries the error inside the value rather than reporting it alongside. In interval arithmetic a quantity is a pair of bounds, and every operation returns an interval guaranteed to contain the true result, so a computation ends holding an answer together with a proof of that answer's accuracy. Ramon Moore conceived of the method in 1958, set out its principles in his 1962 Stanford dissertation, and gave it its standard treatment in Interval Analysis.R. E. Moore, Interval Arithmetic and Automatic Error Analysis in Digital Computing, Ph.D. dissertation, Department of Mathematics, Stanford University, November 1962, issued as Applied Mathematics and Statistics Laboratories Technical Report No. 25. R. E. Moore, Interval Analysis (Englewood Cliffs: Prentice-Hall, 1966); and Methods and Applications of Interval Analysis (Philadelphia: SIAM, 1979). The method now has a standard of its own, IEEE Std 1788-2015. +

+ +

Kulisch and Miranker's screen is what joins interval arithmetic to the formalization of computer arithmetic. The values a machine can represent form a screen lying over the real number space, and a rounding carries the reals onto it. An operation is then defined first over the powerset of the reals, where it is exact, and that exact result is rounded onto the screen, or onto the intervals over the screen, by taking the least interval which encloses it. Machine arithmetic becomes the image of exact arithmetic under a rounding, and the interval is what the rounding leaves behind, which is to say it is the error made part of the value. Kulisch carried the same principle to the exact dot product, where a long accumulator holds a sum of products with no intermediate rounding at all, and to the XSC languages, which put both the intervals and the accumulator in reach of a programmer.Ulrich W. Kulisch and Willard L. Miranker, Computer Arithmetic in Theory and Practice (New York: Academic Press, 1981). For the extension to product spaces see the same authors' "Arithmetic Operations in Interval Spaces," in G. Alefeld and R. D. Grigorieff, eds., Fundamentals of Numerical Computation, Computing Supplementum vol. 2 (Vienna: Springer, 1980), 51–67. +

+ +

What interval arithmetic does not do is keep its bounds tight. An interval records the range of a quantity but forgets the quantity's identity, so the two occurrences of x in x - x are treated as though they were independent, and the result is an interval about zero rather than zero. Bounds therefore widen through a long computation whether or not the underlying quantities are correlated, until what they report is rigorous and useless together. This dependency problem is why interval arithmetic, for all that it proves what it claims, did not displace the alternatives, and it is why the systems described next pursue precision rather than bounds.

+

Hans J. Boehm and Robert Cartwright's work of expanding the precision of a computation on demand continued to mature. Boehm, while working at Google, authored a Java constructive reals library that powers the bundled Android calculator. When a person types in an expression, the system builds an Abstract Syntax Tree (AST) of the expression. The display interface determines how many digits fit on the screen and demands exactly that much precision from the root of the tree. The root function then recursively demands progressively higher precision from its operand functions until it achieves the strict error bounds required to guarantee every digit displayed on the screen is mathematically correct Hans J. Boehm, "Small data computing: Correct calculator arithmetic," Google Research, 2015. This library uses constructive real arithmetic to guarantee fully accurate results through demand driven evaluation..

@@ -4511,7 +4535,7 @@

None of these computation systems, that of Boehm and Cartwright, Mathematica, nor my error analysis approach, makes use of the IEEE 754 standard floating-point arithmetic. Rather they all require the use of variable precision. In the case of Boehm and Cartwright's this occurs through the serialization implied through lazy calls for more precision. In the case of Mathematica it is explicit in the significance arithmetic. In my error analysis approach, the means for variable precision was the High Radix Online Arithmetic Thomas W. Lynch, "High Radix On Line Arithmetic for Credible and Accurate Computing," Real Numbers and Computers, École des Mines de Saint Étienne, France, 1995, pp. 78 89. Thomas W. Lynch and Michael J. Schulte, "Software for High Radix On Line Arithmetic," Reliable Computing, vol. 2, no. 2, 1996, pp. 133 138. DOI: 10.1007/BF02425915..

-

Hence what would be useful for a replacement of the IEEE 754, would be a variable precision number standard. There is one being proposed by John Gustafson called the Posit number representation John L. Gustafson, The End of Error: Unum Computing, Chapman and Hall/CRC, 2015. Posits serve as a hardware friendly alternative to IEEE 754 floating-point numbers, providing variable precision mechanics.. +

Hence what would be useful for a replacement of the IEEE 754, would be a variable precision number standard. John Gustafson's earlier unum proposal was of that kind, carrying an operand of variable width. His later and more widely taken up Posit representation moved away from variable width operands in favour of a fixed width whose internal field boundaries move, so that accuracy is greatest for magnitudes near one and tapers toward the extremes of the range. A Posit is therefore an improvement upon the IEEE 754 within a fixed width, rather than the variable precision standard called for here John L. Gustafson, The End of Error: Unum Computing (Boca Raton: Chapman and Hall/CRC, 2015). John L. Gustafson and Isaac T. Yonemoto, "Beating Floating Point at its Own Game: Posit Arithmetic," Supercomputing Frontiers and Innovations 4, no. 2 (2017): 71–86. John L. Gustafson, Every Bit Counts: Posit Computing (Boca Raton: Chapman and Hall/CRC, 2024). The 2017 paper states plainly that, unlike the earlier unum forms, Posits require neither interval arithmetic nor variable size operands..

This section reviewed finite difference methods as a means to implement function extensions. A person should notice that finite differences are one among many methods for extending functions, and they are not always the best choice. In modern computing it is not common to implement function extensions. Rather it is more common to increment a value across a domain, while completely repeating the evaluation for each new domain value, for example when computing values to place on a plot. However, independent of their importance in applied computing, function extension and difference methods are of theoretical importance for analysis, particularly when applied to nonstandard analysis. These concepts speak directly to the thesis of computational naturalism—deriving mathematics from computational logic—rather than merely proposing practical methods for implementing software.