Number of Digits and Precision
4 Number of Digits and Precision
kNumber of Digits
Standard Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic Format dialog box is not selected.
•Up to 611 digits are stored in memory for integer values.
•Decimal values up to 15 digits are converted to fraction format and saved in memory. When a mathematical expression cannot be converted to fraction format, the result is displayed in decimal format.
•Values stored in memory are displayed
Decimal Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic Format dialog box is selected.
•Values stored in Ans memory have the same number of digits as they have when displayed. A value that is assigned to a variable has the same number of digits as defined for Standard mode values.
•Values are displayed in accordance with how [Display] settings (Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured.
•Displayed values are rounded to the appropriate number of decimal places.
•Some applications store values using a mantissa up to 15 digits long and a
kPrecision
•Internal calculations are performed using 15 digits.
•The error for a single mathematical expression (Decimal mode calculation error) is ±1 at the 10th digit. In the case of exponential display, calculation error is ±1 at the least
significant digit. Note that performing consecutive calculations causes error to be
cumulative. Error is also cumulative for internal consecutive calculations performed for: ^(xy), x , x!, nPr, nCr, etc.
•Error is cumulative and tends to be larger in the vicinity of a function’s singular point(s) and inflection point(s), and the vicinity of zero. With sinh(x) and tanh(x), for example, the inflection point occurs when x = 0. In this vicinity, error is cumulative and precision is poor.
20050501