User's question: “I have to enter in measurements to HurryCover that have more than four decimal point, such as .859375 inches. HC lets you enter this but then rounds it up to four spaces instead of the six that were entered. Is it actually making it the correct size and rounding up the number?”…
Interesting question! As you probably have observed, InDesign itself does not display all entered digits in the Transform panel. If you set a rectangle to be .859375 inches wide, what you get in the GUI is
0.8594 in. As far as I know InDesign's decimal precision is 1/10,000 (up to 4 decimal points) and users have no control over this.
At a lower technical level, all measurements in InDesign are coerced into PostScript points with a decimal precision which can't be higher than 7 digits since the underlying code (cf. InDesign's SDK) uses a machine epsilon of
1e-8, as shown in the header
PMReal.h. As Adobe developers comment, the class
PMReal is used, instead of double, “primarily to hide the epsilon comparisons behind operator definitions.” This also means that “implicit conversion happens when going to a PMReal”.
For all these reasons I strongly (although empirically) suspect that InDesign can't actually handle a precision deeper than
.001 pt (that is,
.0003528 mm, or
.3528 µm.) Since one international inch is 72 (PostScript) points, a precision of
.0001 inches is equivalent to
.0072 pt, that is
2.54016 µm. Note that a red blood cell is about 7 µm. So, in the considered example (
".859375 in") the last two digits likely have the order of magnitude of the cell membrane thickness.
How Does HurryCover Handle Those Decimals?
Anyway, HurryCover tries to deal at best as it can with precise measurements. Actually, all values are internally stored as integers in a custom micropoint unit. In this unit,
1 represents one millionth of a PostScript point. The highest value that HurryCover allows (maximum height) is 3,685,000,000 micropoints (about 1.3 m.) Since ExtendScript can safely handle signed integers in the range
)-2^53,+2^53(, HurryCover's internal unit avoids many floating point arithmetic issues.
What the user sees, however, is a rounded expression of the underlying numbers. When the [PT] unit is in use values are shown with 2 decimal digits. When [IN] is used, up to 4 decimal digits are displayed. In either case, the value you entered is converted into, and stored as, a micropoint number. For example, setting manually the spine to
".859375 in" results in storing the number 61,875,000 (micropoints.) As you can check this exactly reflects the original input, since
61.875 / 72 = 0.859375.
So, although HurryCover displays
"0.8594in" in the main dialog, it actually sends the value 61.875 pt to InDesign in that specific case. From then, we can't know what kind of approximations the application will apply. (As said above I do not bet on an accuracy greater than .001 pt.)
Note. — Incidentally, it should also be pointed out that HurryCover's precision may be impacted by multiple unit conversions. Indeed, if the user switches from [IN] to [PT], then edit some fields, then goes back to [IN], the script has to decide whether the internal values need to be changed with respect to the edited values.