Re: Precision of float values in config generated from Python
Kevin Wheatley <kevin.j....@...>
On Fri, Jan 30, 2015 at 8:50 AM, Kevin Wheatley
<kevin.j....@...> wrote: sounds like limits of float precision to me. that would mean beinglooking through the code the matrix is stored as single precision float and the bindings to python also assume as such, the image processing also works as float, storing the matrix as doubles would be possible, at the expense of some performance loss in processing (speculation on my part :-). This assumes your python uses doubles internally (likely the case). Separate but related note: there is some scattering of precision assumptions through the code too, though these are mostly limited to output of files... FLOAT_DECIMALS is set to 7 and there is a DOUBLE_DECIMALS used to output things, however it should really be using something like: 2 + std::numeric_limits<Target>::digits * 3010/10000; (or max_digits10() if we allow it) Which would make the values in the code incorrect... see http://www2.open-std.org/JTC1/SC22/WG21/docs/papers/2005/n1822.pdf (or http://www.cs.berkeley.edu/~wkahan/ieee754status/IEEE754.PDF) However be aware there are 'limitations' on some platforms (MSVC 7.1 for instance) where you can not rely on round tripping of values in their standard library (dinkumware). I might see if I can knock up a patch for the output precision problems. Kevin
|
|