On Fri, Jan 30, 2015 at 8:50 AM, Kevin Wheatley
<kevin.j....@...> wrote:

sounds like limits of float precision to me. that would mean being
'double' clean through the code,

looking through the code the matrix is stored as single precision
float and the bindings to python also assume as such, the image
processing also works as float, storing the matrix as doubles would be
possible, at the expense of some performance loss in processing
(speculation on my part :-). This assumes your python uses doubles
internally (likely the case).

Separate but related note:

there is some scattering of precision assumptions through the code
too, though these are mostly limited to output of files...

FLOAT_DECIMALS is set to 7 and there is a DOUBLE_DECIMALS used to
output things, however it should really be using something like:

2 + std::numeric_limits<Target>::digits * 3010/10000;

(or max_digits10() if we allow it)

Which would make the values in the code incorrect...

see http://www2.open-std.org/JTC1/SC22/WG21/docs/papers/2005/n1822.pdf
(or http://www.cs.berkeley.edu/~wkahan/ieee754status/IEEE754.PDF)

However be aware there are 'limitations' on some platforms (MSVC 7.1
for instance) where you can not rely on round tripping of values in
their standard library (dinkumware).

I might see if I can knock up a patch for the output precision problems.

Kevin