#### Precision of float values in config generated from Python

Haarm-Pieter Duiker <li...@...>

Hello,

Is there a way to configure the precision used for floating-point values stored in a config?

I have a Python script that generates a config based on a bunch of inputs, including a number of matrices that are declared in Python like so:

matrix = [0.6954522414, 0.1406786965, 0.1638690622,
0.0447945634, 0.8596711185, 0.0955343182,
-0.0055258826, 0.0040252103, 1.0015006723]

Matrices are added to a MatrixTransform through calls something like this:

ocio_transform = ocio.MatrixTransform()
ocio_transform.setMatrix(matrix)

We end up with a statement in the config that looks something like the following:

...
to_reference: !<MatrixTransform> {matrix: [0.695452, 0.140679, 0.163869, 0, 0.0447946, 0.859671, 0.0955343, 0, -0.00552588, 0.00402521, 1.0015, 0, 0, 0, 0, 1]}
...

Ignore the padding from 3x3 to 4x4. The values in the config print with 6, 7 or 8 decimals. The input values had 10.

Is there any way to preserve the precision of the original values? Would it be better to use the spimtx files to store these values?

HP

Kevin Wheatley <kevin.j....@...>

sounds like limits of float precision to me. that would mean being
'double' clean through the code,

Kevin

Kevin Wheatley <kevin.j....@...>

On Fri, Jan 30, 2015 at 8:50 AM, Kevin Wheatley
<kevin.j....@gmail.com> wrote:
sounds like limits of float precision to me. that would mean being
'double' clean through the code,
looking through the code the matrix is stored as single precision
float and the bindings to python also assume as such, the image
processing also works as float, storing the matrix as doubles would be
possible, at the expense of some performance loss in processing
(speculation on my part :-). This assumes your python uses doubles
internally (likely the case).

Separate but related note:

there is some scattering of precision assumptions through the code
too, though these are mostly limited to output of files...

FLOAT_DECIMALS is set to 7 and there is a DOUBLE_DECIMALS used to
output things, however it should really be using something like:

2 + std::numeric_limits<Target>::digits * 3010/10000;

(or max_digits10() if we allow it)

Which would make the values in the code incorrect...

see http://www2.open-std.org/JTC1/SC22/WG21/docs/papers/2005/n1822.pdf
(or http://www.cs.berkeley.edu/~wkahan/ieee754status/IEEE754.PDF)

However be aware there are 'limitations' on some platforms (MSVC 7.1
for instance) where you can not rely on round tripping of values in
their standard library (dinkumware).

I might see if I can knock up a patch for the output precision problems.

Kevin

Kevin Wheatley <kevin.j....@...>

There are cases in the code where the precision of the formatting is
used to generate Cache Ids, changing the precision here would change
the behavior there are also cases with luts where certain precision
limits are assumed, my conclusion would thus be getting to full float
precision should be doable, but with careful adjustments, in
particular there might not be enough test coverage to ensure
maintaining the behaviour exactly.

Going to double precision to go beyond a maximum of 9 digits is a
different matter, does it make a visible difference tot he images?

Kevin

Kevin Wheatley <kevin.j....@...>

There is also a bug in yaml-cpp which prevents this in
src/emitterstate.cpp it needs fixing to read:

bool EmitterState::SetFloatPrecision(int value, FMT_SCOPE scope)
{
if(value < 0 || value > (2 +
std::numeric_limits<float>::digits * 3010/10000))
return false;
_Set(m_floatPrecision, value, scope);
return true;
}

bool EmitterState::SetDoublePrecision(int value, FMT_SCOPE scope)
{
if(value < 0 || value > (2 +
std::numeric_limits<double>::digits * 3010/10000))
return false;
_Set(m_doublePrecision, value, scope);
return true;
}

Then in OCIO's code you need to add something like:

out.SetFloatPrecision(2 +
std::numeric_limits<float>::digits * 3010/10000);
out.SetDoublePrecision(2 +
std::numeric_limits<double>::digits * 3010/10000);

To the save function in OCIOYaml.cpp (near line 1664)

At least by that point ociocheck will read and write a config file
with more precision :-)

Kevin

Haarm-Pieter Duiker <li...@...>

Thanks for digging into this, and my apologies for missing the obvious limit of float precision. I was running into an issues where the limited precision is combined with high-intensity values to produce different results than the reference implementation. I've found a workaround though. More precision preserved throughout the process would still be appreciated.

HP

On Fri, Jan 30, 2015 at 3:53 AM, Kevin Wheatley wrote:
There are cases in the code where the precision of the formatting is
used to generate Cache Ids, changing the precision here would change
the behavior  there are also cases with luts where certain precision
limits are assumed, my conclusion would thus be getting to full float
precision should be doable, but with careful adjustments, in
particular there might not be enough test coverage to ensure
maintaining the behaviour exactly.

Going to double precision to go beyond a maximum of 9 digits is a
different matter, does it make a visible difference tot he images?

Kevin

--
You received this message because you are subscribed to the Google Groups "OpenColorIO Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ocio-dev+u...@....
For more options, visit https://groups.google.com/d/optout.

Dithermaster <dither...@...>

You need to print and later parse 8 digits to uniquely capture a float.

///d@

On Fri, Jan 30, 2015 at 6:32 PM, Haarm-Pieter Duiker wrote:
Thanks for digging into this, and my apologies for missing the obvious limit of float precision. I was running into an issues where the limited precision is combined with high-intensity values to produce different results than the reference implementation. I've found a workaround though. More precision preserved throughout the process would still be appreciated.

HP

On Fri, Jan 30, 2015 at 3:53 AM, Kevin Wheatley wrote:
There are cases in the code where the precision of the formatting is
used to generate Cache Ids, changing the precision here would change
the behavior  there are also cases with luts where certain precision
limits are assumed, my conclusion would thus be getting to full float
precision should be doable, but with careful adjustments, in
particular there might not be enough test coverage to ensure
maintaining the behaviour exactly.

Going to double precision to go beyond a maximum of 9 digits is a
different matter, does it make a visible difference tot he images?

Kevin

--
You received this message because you are subscribed to the Google Groups "OpenColorIO Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ocio-dev+u...@....
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenColorIO Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ocio-dev+u...@....
For more options, visit https://groups.google.com/d/optout.

 1 - 7 of 7