Date
1 - 7 of 7
Precision of float values in config generated from Python
|
Haarm-Pieter Duiker <li...@...>
Hello, Is there a way to configure the precision used for floating-point values stored in a config? I have a Python script that generates a config based on a bunch of inputs, including a number of matrices that are declared in Python like so: matrix = [0.6954522414, 0.1406786965, 0.1638690622, 0.0447945634, 0.8596711185, 0.0955343182, -0.0055258826, 0.0040252103, 1.0015006723] Matrices are added to a MatrixTransform through calls something like this: ocio_transform = ocio.MatrixTransform() ocio_transform.setMatrix(matrix) We end up with a statement in the config that looks something like the following: ... to_reference: !<MatrixTransform> {matrix: [0.695452, 0.140679, 0.163869, 0, 0.0447946, 0.859671, 0.0955343, 0, -0.00552588, 0.00402521, 1.0015, 0, 0, 0, 0, 1]} ... Ignore the padding from 3x3 to 4x4. The values in the config print with 6, 7 or 8 decimals. The input values had 10. Is there any way to preserve the precision of the original values? Would it be better to use the spimtx files to store these values? Thanks in advance for your help, HP |
|
|
|
Kevin Wheatley <kevin.j....@...>
sounds like limits of float precision to me. that would mean being
'double' clean through the code, Kevin |
|
|
|
Kevin Wheatley <kevin.j....@...>
On Fri, Jan 30, 2015 at 8:50 AM, Kevin Wheatley
<kevin.j....@...> wrote: sounds like limits of float precision to me. that would mean beinglooking through the code the matrix is stored as single precision float and the bindings to python also assume as such, the image processing also works as float, storing the matrix as doubles would be possible, at the expense of some performance loss in processing (speculation on my part :-). This assumes your python uses doubles internally (likely the case). Separate but related note: there is some scattering of precision assumptions through the code too, though these are mostly limited to output of files... FLOAT_DECIMALS is set to 7 and there is a DOUBLE_DECIMALS used to output things, however it should really be using something like: 2 + std::numeric_limits<Target>::digits * 3010/10000; (or max_digits10() if we allow it) Which would make the values in the code incorrect... see http://www2.open-std.org/JTC1/SC22/WG21/docs/papers/2005/n1822.pdf (or http://www.cs.berkeley.edu/~wkahan/ieee754status/IEEE754.PDF) However be aware there are 'limitations' on some platforms (MSVC 7.1 for instance) where you can not rely on round tripping of values in their standard library (dinkumware). I might see if I can knock up a patch for the output precision problems. Kevin |
|
|
|
Kevin Wheatley <kevin.j....@...>
There are cases in the code where the precision of the formatting is
used to generate Cache Ids, changing the precision here would change the behavior there are also cases with luts where certain precision limits are assumed, my conclusion would thus be getting to full float precision should be doable, but with careful adjustments, in particular there might not be enough test coverage to ensure maintaining the behaviour exactly. Going to double precision to go beyond a maximum of 9 digits is a different matter, does it make a visible difference tot he images? Kevin |
|
|
|
Kevin Wheatley <kevin.j....@...>
There is also a bug in yaml-cpp which prevents this in
src/emitterstate.cpp it needs fixing to read: bool EmitterState::SetFloatPrecision(int value, FMT_SCOPE scope) { if(value < 0 || value > (2 + std::numeric_limits<float>::digits * 3010/10000)) return false; _Set(m_floatPrecision, value, scope); return true; } bool EmitterState::SetDoublePrecision(int value, FMT_SCOPE scope) { if(value < 0 || value > (2 + std::numeric_limits<double>::digits * 3010/10000)) return false; _Set(m_doublePrecision, value, scope); return true; } Then in OCIO's code you need to add something like: out.SetFloatPrecision(2 + std::numeric_limits<float>::digits * 3010/10000); out.SetDoublePrecision(2 + std::numeric_limits<double>::digits * 3010/10000); To the save function in OCIOYaml.cpp (near line 1664) At least by that point ociocheck will read and write a config file with more precision :-) Kevin |
|
|
|
Haarm-Pieter Duiker <li...@...>
Thanks for digging into this, and my apologies for missing the obvious limit of float precision. I was running into an issues where the limited precision is combined with high-intensity values to produce different results than the reference implementation. I've found a workaround though. More precision preserved throughout the process would still be appreciated. HP On Fri, Jan 30, 2015 at 3:53 AM, Kevin Wheatley <kevin.j....@...> wrote: There are cases in the code where the precision of the formatting is |
|
|
|
Dithermaster <dither...@...>
You need to print and later parse 8 digits to uniquely capture a float. ///d@ On Fri, Jan 30, 2015 at 6:32 PM, Haarm-Pieter Duiker <li...@...> wrote:
|
|
|