1. Messages
  2. Precision of float values in config generated from Python

Re: Precision of float values in config generated from Python

Haarm-Pieter Duiker <li...@...>
 

Thanks for digging into this, and my apologies for missing the obvious limit of float precision. I was running into an issues where the limited precision is combined with high-intensity values to produce different results than the reference implementation. I've found a workaround though. More precision preserved throughout the process would still be appreciated.

HP




On Fri, Jan 30, 2015 at 3:53 AM, Kevin Wheatley <kevin.j....@...> wrote:
There are cases in the code where the precision of the formatting is
used to generate Cache Ids, changing the precision here would change
the behavior  there are also cases with luts where certain precision
limits are assumed, my conclusion would thus be getting to full float
precision should be doable, but with careful adjustments, in
particular there might not be enough test coverage to ensure
maintaining the behaviour exactly.

Going to double precision to go beyond a maximum of 9 digits is a
different matter, does it make a visible difference tot he images?

Kevin

--
You received this message because you are subscribed to the Google Groups "OpenColorIO Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ocio-dev+u...@....
For more options, visit https://groups.google.com/d/optout.
Join ocio-dev@lists.aswf.io to automatically receive all group messages.