Re: Precision of float values in config generated from Python

Kevin Wheatley <kevin.j....@...>

There are cases in the code where the precision of the formatting is
used to generate Cache Ids, changing the precision here would change
the behavior there are also cases with luts where certain precision
limits are assumed, my conclusion would thus be getting to full float
precision should be doable, but with careful adjustments, in
particular there might not be enough test coverage to ensure
maintaining the behaviour exactly.

Going to double precision to go beyond a maximum of 9 digits is a
different matter, does it make a visible difference tot he images?


Join to automatically receive all group messages.