Re: Precision of float values in config generated from Python


Kevin Wheatley <kevin.j....@...>
 

There is also a bug in yaml-cpp which prevents this in
src/emitterstate.cpp it needs fixing to read:

bool EmitterState::SetFloatPrecision(int value, FMT_SCOPE scope)
{
if(value < 0 || value > (2 +
std::numeric_limits<float>::digits * 3010/10000))
return false;
_Set(m_floatPrecision, value, scope);
return true;
}

bool EmitterState::SetDoublePrecision(int value, FMT_SCOPE scope)
{
if(value < 0 || value > (2 +
std::numeric_limits<double>::digits * 3010/10000))
return false;
_Set(m_doublePrecision, value, scope);
return true;
}

Then in OCIO's code you need to add something like:

out.SetFloatPrecision(2 +
std::numeric_limits<float>::digits * 3010/10000);
out.SetDoublePrecision(2 +
std::numeric_limits<double>::digits * 3010/10000);

To the save function in OCIOYaml.cpp (near line 1664)

At least by that point ociocheck will read and write a config file
with more precision :-)

Kevin

Join ocio-dev@lists.aswf.io to automatically receive all group messages.