Precision of float values in config generated from Python


Haarm-Pieter Duiker <li...@...>
 

Hello,

Is there a way to configure the precision used for floating-point values stored in a config?

I have a Python script that generates a config based on a bunch of inputs, including a number of matrices that are declared in Python like so:

matrix = [0.6954522414, 0.1406786965, 0.1638690622,
                   0.0447945634, 0.8596711185, 0.0955343182,
                   -0.0055258826, 0.0040252103, 1.0015006723]

Matrices are added to a MatrixTransform through calls something like this:

ocio_transform = ocio.MatrixTransform()
ocio_transform.setMatrix(matrix)


We end up with a statement in the config that looks something like the following:

...
    to_reference: !<MatrixTransform> {matrix: [0.695452, 0.140679, 0.163869, 0, 0.0447946, 0.859671, 0.0955343, 0, -0.00552588, 0.00402521, 1.0015, 0, 0, 0, 0, 1]}
...

Ignore the padding from 3x3 to 4x4. The values in the config print with 6, 7 or 8 decimals. The input values had 10. 

Is there any way to preserve the precision of the original values? Would it be better to use the spimtx files to store these values?

Thanks in advance for your help,
HP




Join ocio-dev@lists.aswf.io to automatically receive all group messages.