XML Profile Format


Malcolm Humphreys <malcolmh...@...>
 

Are people super into the idea of using xml?

We used xml for nearly everything at RSP, over time it became clear that it's a pretty poor choice for a lot of things.

One of the developers introduced the use of YAML (http://www.yaml.org/ http://en.wikipedia.org/wiki/YAML) and it
took off with everyone, it happened again when I recommended it's introduction at DrD (where pretty much
everything was XML). RV has started to use it for it's package descriptions as well.

As I don't think any other apps will actually be writing ocio profiles other than the ocio api, and people will be hand
writing these profiles. I don't see a benefit to using xml for a profile description, if we using as an interchange format
then maybe.

btw. I really don't like file format discussions but I think this would really help ocio profile readability. If people are
into the idea I'm happy to do the leg work to get it to work.

I have attached a small demo python script for you to check out, it mostly shows off how to define your own
custom yaml serialisation. You need PyYaml (http://pyyaml.org/) to run the demo.

Why I like it:
- super simple to parse and serialise objects
- human readable / editable
- easy to define your own custom types
- using yaml in python is so simple

It would be quite easy to have binary data (eg file luts) interweaved inside a yaml file.
http://en.wikipedia.org/wiki/YAML#Other_specified_data_types
--snip--
picture: !!binary |
R0lGODlhDAAMAIQAAP//9/X
17unp5WZmZgAAAOfn515eXv
Pz7Y6OjuDg4J+fn5OTk6enp
56enmleECcgggoBADs=mZmE
--snip--

http://www.yaml.org/spec/1.2/spec.html#id2759572 - Relation to JSON
http://www.yaml.org/spec/1.2/spec.html#id2759732 - Relation to XML

YAML
--snip--
ocs_profile_version: 1
resource_path: luts
luma_b: '0.0722'
luma_g: '0.7152'
luma_r: '0.2126'
role_compositing_log: lgh
role_scene_linear: lnh
strictparsing: true
displays:
Film1D: !Display {colorspace: vd8, device: sRGB}
Log: !Display {colorspace: lg10, device: sRGB}
colorspaces:
dt8: !ColorSpace
bitdepth: 8ui
description: null
family: vd8
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
- !File {interpolation: null, src: diffusemult.spimtx}
- !ColorSpaceTransform {dst: lnh, src: vd8}
lg10: !ColorSpace
bitdepth: 10ui
description: lg10 converts from scanned film to the show linear reference space.
This is a custom conversion based on several Kodak acquisition stocks. All three
color channels have the same response and there is no cross talk. This transform
has a shoulder and toe compensation. This 10bit uiint version is used for DI
deliveries.
family: lg
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
lnh: !ColorSpace
bitdepth: 16f
description: The show reference space. This is a sensor referred linear representation
of the scene with primaries that correspond to scanned film. 0.18 inthis space
corresponds to a properly exposed 18% grey card.
family: ln
gpuallocation: lg2
gpumax: 6
gpumin: -15
isdata: false
to_reference: null
vd8: !ColorSpace
bitdepth: 8ui
description: vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference
colorspace. This is a 1-d representation does not include any subtractive color
mixing or crosstalk compensation. This is commonly used to interchange files
between internal and external vendors. This is used as the interchange space
for final images on direct to video or television products.
family: vd8
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
- !File {interpolation: linear, src: vd10.spi1d}
--snip--

XML
--snip--
<ocioconfig version="1" resourcepath="luts" strictparsing="true" luma_r="0.2126" luma_g="0.7152" luma_b="0.0722" role_scene_linear="lnh" role_compositing_log="lgh">
<display device="sRGB" name="Film1D" colorspace="vd8"/>
<display device="sRGB" name="Log" colorspace="lg10"/>
<colorspace name="lnh" family="ln" bitdepth="16f" isdata="false" gpuallocation="lg2" gpumin="-15" gpumax="6">
<description>The show reference space. This is a sensor referred linear representation of the scene with primaries that correspond to scanned film. 0.18 in this space corresponds to a properly exposed 18% grey card. </description>
</colorspace>
<colorspace name="lg10" family="lg" bitdepth="10ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>lg10 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 10bit uiint version is used for DI deliveries.</description>
<to_reference>
<group>
<file src="lg10.spi1d" interpolation="nearest" />
</group>
</to_reference>
</colorspace>
<colorspace name="lg16" family="lg" bitdepth="16ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>lg16 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 16bit uiint version is used for DI deliveries in the rare case where this is requested. </description>
<to_reference>
<group>
<file src="lgf.spi1d" interpolation="nearest" />
</group>
</to_reference>
</colorspace>
<colorspace name="lgh" family="lg" bitdepth="16f" isdata="false" gpuallocation="uniform" gpumin="-0.2" gpumax="1.615">
<description>Converts from scanned film to the show linear reference space. This is a custom curve based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This floating point version of the conversion that includes a large range of over-shoot and under-shoot values. This allows for colortiming offsetsto be applied and reversed out losslessly. For internal use only. </description>
<to_reference>
<group>
<file src="lgf.spi1d" interpolation="linear" />
</group>
</to_reference>
</colorspace>
<colorspace name="vd8" family="vd" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference colorspace. This is a 1-d representation does not include any subtractive color mixing or crosstalk compensation. This is commonly used to interchange files between internal and external vendors. This is used as the interchange space for final images on direct to video or television products. </description>
<to_reference>
<group>
<file src="vd10.spi1d" interpolation="linear" />
</group>
</to_reference>
</colorspace>
<colorspace name="dt8" family="dt" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description> </description>
<to_reference>
<group>
<file src="diffusemult.spimtx"/>
<colorspacetransform src="vd8" dst="lnh"/>
</group>
</to_reference>
</colorspace>
</ocioconfig>
--snip--



.malcolm


Jeremy Selan <jeremy...@...>
 

I'm completely in favor of making the OCIO configuration files easier
to read and edit.

I have a few specific concerns about YAML, but it's definitely worth
exploring. Care to take a stab at it?

In defense of XML ...

* It's the 'no format decision' decision. Support is ubiquitous, and
no one ever wonders, "what the hell is this?" YAML is not nearly as
widely known. Never having looked at YAML before, when I first looked
at your configuration file I was wondering which formatting parts were
important. But this could be made more obvious with a few tweaks to
the outputter format.

* We've already got the XML code working.

* The ASC CDL defines an xml spec which I would like to support (for
interoperability with other CDL apps), so we're not going to be be
able to ditch tinyxml.

* This may not be wise in the long term, but I really like how OCIO
has minimal external build dependencies. (Some day when the windows
port needs to be attacked, this should keep life simple I hope!)
Does OSX have YAML as part of its base installation? Do normal linux
distros? (I assume Windows doesnt) Is there a minimal yaml
implementation we could pull internal, similar in spirit to tinyxml?
If so (assuming it can be namespaced properly, and the licence terms
are suitable) I'd love to keep it internal.

I'm not sure that yaml's strong python support is relevant to this
discussion. OCIO serialization is handled on the C++ side, and the
python bindings provide typed interfaces to the transforms. (I.e., a
CDLTransform is a native python class as opposed to a python dict with
particular key value pairs). And, I would prefer that folks
manipulating OCIO profiles in python use the native OCIO API rather
than mucking with the serialized representations. Or is there another
use case I'm missing?

Also, in terms of overall development priorities I wouldn't personally
put this at the top of my list. (I was much more excited about your
adding Houdini lut support, for example). But I can see that making
the format less daunting could be a selling point adoption.

-- Jeremy

On Tue, Oct 5, 2010 at 6:43 PM, Malcolm Humphreys
<malcolmh...@mac.com> wrote:
Are people super into the idea of using xml?

We used xml for nearly everything at RSP, over time it became clear that it's a pretty poor choice for a lot of things.

One of the developers introduced the use of YAML (http://www.yaml.org/  http://en.wikipedia.org/wiki/YAML) and it
took off with everyone, it happened again when I recommended it's introduction at DrD (where pretty much
everything was XML). RV has started to use it for it's package descriptions as well.

As I don't think any other apps will actually be writing ocio profiles other than the ocio api, and people will be hand
writing these profiles. I don't see a benefit to using xml for a profile description, if we using as an interchange format
then maybe.

btw. I really don't like file format discussions but I think this would really help ocio profile readability. If people are
into the idea I'm happy to do the leg work to get it to work.

I have attached a small demo python script for you to check out, it mostly shows off how to define your own
custom yaml serialisation. You need PyYaml (http://pyyaml.org/) to run the demo.

Why I like it:
- super simple to parse and serialise objects
- human readable / editable
- easy to define your own custom types
- using yaml in python is so simple

It would be quite easy to have binary data (eg file luts) interweaved inside a yaml file.
http://en.wikipedia.org/wiki/YAML#Other_specified_data_types
--snip--
picture: !!binary |
 R0lGODlhDAAMAIQAAP//9/X
 17unp5WZmZgAAAOfn515eXv
 Pz7Y6OjuDg4J+fn5OTk6enp
 56enmleECcgggoBADs=mZmE
--snip--

http://www.yaml.org/spec/1.2/spec.html#id2759572 - Relation to JSON
http://www.yaml.org/spec/1.2/spec.html#id2759732 - Relation to XML

YAML
--snip--
ocs_profile_version: 1
resource_path: luts
luma_b: '0.0722'
luma_g: '0.7152'
luma_r: '0.2126'
role_compositing_log: lgh
role_scene_linear: lnh
strictparsing: true
displays:
 Film1D: !Display {colorspace: vd8, device: sRGB}
 Log: !Display {colorspace: lg10, device: sRGB}
colorspaces:
 dt8: !ColorSpace
  bitdepth: 8ui
  description: null
  family: vd8
  gpuallocation: uniform
  gpumax: 1
  gpumin: 0
  isdata: false
  to_reference:
  - !File {interpolation: null, src: diffusemult.spimtx}
  - !ColorSpaceTransform {dst: lnh, src: vd8}
 lg10: !ColorSpace
  bitdepth: 10ui
  description: lg10 converts from scanned film to the show linear reference space.
    This is a custom conversion based on several Kodak acquisition stocks. All three
    color channels have the same response and there is no cross talk. This transform
    has a shoulder and toe compensation. This 10bit uiint version is used for DI
    deliveries.
  family: lg
  gpuallocation: uniform
  gpumax: 1
  gpumin: 0
  isdata: false
  to_reference:
 lnh: !ColorSpace
  bitdepth: 16f
  description: The show reference space. This is a sensor referred linear representation
    of the scene with primaries that correspond to scanned film. 0.18 inthis space
    corresponds to a properly exposed 18% grey card.
  family: ln
  gpuallocation: lg2
  gpumax: 6
  gpumin: -15
  isdata: false
  to_reference: null
 vd8: !ColorSpace
  bitdepth: 8ui
  description: vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference
    colorspace. This is a 1-d representation does not include any subtractive color
    mixing or crosstalk compensation. This is commonly used to interchange files
    between internal and external vendors. This is used as the interchange space
    for final images on direct to video or television products.
  family: vd8
  gpuallocation: uniform
  gpumax: 1
  gpumin: 0
  isdata: false
  to_reference:
  - !File {interpolation: linear, src: vd10.spi1d}
--snip--

XML
--snip--
<ocioconfig version="1" resourcepath="luts" strictparsing="true" luma_r="0.2126" luma_g="0.7152" luma_b="0.0722" role_scene_linear="lnh" role_compositing_log="lgh">
   <display device="sRGB" name="Film1D" colorspace="vd8"/>
   <display device="sRGB" name="Log" colorspace="lg10"/>
   <colorspace name="lnh" family="ln" bitdepth="16f" isdata="false" gpuallocation="lg2" gpumin="-15" gpumax="6">
       <description>The show reference space. This is a sensor referred linear representation of the scene with primaries that correspond to scanned film. 0.18 in this space corresponds to a properly exposed 18% grey card. </description>
   </colorspace>
   <colorspace name="lg10" family="lg" bitdepth="10ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description>lg10 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 10bit uiint version is used for DI deliveries.</description>
       <to_reference>
           <group>
               <file src="lg10.spi1d" interpolation="nearest" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="lg16" family="lg" bitdepth="16ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description>lg16 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 16bit uiint version is used for DI deliveries in the rare case where this is requested. </description>
       <to_reference>
           <group>
               <file src="lgf.spi1d" interpolation="nearest" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="lgh" family="lg" bitdepth="16f" isdata="false" gpuallocation="uniform" gpumin="-0.2" gpumax="1.615">
       <description>Converts from scanned film to the show linear reference space. This is a custom curve based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This floating point version of the conversion that includes a large range of over-shoot and under-shoot values. This allows for colortiming offsetsto be applied and reversed out losslessly. For internal use only. </description>
       <to_reference>
           <group>
               <file src="lgf.spi1d" interpolation="linear" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="vd8" family="vd" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description>vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference colorspace. This is a 1-d representation does not include any subtractive color mixing or crosstalk compensation. This is commonly used to interchange files between internal and external vendors. This is used as the interchange space for final images on direct to video or television products. </description>
       <to_reference>
           <group>
               <file src="vd10.spi1d" interpolation="linear" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="dt8" family="dt" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description> </description>
       <to_reference>
           <group>
               <file src="diffusemult.spimtx"/>
               <colorspacetransform src="vd8" dst="lnh"/>
           </group>
       </to_reference>
   </colorspace>
</ocioconfig>
--snip--




.malcolm


bsloan <bsl...@...>
 

http://code.google.com/p/yaml-cpp/

...is the compiled yaml io library. I believe yaml support is built
into the python distro.



-blake

On Oct 7, 2:26 pm, Jeremy Selan <jeremy...@gmail.com> wrote:
I'm completely in favor of making the OCIO configuration files easier
to read and edit.

I have a few specific concerns about YAML, but it's definitely worth
exploring.  Care to take a stab at it?

In defense of XML ...

* It's the 'no format decision' decision. Support is ubiquitous, and
no one ever wonders, "what the hell is this?"  YAML is not nearly as
widely known. Never having looked at YAML before, when I first looked
at your configuration file I was wondering which formatting parts were
important.   But this could be made more obvious with a few tweaks to
the outputter format.

* We've already got the XML code working.

* The ASC CDL defines an xml spec which I would like to support (for
interoperability with other CDL apps),  so we're not going to be be
able to ditch tinyxml.

* This may not be wise in the long term, but I really like how OCIO
has minimal external build dependencies.  (Some day when the windows
port needs to be attacked, this should keep life simple I hope!)
Does OSX have YAML as part of its base installation? Do normal linux
distros?  (I assume Windows doesnt)  Is there a minimal yaml
implementation we could pull internal, similar in spirit to tinyxml?
If so (assuming it can be namespaced properly, and the licence terms
are suitable) I'd love to keep it internal.

I'm not sure that yaml's strong python support is relevant to this
discussion. OCIO serialization is handled on the C++ side, and the
python bindings provide typed interfaces to the transforms.  (I.e., a
CDLTransform is a native python class as opposed to a python dict with
particular key value pairs).   And, I would prefer that folks
manipulating OCIO profiles in python use the native OCIO API rather
than mucking with the serialized representations.  Or is there another
use case I'm missing?

Also, in terms of overall development priorities I wouldn't personally
put this at the top of my list. (I was much more excited about your
adding Houdini lut support, for example). But I can see that making
the format less daunting could be a selling point adoption.

-- Jeremy

On Tue, Oct 5, 2010 at 6:43 PM, Malcolm Humphreys

<malcolmh...@mac.com> wrote:
Are people super into the idea of using xml?
We used xml for nearly everything at RSP, over time it became clear that it's a pretty poor choice for a lot of things.
One of the developers introduced the use of YAML (http://www.yaml.org/ http://en.wikipedia.org/wiki/YAML) and it
took off with everyone, it happened again when I recommended it's introduction at DrD (where pretty much
everything was XML). RV has started to use it for it's package descriptions as well.
As I don't think any other apps will actually be writing ocio profiles other than the ocio api, and people will be hand
writing these profiles. I don't see a benefit to using xml for a profile description, if we using as an interchange format
then maybe.
btw. I really don't like file format discussions but I think this would really help ocio profile readability. If people are
into the idea I'm happy to do the leg work to get it to work.
I have attached a small demo python script for you to check out, it mostly shows off how to define your own
custom yaml serialisation. You need PyYaml (http://pyyaml.org/) to run the demo.
Why I like it:
- super simple to parse and serialise objects
- human readable / editable
- easy to define your own custom types
- using yaml in python is so simple
It would be quite easy to have binary data (eg file luts) interweaved inside a yaml file.
http://en.wikipedia.org/wiki/YAML#Other_specified_data_types
--snip--
picture: !!binary |
 R0lGODlhDAAMAIQAAP//9/X
 17unp5WZmZgAAAOfn515eXv
 Pz7Y6OjuDg4J+fn5OTk6enp
 56enmleECcgggoBADs=mZmE
--snip--
http://www.yaml.org/spec/1.2/spec.html#id2759572- Relation to JSON
http://www.yaml.org/spec/1.2/spec.html#id2759732- Relation to XML
YAML
--snip--
ocs_profile_version: 1
resource_path: luts
luma_b: '0.0722'
luma_g: '0.7152'
luma_r: '0.2126'
role_compositing_log: lgh
role_scene_linear: lnh
strictparsing: true
displays:
 Film1D: !Display {colorspace: vd8, device: sRGB}
 Log: !Display {colorspace: lg10, device: sRGB}
colorspaces:
 dt8: !ColorSpace
  bitdepth: 8ui
  description: null
  family: vd8
  gpuallocation: uniform
  gpumax: 1
  gpumin: 0
  isdata: false
  to_reference:
  - !File {interpolation: null, src: diffusemult.spimtx}
  - !ColorSpaceTransform {dst: lnh, src: vd8}
 lg10: !ColorSpace
  bitdepth: 10ui
  description: lg10 converts from scanned film to the show linear reference space.
    This is a custom conversion based on several Kodak acquisition stocks. All three
    color channels have the same response and there is no cross talk. This transform
    has a shoulder and toe compensation. This 10bit uiint version is used for DI
    deliveries.
  family: lg
  gpuallocation: uniform
  gpumax: 1
  gpumin: 0
  isdata: false
  to_reference:
 lnh: !ColorSpace
  bitdepth: 16f
  description: The show reference space. This is a sensor referred linear representation
    of the scene with primaries that correspond to scanned film. 0.18 inthis space
    corresponds to a properly exposed 18% grey card.
  family: ln
  gpuallocation: lg2
  gpumax: 6
  gpumin: -15
  isdata: false
  to_reference: null
 vd8: !ColorSpace
  bitdepth: 8ui
  description: vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference
    colorspace. This is a 1-d representation does not include any subtractive color
    mixing or crosstalk compensation. This is commonly used to interchange files
    between internal and external vendors. This is used as the interchange space
    for final images on direct to video or television products.
  family: vd8
  gpuallocation: uniform
  gpumax: 1
  gpumin: 0
  isdata: false
  to_reference:
  - !File {interpolation: linear, src: vd10.spi1d}
--snip--
XML
--snip--
<ocioconfig version="1" resourcepath="luts" strictparsing="true" luma_r="0.2126" luma_g="0.7152" luma_b="0.0722" role_scene_linear="lnh" role_compositing_log="lgh">
   <display device="sRGB" name="Film1D" colorspace="vd8"/>
   <display device="sRGB" name="Log" colorspace="lg10"/>
   <colorspace name="lnh" family="ln" bitdepth="16f" isdata="false" gpuallocation="lg2" gpumin="-15" gpumax="6">
       <description>The show reference space. This is a sensor referred linear representation of the scene with primaries that correspond to scanned film. 0.18 in this space corresponds to a properly exposed 18% grey card. </description>
   </colorspace>
   <colorspace name="lg10" family="lg" bitdepth="10ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description>lg10 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 10bit uiint version is used for DI deliveries.</description>
       <to_reference>
           <group>
               <file src="lg10.spi1d" interpolation="nearest" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="lg16" family="lg" bitdepth="16ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description>lg16 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 16bit uiint version is used for DI deliveries in the rare case where this is requested. </description>
       <to_reference>
           <group>
               <file src="lgf.spi1d" interpolation="nearest" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="lgh" family="lg" bitdepth="16f" isdata="false" gpuallocation="uniform" gpumin="-0.2" gpumax="1.615">
       <description>Converts from scanned film to the show linear reference space. This is a custom curve based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This floating point version of the conversion that includes a large range of over-shoot and under-shoot values. This allows for colortiming offsetsto be applied and reversed out losslessly. For internal use only. </description>
       <to_reference>
           <group>
               <file src="lgf.spi1d" interpolation="linear" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="vd8" family="vd" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description>vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference colorspace. This is a 1-d representation does not include any subtractive color mixing or crosstalk compensation. This is commonly used to interchange files between internal and external vendors. This is used as the interchange space for final images on direct to video or television products. </description>
       <to_reference>
           <group>
               <file src="vd10.spi1d" interpolation="linear" />
           </group>
       </to_reference>
   </colorspace>
   <colorspace name="dt8" family="dt" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
       <description> </description>
       <to_reference>
         
...

read more »


Malcolm Humphreys <malcolmh...@...>
 

Sorry I will be responding to this soon, bit busy for the next 3 days.

.malcolm

On 09/10/2010, at 9:35 AM, bsloan wrote:

http://code.google.com/p/yaml-cpp/

...is the compiled yaml io library. I believe yaml support is built
into the python distro.



-blake

On Oct 7, 2:26 pm, Jeremy Selan <jeremy...@gmail.com> wrote:
I'm completely in favor of making the OCIO configuration files easier
to read and edit.

I have a few specific concerns about YAML, but it's definitely worth
exploring. Care to take a stab at it?

In defense of XML ...

* It's the 'no format decision' decision. Support is ubiquitous, and
no one ever wonders, "what the hell is this?" YAML is not nearly as
widely known. Never having looked at YAML before, when I first looked
at your configuration file I was wondering which formatting parts were
important. But this could be made more obvious with a few tweaks to
the outputter format.

* We've already got the XML code working.

* The ASC CDL defines an xml spec which I would like to support (for
interoperability with other CDL apps), so we're not going to be be
able to ditch tinyxml.

* This may not be wise in the long term, but I really like how OCIO
has minimal external build dependencies. (Some day when the windows
port needs to be attacked, this should keep life simple I hope!)
Does OSX have YAML as part of its base installation? Do normal linux
distros? (I assume Windows doesnt) Is there a minimal yaml
implementation we could pull internal, similar in spirit to tinyxml?
If so (assuming it can be namespaced properly, and the licence terms
are suitable) I'd love to keep it internal.

I'm not sure that yaml's strong python support is relevant to this
discussion. OCIO serialization is handled on the C++ side, and the
python bindings provide typed interfaces to the transforms. (I.e., a
CDLTransform is a native python class as opposed to a python dict with
particular key value pairs). And, I would prefer that folks
manipulating OCIO profiles in python use the native OCIO API rather
than mucking with the serialized representations. Or is there another
use case I'm missing?

Also, in terms of overall development priorities I wouldn't personally
put this at the top of my list. (I was much more excited about your
adding Houdini lut support, for example). But I can see that making
the format less daunting could be a selling point adoption.

-- Jeremy

On Tue, Oct 5, 2010 at 6:43 PM, Malcolm Humphreys

<malcolmh...@mac.com> wrote:
Are people super into the idea of using xml?
We used xml for nearly everything at RSP, over time it became clear that it's a pretty poor choice for a lot of things.
One of the developers introduced the use of YAML (http://www.yaml.org/ http://en.wikipedia.org/wiki/YAML) and it
took off with everyone, it happened again when I recommended it's introduction at DrD (where pretty much
everything was XML). RV has started to use it for it's package descriptions as well.
As I don't think any other apps will actually be writing ocio profiles other than the ocio api, and people will be hand
writing these profiles. I don't see a benefit to using xml for a profile description, if we using as an interchange format
then maybe.
btw. I really don't like file format discussions but I think this would really help ocio profile readability. If people are
into the idea I'm happy to do the leg work to get it to work.
I have attached a small demo python script for you to check out, it mostly shows off how to define your own
custom yaml serialisation. You need PyYaml (http://pyyaml.org/) to run the demo.
Why I like it:
- super simple to parse and serialise objects
- human readable / editable
- easy to define your own custom types
- using yaml in python is so simple
It would be quite easy to have binary data (eg file luts) interweaved inside a yaml file.
http://en.wikipedia.org/wiki/YAML#Other_specified_data_types
--snip--
picture: !!binary |
R0lGODlhDAAMAIQAAP//9/X
17unp5WZmZgAAAOfn515eXv
Pz7Y6OjuDg4J+fn5OTk6enp
56enmleECcgggoBADs=mZmE
--snip--
http://www.yaml.org/spec/1.2/spec.html#id2759572- Relation to JSON
http://www.yaml.org/spec/1.2/spec.html#id2759732- Relation to XML
YAML
--snip--
ocs_profile_version: 1
resource_path: luts
luma_b: '0.0722'
luma_g: '0.7152'
luma_r: '0.2126'
role_compositing_log: lgh
role_scene_linear: lnh
strictparsing: true
displays:
Film1D: !Display {colorspace: vd8, device: sRGB}
Log: !Display {colorspace: lg10, device: sRGB}
colorspaces:
dt8: !ColorSpace
bitdepth: 8ui
description: null
family: vd8
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
- !File {interpolation: null, src: diffusemult.spimtx}
- !ColorSpaceTransform {dst: lnh, src: vd8}
lg10: !ColorSpace
bitdepth: 10ui
description: lg10 converts from scanned film to the show linear reference space.
This is a custom conversion based on several Kodak acquisition stocks. All three
color channels have the same response and there is no cross talk. This transform
has a shoulder and toe compensation. This 10bit uiint version is used for DI
deliveries.
family: lg
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
lnh: !ColorSpace
bitdepth: 16f
description: The show reference space. This is a sensor referred linear representation
of the scene with primaries that correspond to scanned film. 0.18 inthis space
corresponds to a properly exposed 18% grey card.
family: ln
gpuallocation: lg2
gpumax: 6
gpumin: -15
isdata: false
to_reference: null
vd8: !ColorSpace
bitdepth: 8ui
description: vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference
colorspace. This is a 1-d representation does not include any subtractive color
mixing or crosstalk compensation. This is commonly used to interchange files
between internal and external vendors. This is used as the interchange space
for final images on direct to video or television products.
family: vd8
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
- !File {interpolation: linear, src: vd10.spi1d}
--snip--
XML
--snip--
<ocioconfig version="1" resourcepath="luts" strictparsing="true" luma_r="0.2126" luma_g="0.7152" luma_b="0.0722" role_scene_linear="lnh" role_compositing_log="lgh">
<display device="sRGB" name="Film1D" colorspace="vd8"/>
<display device="sRGB" name="Log" colorspace="lg10"/>
<colorspace name="lnh" family="ln" bitdepth="16f" isdata="false" gpuallocation="lg2" gpumin="-15" gpumax="6">
<description>The show reference space. This is a sensor referred linear representation of the scene with primaries that correspond to scanned film. 0.18 in this space corresponds to a properly exposed 18% grey card. </description>
</colorspace>
<colorspace name="lg10" family="lg" bitdepth="10ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>lg10 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 10bit uiint version is used for DI deliveries.</description>
<to_reference>
<group>
<file src="lg10.spi1d" interpolation="nearest" />
</group>
</to_reference>
</colorspace>
<colorspace name="lg16" family="lg" bitdepth="16ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>lg16 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 16bit uiint version is used for DI deliveries in the rare case where this is requested. </description>
<to_reference>
<group>
<file src="lgf.spi1d" interpolation="nearest" />
</group>
</to_reference>
</colorspace>
<colorspace name="lgh" family="lg" bitdepth="16f" isdata="false" gpuallocation="uniform" gpumin="-0.2" gpumax="1.615">
<description>Converts from scanned film to the show linear reference space. This is a custom curve based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This floating point version of the conversion that includes a large range of over-shoot and under-shoot values. This allows for colortiming offsetsto be applied and reversed out losslessly. For internal use only. </description>
<to_reference>
<group>
<file src="lgf.spi1d" interpolation="linear" />
</group>
</to_reference>
</colorspace>
<colorspace name="vd8" family="vd" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference colorspace. This is a 1-d representation does not include any subtractive color mixing or crosstalk compensation. This is commonly used to interchange files between internal and external vendors. This is used as the interchange space for final images on direct to video or television products. </description>
<to_reference>
<group>
<file src="vd10.spi1d" interpolation="linear" />
</group>
</to_reference>
</colorspace>
<colorspace name="dt8" family="dt" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description> </description>
<to_reference>
...

read more »


Malcolm Humphreys <malcolmh...@...>
 

I'm completely in favor of making the OCIO configuration files easier
to read and edit.

I have a few specific concerns about YAML, but it's definitely worth
exploring. Care to take a stab at it?
Ok lets jump in :)

In defense of XML ...

* It's the 'no format decision' decision. Support is ubiquitous, and
no one ever wonders, "what the hell is this?" YAML is not nearly as
widely known. Never having looked at YAML before, when I first looked
at your configuration file I was wondering which formatting parts were
important. But this could be made more obvious with a few tweaks to
the outputter format.
There are a few good articles on the subject of choosing XML 'just because', these two I like.

http://www.ibm.com/developerworks/library/x-sbxml.html
http://www.ibm.com/developerworks/xml/library/x-matters23.html

I think XML is a great fit for a lot of things and I would still choose it for these but it's used a
little too much 'just because'.

* We've already got the XML code working.
True, but it feels a bit ridged and requires a lot of code to do relatively simple serialisation. I find
this is inherently part of using xml, and is nothing different from anywhere else I have used it or
seen it used.

Looking at it, the current serialisation boils down to dumping Config::Impl to disk
http://github.com/imageworks/OpenColorIO/blob/master/src/core/Config.cpp (line: 136)

Config::Impl consists of:
- a few std::string's
- RGB float array
- prefixed list of roles
- unordered vector of display devices.
- unordered vector of colorspaces

Config::writeXML (line: 761) and Config::Impl::loadXmlElement (line: 836) do most of the heavy
lifting in terms of overall XML structure.
(Looking at this now there is a bit too much knowledge about <display>'s which we might want
to refactor into a clearer core concept.)

ParseUtils.cpp does most of the string -> ocio data type glue while XmlIO.cpp takes care of all
the logic and structure for every Transform() type.

See below I have attached a yaml-cpp demo for doing the same thing.

* The ASC CDL defines an xml spec which I would like to support (for
interoperability with other CDL apps), so we're not going to be be
able to ditch tinyxml.
I don't think this should change.

But If we were going to have a way to support OCIO Transform() plugins (at compile time or run
time). I think it would be important for each Transform() to know how to seialize themselves.

This means tinyxml would become only a build dependency for the CDL Transform() plugin and
not the entire core.

* This may not be wise in the long term, but I really like how OCIO
has minimal external build dependencies. (Some day when the windows
port needs to be attacked, this should keep life simple I hope!)
Does OSX have YAML as part of its base installation? Do normal linux
distros? (I assume Windows doesnt) Is there a minimal yaml
implementation we could pull internal, similar in spirit to tinyxml?
If so (assuming it can be namespaced properly, and the licence terms
are suitable) I'd love to keep it internal.
Agreed, this should all be internal to OCIO.

Both http://code.google.com/p/yaml-cpp and http://pyyaml.org/wiki/LibYAML are MIT Licensed
(http://en.wikipedia.org/wiki/MIT_License). It's pretty open licence you just need to include a
named reference and MIT licence with OCIO.

I have never used the LibYAML but I do like the stream / state based style yaml-cpp has
chosen. http://code.google.com/p/yaml-cpp/wiki/HowToEmitYAML

yaml-cpp is namespaced with YAML:: and I consider it to be pretty small lib.

--snip stlemitter.h--
namespace YAML
{
template <typename T>
inline Emitter& operator << (Emitter& emitter, const std::vector <T>& v) {
typedef typename std::vector <T> vec;
emitter << BeginSeq;
for(typename vec::const_iterator it=v.begin();it!=v.end();++it)
emitter << *it;
emitter << EndSeq;
return emitter;
}
...
--snip stlemitter.h--

I'm not sure that yaml's strong python support is relevant to this
discussion. OCIO serialization is handled on the C++ side, and the
python bindings provide typed interfaces to the transforms. (I.e., a
CDLTransform is a native python class as opposed to a python dict with
particular key value pairs). And, I would prefer that folks
manipulating OCIO profiles in python use the native OCIO API rather
than mucking with the serialized representations. Or is there another
use case I'm missing?
Sure its just an easy way to prototype the yaml, like you said this should all be internal to OCIO.
I don't think you will be able to stop people from mucking with the XML directly in python either.

Attached is a quick yaml-cpp demo which setups a similar structure to configs/testing/config.ocio
then serializes it to YAML, parses that YAML into a new config and checks some of the
data.


Also, in terms of overall development priorities I wouldn't personally
put this at the top of my list. (I was much more excited about your
adding Houdini lut support, for example). But I can see that making
the format less daunting could be a selling point adoption.
I see the profiles as the key concept that makes OCIO interesting. All the other parts are important
on the implementation side of things, but the day to day interaction will be all about the profiles.

As long as the profiles are clear and easily extendable I think we will be in a good place.

.malcolm

-- Jeremy

On Tue, Oct 5, 2010 at 6:43 PM, Malcolm Humphreys
<malcolmh...@mac.com> wrote:
Are people super into the idea of using xml?

We used xml for nearly everything at RSP, over time it became clear that it's a pretty poor choice for a lot of things.

One of the developers introduced the use of YAML (http://www.yaml.org/ http://en.wikipedia.org/wiki/YAML) and it
took off with everyone, it happened again when I recommended it's introduction at DrD (where pretty much
everything was XML). RV has started to use it for it's package descriptions as well.

As I don't think any other apps will actually be writing ocio profiles other than the ocio api, and people will be hand
writing these profiles. I don't see a benefit to using xml for a profile description, if we using as an interchange format
then maybe.

btw. I really don't like file format discussions but I think this would really help ocio profile readability. If people are
into the idea I'm happy to do the leg work to get it to work.

I have attached a small demo python script for you to check out, it mostly shows off how to define your own
custom yaml serialisation. You need PyYaml (http://pyyaml.org/) to run the demo.

Why I like it:
- super simple to parse and serialise objects
- human readable / editable
- easy to define your own custom types
- using yaml in python is so simple

It would be quite easy to have binary data (eg file luts) interweaved inside a yaml file.
http://en.wikipedia.org/wiki/YAML#Other_specified_data_types
--snip--
picture: !!binary |
R0lGODlhDAAMAIQAAP//9/X
17unp5WZmZgAAAOfn515eXv
Pz7Y6OjuDg4J+fn5OTk6enp
56enmleECcgggoBADs=mZmE
--snip--

http://www.yaml.org/spec/1.2/spec.html#id2759572 - Relation to JSON
http://www.yaml.org/spec/1.2/spec.html#id2759732 - Relation to XML

YAML
--snip--
ocs_profile_version: 1
resource_path: luts
luma_b: '0.0722'
luma_g: '0.7152'
luma_r: '0.2126'
role_compositing_log: lgh
role_scene_linear: lnh
strictparsing: true
displays:
Film1D: !Display {colorspace: vd8, device: sRGB}
Log: !Display {colorspace: lg10, device: sRGB}
colorspaces:
dt8: !ColorSpace
bitdepth: 8ui
description: null
family: vd8
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
- !File {interpolation: null, src: diffusemult.spimtx}
- !ColorSpaceTransform {dst: lnh, src: vd8}
lg10: !ColorSpace
bitdepth: 10ui
description: lg10 converts from scanned film to the show linear reference space.
This is a custom conversion based on several Kodak acquisition stocks. All three
color channels have the same response and there is no cross talk. This transform
has a shoulder and toe compensation. This 10bit uiint version is used for DI
deliveries.
family: lg
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
lnh: !ColorSpace
bitdepth: 16f
description: The show reference space. This is a sensor referred linear representation
of the scene with primaries that correspond to scanned film. 0.18 inthis space
corresponds to a properly exposed 18% grey card.
family: ln
gpuallocation: lg2
gpumax: 6
gpumin: -15
isdata: false
to_reference: null
vd8: !ColorSpace
bitdepth: 8ui
description: vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference
colorspace. This is a 1-d representation does not include any subtractive color
mixing or crosstalk compensation. This is commonly used to interchange files
between internal and external vendors. This is used as the interchange space
for final images on direct to video or television products.
family: vd8
gpuallocation: uniform
gpumax: 1
gpumin: 0
isdata: false
to_reference:
- !File {interpolation: linear, src: vd10.spi1d}
--snip--

XML
--snip--
<ocioconfig version="1" resourcepath="luts" strictparsing="true" luma_r="0.2126" luma_g="0.7152" luma_b="0.0722" role_scene_linear="lnh" role_compositing_log="lgh">
<display device="sRGB" name="Film1D" colorspace="vd8"/>
<display device="sRGB" name="Log" colorspace="lg10"/>
<colorspace name="lnh" family="ln" bitdepth="16f" isdata="false" gpuallocation="lg2" gpumin="-15" gpumax="6">
<description>The show reference space. This is a sensor referred linear representation of the scene with primaries that correspond to scanned film. 0.18 in this space corresponds to a properly exposed 18% grey card. </description>
</colorspace>
<colorspace name="lg10" family="lg" bitdepth="10ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>lg10 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 10bit uiint version is used for DI deliveries.</description>
<to_reference>
<group>
<file src="lg10.spi1d" interpolation="nearest" />
</group>
</to_reference>
</colorspace>
<colorspace name="lg16" family="lg" bitdepth="16ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>lg16 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 16bit uiint version is used for DI deliveries in the rare case where this is requested. </description>
<to_reference>
<group>
<file src="lgf.spi1d" interpolation="nearest" />
</group>
</to_reference>
</colorspace>
<colorspace name="lgh" family="lg" bitdepth="16f" isdata="false" gpuallocation="uniform" gpumin="-0.2" gpumax="1.615">
<description>Converts from scanned film to the show linear reference space. This is a custom curve based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This floating point version of the conversion that includes a large range of over-shoot and under-shoot values. This allows for colortiming offsetsto be applied and reversed out losslessly. For internal use only. </description>
<to_reference>
<group>
<file src="lgf.spi1d" interpolation="linear" />
</group>
</to_reference>
</colorspace>
<colorspace name="vd8" family="vd" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description>vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference colorspace. This is a 1-d representation does not include any subtractive color mixing or crosstalk compensation. This is commonly used to interchange files between internal and external vendors. This is used as the interchange space for final images on direct to video or television products. </description>
<to_reference>
<group>
<file src="vd10.spi1d" interpolation="linear" />
</group>
</to_reference>
</colorspace>
<colorspace name="dt8" family="dt" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1">
<description> </description>
<to_reference>
<group>
<file src="diffusemult.spimtx"/>
<colorspacetransform src="vd8" dst="lnh"/>
</group>
</to_reference>
</colorspace>
</ocioconfig>
--snip--




.malcolm


Jeremy Selan <jeremy...@...>
 

Ok, I'm convinced. Let's give YAML a shot.

In terms of broad technical strokes, I think either YAML or XML would
be sufficient. But given your enthusiasm for YAML, and it's superior
human readability, there's no argument against it. (Other than the
work / support involved).

A few additional comments as we get going:

* This will break the API, so before I roll the commit(s) into the
master branch we'll tag off a stable 0.6 branch. Feel free to do
anything you want in your local git branch, of course.

* As we introduce OCIO "dynamic" Transform plugins, my hope is to not
burden them with the task of serialization. But let's not worry about
this for now.

* I agree that Yaml-cpp seems like our best option of the ones you've
mentioned. Let's go with it. Is is feasible to wrap it into the
OCIO namespace? see src/core/tinyxml/tinyxml.h as an example It's
got a lot of source files, but I really would prefer if possible to
keep all OCIO symbols inside a versioned namespace.

* For the OCIO:Config implementation, the list of display devices and
color spaces actually do preserve order. While the ordering is not
relevant to color processing, it is useful in a UI context.

* Keeping the serialization code internally as statics - not on the
Transforms themselves - is useful so we dont need to expose the
serialization to the public API. Or do you have a better
implementation in mind?

As soon as you have even a stub implementation -- it doesnt need to
build -- I'd love to see it!

Thanks for all your work on this issue,

Jeremy


Malcolm Humphreys <malcolmh...@...>
 

Ok, I'm convinced. Let's give YAML a shot.
Ok, so there is a initial go at this in the other mail.

In terms of broad technical strokes, I think either YAML or XML would
be sufficient. But given your enthusiasm for YAML, and it's superior
human readability, there's no argument against it. (Other than the
work / support involved).

A few additional comments as we get going:
* I agree that Yaml-cpp seems like our best option of the ones you've
mentioned. Let's go with it. Is is feasible to wrap it into the
OCIO namespace? see src/core/tinyxml/tinyxml.h as an example It's
got a lot of source files, but I really would prefer if possible to
keep all OCIO symbols inside a versioned namespace.
I have added a local copy of yaml-cpp and a patch to make it build static.
Once we get a little further down the road, we can work with the yaml-cpp
project to get a customisable namespace for our purpose.

* For the OCIO:Config implementation, the list of display devices and
color spaces actually do preserve order. While the ordering is not
relevant to color processing, it is useful in a UI context.
So the yaml is slightly different to the demo's I sent through before displays
and colorspace are now YAML sequences so order is preserved. (there is
a !!omap tag for doing ordered maps but a sequence matches nicely to the
current OCIO structure http://www.yaml.org/spec/1.2/spec.html#id2761292)

eg.
displays:
- !<Display> ...
colorspaces:
- !<ColorSpace> ...

Roles are a map but could be a sequence also depending on preference.
roles:
default: !<Role> {colorspace: raw}

Both roles and displays get emitted in the Flow style (all on one line) but
could be emitted in the Block style if thats more legible.

Note.
-- this --
displays:
- !<Display> {device: sRGB, name: Raw, colorspace: raw}
-- and this --
displays:
- !<Display>
device: sRGB
name: Raw
colorspace: raw
--
Will parse the same, but it's nice to choose a style that makes sense for
emission.

Another thing which might be nice is having the following 'detail:' and
'gpu:' fields to make the profile not so vertically challenged.
--snip--
ocs_profile_version: 1
resource_path:
strictparsing: false
luma: [ 0.2126, 0.7152, 0.0722 ]
roles:
default: !<Role> {colorspace: raw}
displays:
- !<Display> {device: sRGB, name: Raw, colorspace: raw}
colorspaces:
- !<ColorSpace>
name: raw
detail: { family: raw, bitdepth: 32f, isdata: true }
gpu: { gpuallocation: uniform, gpumin: 0, gpumax: 1 }
description: |
A raw color space. Conversions to and from this space
are no-ops.
--snip--

* Keeping the serialization code internally as statics - not on the
Transforms themselves - is useful so we dont need to expose the
serialization to the public API. Or do you have a better
implementation in mind?
I have some ideas but I thought it would be better to just do a swap
out replacement first, then tackle plug-able Transform() serialization later.

Thanks for all your work on this issue,
No problem.

.malcolm