Date
1 - 7 of 7
XML Profile Format
|
Malcolm Humphreys <malcolmh...@...>
Are people super into the idea of using xml?
We used xml for nearly everything at RSP, over time it became clear that it's a pretty poor choice for a lot of things. One of the developers introduced the use of YAML (http://www.yaml.org/ http://en.wikipedia.org/wiki/YAML) and it took off with everyone, it happened again when I recommended it's introduction at DrD (where pretty much everything was XML). RV has started to use it for it's package descriptions as well. As I don't think any other apps will actually be writing ocio profiles other than the ocio api, and people will be hand writing these profiles. I don't see a benefit to using xml for a profile description, if we using as an interchange format then maybe. btw. I really don't like file format discussions but I think this would really help ocio profile readability. If people are into the idea I'm happy to do the leg work to get it to work. I have attached a small demo python script for you to check out, it mostly shows off how to define your own custom yaml serialisation. You need PyYaml (http://pyyaml.org/) to run the demo. Why I like it: - super simple to parse and serialise objects - human readable / editable - easy to define your own custom types - using yaml in python is so simple It would be quite easy to have binary data (eg file luts) interweaved inside a yaml file. http://en.wikipedia.org/wiki/YAML#Other_specified_data_types --snip-- picture: !!binary | R0lGODlhDAAMAIQAAP//9/X 17unp5WZmZgAAAOfn515eXv Pz7Y6OjuDg4J+fn5OTk6enp 56enmleECcgggoBADs=mZmE --snip-- http://www.yaml.org/spec/1.2/spec.html#id2759572 - Relation to JSON http://www.yaml.org/spec/1.2/spec.html#id2759732 - Relation to XML YAML --snip-- ocs_profile_version: 1 resource_path: luts luma_b: '0.0722' luma_g: '0.7152' luma_r: '0.2126' role_compositing_log: lgh role_scene_linear: lnh strictparsing: true displays: Film1D: !Display {colorspace: vd8, device: sRGB} Log: !Display {colorspace: lg10, device: sRGB} colorspaces: dt8: !ColorSpace bitdepth: 8ui description: null family: vd8 gpuallocation: uniform gpumax: 1 gpumin: 0 isdata: false to_reference: - !File {interpolation: null, src: diffusemult.spimtx} - !ColorSpaceTransform {dst: lnh, src: vd8} lg10: !ColorSpace bitdepth: 10ui description: lg10 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 10bit uiint version is used for DI deliveries. family: lg gpuallocation: uniform gpumax: 1 gpumin: 0 isdata: false to_reference: lnh: !ColorSpace bitdepth: 16f description: The show reference space. This is a sensor referred linear representation of the scene with primaries that correspond to scanned film. 0.18 inthis space corresponds to a properly exposed 18% grey card. family: ln gpuallocation: lg2 gpumax: 6 gpumin: -15 isdata: false to_reference: null vd8: !ColorSpace bitdepth: 8ui description: vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference colorspace. This is a 1-d representation does not include any subtractive color mixing or crosstalk compensation. This is commonly used to interchange files between internal and external vendors. This is used as the interchange space for final images on direct to video or television products. family: vd8 gpuallocation: uniform gpumax: 1 gpumin: 0 isdata: false to_reference: - !File {interpolation: linear, src: vd10.spi1d} --snip-- XML --snip-- <ocioconfig version="1" resourcepath="luts" strictparsing="true" luma_r="0.2126" luma_g="0.7152" luma_b="0.0722" role_scene_linear="lnh" role_compositing_log="lgh"> <display device="sRGB" name="Film1D" colorspace="vd8"/> <display device="sRGB" name="Log" colorspace="lg10"/> <colorspace name="lnh" family="ln" bitdepth="16f" isdata="false" gpuallocation="lg2" gpumin="-15" gpumax="6"> <description>The show reference space. This is a sensor referred linear representation of the scene with primaries that correspond to scanned film. 0.18 in this space corresponds to a properly exposed 18% grey card. </description> </colorspace> <colorspace name="lg10" family="lg" bitdepth="10ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1"> <description>lg10 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 10bit uiint version is used for DI deliveries.</description> <to_reference> <group> <file src="lg10.spi1d" interpolation="nearest" /> </group> </to_reference> </colorspace> <colorspace name="lg16" family="lg" bitdepth="16ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1"> <description>lg16 converts from scanned film to the show linear reference space. This is a custom conversion based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This 16bit uiint version is used for DI deliveries in the rare case where this is requested. </description> <to_reference> <group> <file src="lgf.spi1d" interpolation="nearest" /> </group> </to_reference> </colorspace> <colorspace name="lgh" family="lg" bitdepth="16f" isdata="false" gpuallocation="uniform" gpumin="-0.2" gpumax="1.615"> <description>Converts from scanned film to the show linear reference space. This is a custom curve based on several Kodak acquisition stocks. All three color channels have the same response and there is no cross talk. This transform has a shoulder and toe compensation. This floating point version of the conversion that includes a large range of over-shoot and under-shoot values. This allows for colortiming offsetsto be applied and reversed out losslessly. For internal use only. </description> <to_reference> <group> <file src="lgf.spi1d" interpolation="linear" /> </group> </to_reference> </colorspace> <colorspace name="vd8" family="vd" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1"> <description>vd8 is an 8 bit sRGB like,2.2 gamma, conversion to and from the reference colorspace. This is a 1-d representation does not include any subtractive color mixing or crosstalk compensation. This is commonly used to interchange files between internal and external vendors. This is used as the interchange space for final images on direct to video or television products. </description> <to_reference> <group> <file src="vd10.spi1d" interpolation="linear" /> </group> </to_reference> </colorspace> <colorspace name="dt8" family="dt" bitdepth="8ui" isdata="false" gpuallocation="uniform" gpumin="0" gpumax="1"> <description> </description> <to_reference> <group> <file src="diffusemult.spimtx"/> <colorspacetransform src="vd8" dst="lnh"/> </group> </to_reference> </colorspace> </ocioconfig> --snip-- .malcolm
|
|
|
|
Jeremy Selan <jeremy...@...>
I'm completely in favor of making the OCIO configuration files easier
to read and edit. I have a few specific concerns about YAML, but it's definitely worth exploring. Care to take a stab at it? In defense of XML ... * It's the 'no format decision' decision. Support is ubiquitous, and no one ever wonders, "what the hell is this?" YAML is not nearly as widely known. Never having looked at YAML before, when I first looked at your configuration file I was wondering which formatting parts were important. But this could be made more obvious with a few tweaks to the outputter format. * We've already got the XML code working. * The ASC CDL defines an xml spec which I would like to support (for interoperability with other CDL apps), so we're not going to be be able to ditch tinyxml. * This may not be wise in the long term, but I really like how OCIO has minimal external build dependencies. (Some day when the windows port needs to be attacked, this should keep life simple I hope!) Does OSX have YAML as part of its base installation? Do normal linux distros? (I assume Windows doesnt) Is there a minimal yaml implementation we could pull internal, similar in spirit to tinyxml? If so (assuming it can be namespaced properly, and the licence terms are suitable) I'd love to keep it internal. I'm not sure that yaml's strong python support is relevant to this discussion. OCIO serialization is handled on the C++ side, and the python bindings provide typed interfaces to the transforms. (I.e., a CDLTransform is a native python class as opposed to a python dict with particular key value pairs). And, I would prefer that folks manipulating OCIO profiles in python use the native OCIO API rather than mucking with the serialized representations. Or is there another use case I'm missing? Also, in terms of overall development priorities I wouldn't personally put this at the top of my list. (I was much more excited about your adding Houdini lut support, for example). But I can see that making the format less daunting could be a selling point adoption. -- Jeremy On Tue, Oct 5, 2010 at 6:43 PM, Malcolm Humphreys <malcolmh...@...> wrote: Are people super into the idea of using xml?
|
|
|
|
bsloan <bsl...@...>
http://code.google.com/p/yaml-cpp/
toggle quoted messageShow quoted text
...is the compiled yaml io library. I believe yaml support is built into the python distro. -blake
On Oct 7, 2:26 pm, Jeremy Selan <jeremy...@...> wrote:
I'm completely in favor of making the OCIO configuration files easier
|
|
|
|
Malcolm Humphreys <malcolmh...@...>
Sorry I will be responding to this soon, bit busy for the next 3 days.
toggle quoted messageShow quoted text
.malcolm
On 09/10/2010, at 9:35 AM, bsloan wrote:
http://code.google.com/p/yaml-cpp/
|
|
|
|
Malcolm Humphreys <malcolmh...@...>
I'm completely in favor of making the OCIO configuration files easierOk lets jump in :) In defense of XML ...There are a few good articles on the subject of choosing XML 'just because', these two I like. http://www.ibm.com/developerworks/library/x-sbxml.html http://www.ibm.com/developerworks/xml/library/x-matters23.html I think XML is a great fit for a lot of things and I would still choose it for these but it's used a little too much 'just because'. * We've already got the XML code working.True, but it feels a bit ridged and requires a lot of code to do relatively simple serialisation. I find this is inherently part of using xml, and is nothing different from anywhere else I have used it or seen it used. Looking at it, the current serialisation boils down to dumping Config::Impl to disk http://github.com/imageworks/OpenColorIO/blob/master/src/core/Config.cpp (line: 136) Config::Impl consists of: - a few std::string's - RGB float array - prefixed list of roles - unordered vector of display devices. - unordered vector of colorspaces Config::writeXML (line: 761) and Config::Impl::loadXmlElement (line: 836) do most of the heavy lifting in terms of overall XML structure. (Looking at this now there is a bit too much knowledge about <display>'s which we might want to refactor into a clearer core concept.) ParseUtils.cpp does most of the string -> ocio data type glue while XmlIO.cpp takes care of all the logic and structure for every Transform() type. See below I have attached a yaml-cpp demo for doing the same thing. * The ASC CDL defines an xml spec which I would like to support (forI don't think this should change. But If we were going to have a way to support OCIO Transform() plugins (at compile time or run time). I think it would be important for each Transform() to know how to seialize themselves. This means tinyxml would become only a build dependency for the CDL Transform() plugin and not the entire core. * This may not be wise in the long term, but I really like how OCIOAgreed, this should all be internal to OCIO. Both http://code.google.com/p/yaml-cpp and http://pyyaml.org/wiki/LibYAML are MIT Licensed (http://en.wikipedia.org/wiki/MIT_License). It's pretty open licence you just need to include a named reference and MIT licence with OCIO. I have never used the LibYAML but I do like the stream / state based style yaml-cpp has chosen. http://code.google.com/p/yaml-cpp/wiki/HowToEmitYAML yaml-cpp is namespaced with YAML:: and I consider it to be pretty small lib. --snip stlemitter.h-- namespace YAML { template <typename T> inline Emitter& operator << (Emitter& emitter, const std::vector <T>& v) { typedef typename std::vector <T> vec; emitter << BeginSeq; for(typename vec::const_iterator it=v.begin();it!=v.end();++it) emitter << *it; emitter << EndSeq; return emitter; } ... --snip stlemitter.h-- I'm not sure that yaml's strong python support is relevant to thisSure its just an easy way to prototype the yaml, like you said this should all be internal to OCIO. I don't think you will be able to stop people from mucking with the XML directly in python either. Attached is a quick yaml-cpp demo which setups a similar structure to configs/testing/config.ocio then serializes it to YAML, parses that YAML into a new config and checks some of the data. Also, in terms of overall development priorities I wouldn't personallyI see the profiles as the key concept that makes OCIO interesting. All the other parts are important on the implementation side of things, but the day to day interaction will be all about the profiles. As long as the profiles are clear and easily extendable I think we will be in a good place. .malcolm -- Jeremy
|
|
|
|
Jeremy Selan <jeremy...@...>
Ok, I'm convinced. Let's give YAML a shot.
In terms of broad technical strokes, I think either YAML or XML would be sufficient. But given your enthusiasm for YAML, and it's superior human readability, there's no argument against it. (Other than the work / support involved). A few additional comments as we get going: * This will break the API, so before I roll the commit(s) into the master branch we'll tag off a stable 0.6 branch. Feel free to do anything you want in your local git branch, of course. * As we introduce OCIO "dynamic" Transform plugins, my hope is to not burden them with the task of serialization. But let's not worry about this for now. * I agree that Yaml-cpp seems like our best option of the ones you've mentioned. Let's go with it. Is is feasible to wrap it into the OCIO namespace? see src/core/tinyxml/tinyxml.h as an example It's got a lot of source files, but I really would prefer if possible to keep all OCIO symbols inside a versioned namespace. * For the OCIO:Config implementation, the list of display devices and color spaces actually do preserve order. While the ordering is not relevant to color processing, it is useful in a UI context. * Keeping the serialization code internally as statics - not on the Transforms themselves - is useful so we dont need to expose the serialization to the public API. Or do you have a better implementation in mind? As soon as you have even a stub implementation -- it doesnt need to build -- I'd love to see it! Thanks for all your work on this issue, Jeremy
|
|
|
|
Malcolm Humphreys <malcolmh...@...>
Ok, I'm convinced. Let's give YAML a shot.Ok, so there is a initial go at this in the other mail. In terms of broad technical strokes, I think either YAML or XML would * I agree that Yaml-cpp seems like our best option of the ones you'veI have added a local copy of yaml-cpp and a patch to make it build static. Once we get a little further down the road, we can work with the yaml-cpp project to get a customisable namespace for our purpose. * For the OCIO:Config implementation, the list of display devices andSo the yaml is slightly different to the demo's I sent through before displays and colorspace are now YAML sequences so order is preserved. (there is a !!omap tag for doing ordered maps but a sequence matches nicely to the current OCIO structure http://www.yaml.org/spec/1.2/spec.html#id2761292) eg. displays: - !<Display> ... colorspaces: - !<ColorSpace> ... Roles are a map but could be a sequence also depending on preference. roles: default: !<Role> {colorspace: raw} Both roles and displays get emitted in the Flow style (all on one line) but could be emitted in the Block style if thats more legible. Note. -- this -- displays: - !<Display> {device: sRGB, name: Raw, colorspace: raw} -- and this -- displays: - !<Display> device: sRGB name: Raw colorspace: raw -- Will parse the same, but it's nice to choose a style that makes sense for emission. Another thing which might be nice is having the following 'detail:' and 'gpu:' fields to make the profile not so vertically challenged. --snip-- ocs_profile_version: 1 resource_path: strictparsing: false luma: [ 0.2126, 0.7152, 0.0722 ] roles: default: !<Role> {colorspace: raw} displays: - !<Display> {device: sRGB, name: Raw, colorspace: raw} colorspaces: - !<ColorSpace> name: raw detail: { family: raw, bitdepth: 32f, isdata: true } gpu: { gpuallocation: uniform, gpumin: 0, gpumax: 1 } description: | A raw color space. Conversions to and from this space are no-ops. --snip-- * Keeping the serialization code internally as statics - not on theI have some ideas but I thought it would be better to just do a swap out replacement first, then tackle plug-able Transform() serialization later. Thanks for all your work on this issue,No problem. .malcolm
|
|
|