1. Osl-Dev
  2. Topics

OCIOv2 with OSL native support

Patrick Hodoul <patrick.hodoul@...>
 
Edited

Hi,

My name is Patrick Hodoul, Principal Software Developer at Autodesk, dev lead for the OCIOv2 initiative and member of the OCIO TSC (https://github.com/AcademySoftwareFoundation/OpenColorIO)

I currently investigate/prototype the native OSL generation support in OCIO. But I have very little knowledge about OSL. So, I will start with very basic questions to help me starting in the right direction:
  • For the language specifications I use the default link from the OSL home page (which is for the master branch). Is that ok to start from master? or should I switch to the latest release?
  • As I'm starting I would like to have some preliminary validations. When generating an OSL program, I currently use oslc command line tool to validate it (e.g. ./oslc -v -Werror cm1.osl). Is it enough? Is it the right approach?
  • While adding the OSL support, I did not find any support for a vector of 4 (floats, ints, etc.). I only found "vector" (i.e. three values). Is there a kind of vec4 support?
  • Currently the color transformation (code generated at runtime to perform the color transform from one color space to another one, which applies the processing pixel per pixel) generates a "generic shader" -> "shader OSL_OCIOMain(color inColor = 0, output color outColor = 0)". Is that the right direction?

Thanks for your help,
Patrick
Larry Gritz
 

Hi, Patrick!

Before we dive in, can you explain what you mean by "native OSL generation support in OCIO"? 

Are you talking about OCIO using OSL, or OSL using OCIO? Can you give a very brief description of what you're trying to accomplish?

Some other answers inline:

  • For the language specifications I use the default link from the OSL home page (which is for the master branch). Is that ok to start from master? or should I switch to the latest release?
I think either is fine? Depends on your timeline, what exactly you need to do. Some areas there are little differences between master and release, others are large (but a new release will come at some point).

  • As I'm starting I would like to have some preliminary validations. When generating an OSL program, I currently use oslc command line tool to validate it (e.g. ./oslc -v -Werror cm1.osl). Is it enough? Is it the right approach?
What do you mean by validation? That would certainly tell you if it's valid syntax.

  • While adding the OSL support, I did not find any support for a vector of 4 (floats, ints, etc.). I only found "vector" (i.e. three values). Is there a kind of vec4 support?

Currently, there are vector4 and color4 helper classes in (shader headers) vector4.h and color4.h, which you can find in the OSL install area's "shader" subdirectory. These are basically just structs containing 4 floats (or a color and a float), and some operator overloading to make it intuitive to use. These mostly work fine, but there are plans to eventually make native types for vector2, vector4, and color4.

  • Currently the color transformation (code generated at runtime to perform the color transform from one color space to another one, which applies the processing pixel per pixel) generates a "generic shader" -> "shader OSL_OCIOMain(color inColor = 0, output color outColor = 0)". Is that the right direction?
Here's the part where I'm not quite sure what you're aiming for. Are you saying that you trying to generate an OSL shader that implements an OCIO color transform?

We get here quickly into another circular dependency, perhaps, because currently the complete shader to do any color transformations is

    shader do_ocio (color inColor = 0, string fromspace = "", string tospace = "", output color outColor = 0)
    {
        outColor = transformc(fromspace, tospace, inColor);
    }

and it assumes that the built-in function transformc, deep down below, has access to OCIO. :-)


--
Larry Gritz
Patrick Hodoul <patrick.hodoul@...>
 

On Mon, Jul 5, 2021 at 10:51 PM, Larry Gritz wrote:
Before we dive in, can you explain what you mean by "native OSL generation support in OCIO"? 
 
Are you talking about OCIO using OSL, or OSL using OCIO? Can you give a very brief description of what you're trying to accomplish?

The idea is to follow to current OCIO design when generating HLSL, GLSL & Cg code because there are all (including OSL) C like syntax.
OCIO currently generates (at runtime) some fragment shader code as a string, and let the host (e.g. Maya, Arnold, etc.) 'glue' it to its 'global' fragment shader, download/compile the final fragment shader code, etc. By doing so, there is no dependencies in the OCIO library to some 'tools/libraries' of the targeted language. OCIO only generates a string (plus some extra data for specific needs when needed).

Only the unit tests need some dependencies. For example, the GLSL unit test framework used by OCIO needs opengl, glut to run but the library does not.

So my current OCIO prototype generates a string containing some OSL code, and I manually use 'oslc' to validate the syntax. That's the short-term goal as it helps me to first better understand OSL, and second to see if the OCIO public API needs some changes. So the prototype generates something like:

shader OSL_OCIOMain(color inColor = 0, output color outColor = 0)
{
color OCIOMain(color inPixel)
{
  <some mathematical operations to implement the color processing>
}
outColor = OCIOMain(inColor);
}

That's a prototype and I'm open to suggestions.

Another approach could be to offer CM support (using OCIO) directly in OSL e.g. something like your suggestion. Is that a good design short-term & long-term? What effort it represents?
(My feeling is the approach will still need OCIO to generate some OSL code)

Not having your knowledge of OSL that's difficult for me to see pros/cons for OSL.
But I can only highlight that the current approach introduces no cyclic dependencies but imposes some extra work to the host.
Jonathan Stone
 

Another option worth considering would be to generate a MaterialX graph for the color transform math, allowing shading code to be generated in any supported language.  The current set of supported languages includes OSL, MDL and GLSL, with SPIR-V and ESSL well along in development.  This might help to decouple the logic for color transform generation from the diversity of shading languages that would be useful to support.
Chris Kulla
 

Hi Patrick,

Just to clarify, Larry's suggestion about using "transformc" is what is currently implemented in OSL (though it is an optional feature that only works if you compile with OCIO as a dependency).

It is not necessarily the most performant path since it has to call through interfaces designed to process many colors at once for a single color -- but it does work. When it was added, we expected that all uses of it would be confined to transforms of shader parameters and therefore we could rely on runtime optimization to constant fold the transformation, so render time performance was not impacted.

I can't really think of any major advantage to emitting OSL shader text instead. As you said, it would require a lot of support code in the host emitting the material network which seems like a big downside.

Do you see any drawbacks to what is currently implemented?


Chris
Thomas METAIS
 

Hi,
 
I would like to take the opportunity to share a limitation we've experienced with the current implementation of color tranformations using OCIO in OSL.
 
Our users make massive use of color transformations in their shading networks, sometimes going back and forth between colorspaces (transformc(c0, c1, col) --> col = f(col) --> transformc(c1, c0, col)).
 
Those operations are really expensive when using OCIO in OSL.
Indeed in the implementation of ocio_transform in the ShadingSystem, the OCIO color processor is re-created each time the transformation is different from the last time it was called.
 
In our OSL integration we had to bypass transformc by using a mere matrix transform when possible, ie when the color transformation is linear, but this is not always the case.
 
In their C++ implementation, our shaders use a map to store color processors, maybe a similar idea could be interesting in OSL shadingSystem?
 
 
Thanks for your help,
Thomas (Illumination MacGuff)
Patrick Hodoul <patrick.hodoul@...>
 

Thanks Larry & Chris, I completely missed that 'transformc' also supports OCIO (through OIIO). I now have to read/investigate :) 
Larry Gritz
 

On Jul 8, 2021, at 5:13 AM, Thomas METAIS <thomas.metais@...> wrote:

Our users make massive use of color transformations in their shading networks, sometimes going back and forth between colorspaces (transformc(c0, c1, col) --> col = f(col) --> transformc(c1, c0, col)).

I've certainly seen many examples of shaders where colors are transformed in and out of color spaces for particular computations, but usually it's for artistic control, like using a color space such as HSL, so that a computation can adjust saturation separately, etc. I don't think I've seen any examples where shaders transform into and out of an OCIO-defined space in the middle of a shader.

Can you give some concrete examples of how this is used?

If this is a common idiom, we definitely can try to speed it up. We made sure that transformc *could* transform between arbitrary OCIO-defined color spaces, but because it's not something we use commonly and weren't aware that others did, we haven't really looked at optimization. I'm sure there's lots of low-hanging fruit.

 
Those operations are really expensive when using OCIO in OSL.
Indeed in the implementation of ocio_transform in the ShadingSystem, the OCIO color processor is re-created each time the transformation is different from the last time it was called.
 
In our OSL integration we had to bypass transformc by using a mere matrix transform when possible, ie when the color transformation is linear, but this is not always the case.
 
In their C++ implementation, our shaders use a map to store color processors, maybe a similar idea could be interesting in OSL shadingSystem?


I'm pretty sure that the "creation" in that it's really calling in the OIIO layer is caching the processors into a map. I can double check this, but it's possible that the slow part is not because it's literally creating new color processors on every call. In any case, I bet there are some big improvements we can make here, such as using OSL's runtime optimization to skip the map when the color space names can be figured out during optimization, akin to what we do to replace texture names with handles.


--
Larry Gritz
Larry Gritz
 

We allow transformation between arbitrary color spaces, falling back on OCIO when encountering names we don't know (but figure might be in the user's color config). But this was intended originally as an edge case that should work, but probably would be rare and didn't need to be high performance.

There are two big drawbacks to this scheme:

1. As Thomas noted, if you rely on this heavily, it's probably very slow. I'm sure we can eliminate some overhead, but any way you shake it, calling all the way down through several layers to OCIO for a transformation that in the end is just a simple matrix multiply or a gamma curve is going to be needlessly more expensive than the underlying math. This is especially true for the shading case where we are transforming one point at a time (whereas when transforming a whole image of pixels as one call to OCIO, it's much lower overhead).

2. We currently have no idea how this is going to work on the GPU.

So Patrick's question opens up some interesting new ways of thinking about the issue. What if the way we used OCIO was not to hand it a color and ask it to transform it for us (for each and every individual color value), but rather to ask OCIO for the OSL shader code that implements the transformation, which we can fold into the code as we JIT? It's a much harder/bigger problem, but would (a) JUST WORK on the GPU, to the extent that the rest of OSL math works, (b) is probably more efficient by eliminating all the mapping of color space names to processors, and point-by-point calling of the processors, (c) would subject the implementation of the color space transform math itself to the squeeze of the OSL runtime optimizer.

It's a much bigger ask of the OCIO team than would ever have occurred to us to propose. But if it existed, would certainly represent a fresh way to think about the problem.


On Jul 8, 2021, at 6:01 AM, Patrick Hodoul <patrick.hodoul@...> wrote:

Thanks Larry & Chris, I completely missed that 'transformc' also supports OCIO (through OIIO). I now have to read/investigate :) 

--
Larry Gritz
Patrick Hodoul <patrick.hodoul@...>
 
Edited

On Jul 8, 2021, at 6:01 AM, Patrick Hodoul <patrick.hodoul@...> wrote:
Thanks Larry & Chris, I completely missed that 'transformc' also supports OCIO (through OIIO). I now have to read/investigate :) 
I now have more questions than answers to correctly understand the workflow.

Arnold supports OSL using the OSL backend to PTX, and I saw a demo using the 'texture()' method running in CPU & GPU. That's great. Arnold is only a consumer (no need to know how that's implemented in PTX) relying on OSL technology(ies) to run without any 'home-maid' steps. This is exactly the kind of behavior I would like to have for OCIO & OSL workflow. 

But I was not able to find anything concerning 'transformc()'. The two methods are interesting because they both are using external libraries (OIIO for texture(), and OIIO / OCIO for transformc()).
  • Is the 'transformc()' method supported in PTX?
    (Looking in the code I notice that OCIOColorSystem is disabled if __CUDACC__ is ON)
  • I'm curious, how OIIO vs PTX is working?
  • Why supporting OCIO through OIIO, and not directly?

2. We currently have no idea how this is going to work on the GPU.
Referring to the previous first point, does it means that there is currently no GPU (i.e. PTX) support for 'transformc()'?
If the answer is 'no GPU support' then using transformc() as-is is problematic.

Based on my understanding of Larry suggestion, what about a third option which consists to something like

    shader do_ocio (color inColor = 0, output color outColor = 0)
    {
        outColor = transformc("/usr/hodoulp/aces_x_x/config.ocio", "Sony SLog3 / SGamut3", "ACEScg", inColor);
    }

oslc first 'expands' the color transformation and second generates the OSL bytecode

    shader do_ocio (color inColor = 0, output color outColor = 0)
    {
        color custom_transformc(color inPixel)
        {
             color4 outColor = {inPixel, 1.};

             // Add Range processing
            {
                 outColor.rgb = outColor.rgb * vector(2.3549, 2.3549, 2.3549) + vector(0.02659, 0.02659, 0.02659);
                 outColor.rgb = max(vector(0.0009, 0.0009, 0.0009), outColor.rgb);
                 outColor.rgb = min(vector(2.5, 2.5, 2.5), outColor.rgb);
            } 

            // Add FixedFunction 'REC2100_Surround (Forward)' processing
           {
                float Y = max( 1e-4, 0.2627 * outColor.rgb[0] + 0.6780 * outColor.rgb[1] + 0.0593 * outColor.rgb[2] );
                float Ypow_over_Y = pow( Y, -0.25);
                outColor.rgb = outColor.rgb * Ypow_over_Y;
            }

            <...>

            return outColor.rgb;
        }

        outColor = custom_transformc(inColor);
    }


My prototype is currently generating a string containing a OSL shader. The advantages are to be perfectly generic and no contraints on OSL vs. OCIO. OSL receives a shader code to compile (not knowing where it's coming from) so it will always work. And the host will have the 'best' performance (i.e. no more layers). The fallbacks are a bloated shader code (depending of the color transformation that could represent lot of code) and to add some work on the host to manually 'glue' everything at runtime (i.e. adding an extra home-made step which is not a good idea).

The third approach keeps the simplicity of the current OSL proposal (i.e. transformc) with the portability & performance (i.e. only process OSL code) of my proposal. 

I notice that my post is going in different directions with questions of different topics, I'm sorry but I still struggle to see & understand all the pieces.

Patrick
Patrick Hodoul <patrick.hodoul@...>
 
Edited

Here is the status discussed at the 2021-07-08 OSL TSC meeting (Refer to https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/edit):

  • OCIO + OSL discussion

    • Patrick presented some context on the mailing list thread. OCIO currently support a few realtime shader languages and wanted to explore supporting OSL.

    • The goal is to better support the color transformations exposed by MaterialX graphs.

    • We discussed a few different approaches, the consensus seems to be that letting OCIO generate OSL (or OSO) code for the relevant transforms is probably a good starting point. We can look into integrating that more deeply into OSL itself at some point such that it becomes the “backend” of the existing transformc call. This way would allow it to support all the backends that OSL supports because the translation would happen at the OSO bytecode level.


The approach provides a major improvement to OCIO (by improving MatX integration via OSL) and to OSL (by adding color mgt on GPU). That's great because both projects benefit from the integration. We also discussed about optimal integration(s) of color management in OSL and increasing OCIO realtime shader language support. Some put the emphasis on technology like LLVM (used by the OSL backend and by tools like https://halide-lang.org) to open new perspectives for OCIO language support. The suggestion is very interesting.

Concerning the current limitations to the color mgt support in OSL (i.e. dynamic properties (uniforms in GLSL/HLSL) and lookup tables (texture on GPU)), the latter one can be investigated at some point using 'texture' as example because it already supports the memory buffer move from CPU to GPU.

Concerning the optimal integration of color mgt in OSL, the enhancement of 'texture' with color mgt support was also discussed and seen as a potential future improvement.

Thanks everyone for your wise advices. It's now time to work :)
 
Patrick
Larry Gritz
 

Just to directly answer a few of Patrick's questions that I hadn't responded to:

On Jul 8, 2021, at 11:10 AM, Patrick Hodoul <patrick.hodoul@...> wrote:

But I was not able to find anything concerning 'transformc()'. The two methods are interesting because they both are using external libraries (OIIO for texture(), and OIIO / OCIO for transformc()).
  • Is the 'transformc()' method supported in PTX?
    (Looking in the code I notice that OCIOColorSystem is disabled if __CUDACC__ is ON)

The call is supported for PTX, but (currently) not any of the transformations that require OCIO support. So you can transformc("RGB", "HSL", mycolor), but not transformc("alocgf", "acesf", mycolor).

  • I'm curious, how OIIO vs PTX is working?
OIIO vs PTX? I'm not sure what you mean.

OSL PTX output works very well for the core language and basic texturing. But there are many missing OSL features left to implement to get us to full feature parity in GPU rendering, in particular, most of the things that are mediated by RendererServices (which are the things that on the CPU side, the renderer is really supplying the answers that OSL by itself can't be expected to know).

  • Why supporting OCIO through OIIO, and not directly?
The general rule here is that since OIIO is a required dependency of OSL (but not the other way around), functionality that is needed in a roughly equivalent manner by both packages is put into OIIO rather than replicated in both. OIIO needs only a very tiny slice of OCIO's functionality (get a processor that transforms from one named space too another, and use that processor to transform individual colors or arrays of them). OSL's needs for OCIO are even more modest, and are a strict subset of the OCIO-related functionality that OIIO exports through its public APIs. So just relying on OIIO's APIs makes the OCIO-related code footprint within OSL extremely minimal, and that saves me a lot of work. Plus, it eliminates a whole class of problems that could come from an app that uses an OIIO built with one version of OCIO and an OSL that used a different OCIO -- if OSL gets its OCIO functionality from OIIO, there is no chance of a conflict.

There's an alternate universe in which a butterfly flapped its wings slightly differently in 2008 and the result was that OIIO & OSL are one single project. But in our universe, they ended up in different repos, and sometimes we make judgment calls about which side of the line certain functionality lives on.

--
Larry Gritz
Patrick Hodoul <patrick.hodoul@...>
 


The first two points were discussed during the OSL TSC meeting.
  1. On long term, the goal is to enhance transformc() to support arbitrary OCIO config file in GPU mode
  2. The short-term problem to support any arbitrary color transformations is the LUT support in GPU mode. The challenge is to have a memory buffer exchange from the CPU to the GPU memory. My question was to understand how the image buffer exchange is done by the texture() method implementation in order to mimic the 'work' for LUT memory buffer. During the TSC meeting, we discussed about that topic and someone (I do not remember who) volunteered to help.

The last point is the transformc() method which currently performs the color transformation using OCIO through OIIO. On the other hand, a previous post (in that discussion thread) mentions an important performance hit when doing color transformations. So, one potential change is to directly use OCIO in OSL (instead of the indirect use through OIIO) to better benefit from all the latest OCIOv2 improvements (i.e. caches) and to avoid the OIIO extra layer.

Note:
The dependency on OCIO is already existing but hidden in OIIO. The change will only make it explicit (mandatory or optional depending of the OSL needs) which I think, is a better approach.
The mismatch between OCIO in OIIO and OCIO is not a problem for OSL at least because texture() and texture3D() methods are using OIIO (without any color mgt), and transformc() methods will use OCIO.

I will try to be at the next OSL TSC meeting (the 19th of August) so we can continue the discussion if needed.

Patrick

1 - 13 of 13