Review: Inital pass at searchpath impl


Malcolm Humphreys <malcolmh...@...>
 

Inital pass at searchpath impl
- added environment variable string expansion code in PathUtils.h (GetEnvMap() & EnvExpand())
- added FileExists() function to PathUtils.h
- added searchpath functionality into Config:: class (setSearchPath(path), getSearchPath(expand), findFile(filename)) this hasn't be plugged into anything yet.
- added ocio_core_tests.sh.in so we can pass in some env vars for the unit tests

http://github.com/malcolmhumphreys/OpenColorIO/commit/80bfd194bd7fd3d787e7a8795ad63be6050702c6

.malcolm


Malcolm Humphreys <malcolmh...@...>
 

- removed the EnvExpand recusion limits as this didn't work in the production env which
has more than 30 env vars. might want to add limits in later if this becomes a problem.

http://github.com/malcolmhumphreys/OpenColorIO/commit/a172426e59ddebc4ccd0691b796bfe493677ec9f

.malcolm

On 25/10/2010, at 6:17 PM, Malcolm Humphreys wrote:

Inital pass at searchpath impl
- added environment variable string expansion code in PathUtils.h (GetEnvMap() & EnvExpand())
- added FileExists() function to PathUtils.h
- added searchpath functionality into Config:: class (setSearchPath(path), getSearchPath(expand), findFile(filename)) this hasn't be plugged into anything yet.
- added ocio_core_tests.sh.in so we can pass in some env vars for the unit tests

http://github.com/malcolmhumphreys/OpenColorIO/commit/80bfd194bd7fd3d787e7a8795ad63be6050702c6

.malcolm


Malcolm Humphreys <malcolmh...@...>
 

- fixed a bug with abs searchpath entries
- empty '::' searchpaths now resolve to the profile cwd

http://github.com/malcolmhumphreys/OpenColorIO/commit/5d922362259e86073592481d428926de1054414d

.malcolm

On 25/10/2010, at 6:59 PM, Malcolm Humphreys wrote:

- removed the EnvExpand recusion limits as this didn't work in the production env which
has more than 30 env vars. might want to add limits in later if this becomes a problem.

http://github.com/malcolmhumphreys/OpenColorIO/commit/a172426e59ddebc4ccd0691b796bfe493677ec9f

.malcolm

On 25/10/2010, at 6:17 PM, Malcolm Humphreys wrote:

Inital pass at searchpath impl
- added environment variable string expansion code in PathUtils.h (GetEnvMap() & EnvExpand())
- added FileExists() function to PathUtils.h
- added searchpath functionality into Config:: class (setSearchPath(path), getSearchPath(expand), findFile(filename)) this hasn't be plugged into anything yet.
- added ocio_core_tests.sh.in so we can pass in some env vars for the unit tests

http://github.com/malcolmhumphreys/OpenColorIO/commit/80bfd194bd7fd3d787e7a8795ad63be6050702c6

.malcolm


Jeremy Selan <jeremy...@...>
 

Malcolm,

I wish I had known you were thinking about writing this particular
piece of code; I probably would have asked you to hold off on it for a
little while... (This is my mistake, I do realize that this task was
listed in the OCIO TODO file).

We've just started testing the current version of OCIO at Imageworks,
and already we've run into some major confusion regarding lut paths.
This past week's experiences (with Joseph playing the kind role of
guinea pig) illuminates this discussion, and makes me think that some
backtracking may be in order on this topic.

* An OCIO profile is not just a single configuration file in isolation
- it's a config AND a set of associated luts. Keeping these together
all times (and conceptually thinking of them as inseparable) is
probably the only way to avoid ambiguity and errors. An OCIO config
is not useful if it gets separated from its luts, and vice-versa. In
fact, if it's possible to separate the two, it WILL happen, and only
bad will come of it. It's hit us once this week, I'd like to prevent
anyone else from ever having to track down this type of issue.

* I'd like for an OCIO profile to be able to be distributed, and used,
either as a single self contained file or as a working directory.

* Consider, what would the OCIO world look like if we 'locked down' a
fixed directory layout that all profiles had to conform to?

Fixed OCIO profile (in directory form)

rootdir/
config.txt
filetransforms/
a.lut, b.lut, etc.

Point $OCIO at the root directory, (not the config,txt file), and
everything "just works". All search paths, resource paths, go away. A
bit inconvenient? Perhaps. But unambiguous and simple? Definitely.

* With this model, the "single file" profile + luts representation
becomes trivial: we would just support pointing $OCIO at a compressed
version of the .ocio directory layout (zip, tgz, etc).

Want to know what's inside your .ocio.zip config? Easy, just unzip it
using the normal os tools. You can also unzip a profile, drag and
drop a new lut internally, and then re-zip it for distribution. Super
easy, and no lut transcoding needed.

But what would the API look like if we went with this approach? It
would be more directory / file based, and less stream centric.
Would the "write" call write a directory instead? (and copy all
referenced luts into a fixed local lut dir?) Probably.

As foreign as this idea sounds, putting the OCIO configuration on
rails and locking down a fixed directory layout may really help to
minimize errors and confusion going forward.

On a slighly different topic, I think it will be preferable when
possible to minimize the use of envvars. Consider the case where the
'global config' (created with $OCIO) is not used. (where config
objects are instead manually initialized). Any options specified with
envvars, such as search paths, will by definition be 'global' and will
have to be worked around (unset) in the the other profiles.

-- Jeremy


---------- Forwarded message ----------
From: Malcolm Humphreys <malcolmh...@mac.com>
Date: Mon, Oct 25, 2010 at 3:29 AM
Subject: Re: [ocio-dev] Review: Inital pass at searchpath impl
To: ocio...@googlegroups.com


- fixed a bug with abs searchpath entries
- empty '::' searchpaths now resolve to the profile cwd

http://github.com/malcolmhumphreys/OpenColorIO/commit/5d922362259e86073592481d428926de1054414d

.malcolm

On 25/10/2010, at 6:59 PM, Malcolm Humphreys wrote:

- removed the EnvExpand recusion limits as this didn't work in the production env which
has more than 30 env vars. might want to add limits in later if this becomes a problem.

http://github.com/malcolmhumphreys/OpenColorIO/commit/a172426e59ddebc4ccd0691b796bfe493677ec9f

.malcolm

On 25/10/2010, at 6:17 PM, Malcolm Humphreys wrote:

Inital pass at searchpath impl
- added environment variable string expansion code in PathUtils.h (GetEnvMap() & EnvExpand())
- added FileExists() function to PathUtils.h
- added searchpath functionality into Config:: class (setSearchPath(path), getSearchPath(expand), findFile(filename)) this hasn't be plugged into anything yet.
- added ocio_core_tests.sh.in so we can pass in some env vars for the unit tests

http://github.com/malcolmhumphreys/OpenColorIO/commit/80bfd194bd7fd3d787e7a8795ad63be6050702c6

.malcolm


Jeremy Selan <jeremy...@...>
 

I've checked in this patch to the master branch. The code itself
looks good; whether we choose to backtrack from that API direction in
the upcoming weeks can/should be a separate issue from this particular
checkin.

-- Jeremy

On Mon, Oct 25, 2010 at 10:55 PM, Jeremy Selan <jeremy...@gmail.com> wrote:
Malcolm,

I wish I had known you were thinking about writing this particular
piece of code; I probably would have asked you to hold off on it for a
little while... (This is my mistake, I do realize that this task was
listed in the OCIO TODO file).

We've just started testing the current version of OCIO at Imageworks,
and already we've run into some major confusion regarding lut paths.
This past week's experiences (with Joseph playing the kind role of
guinea pig) illuminates this discussion, and makes me think that some
backtracking may be in order on this topic.

* An OCIO profile is not just a single configuration file in isolation
- it's a config AND a set of associated luts.  Keeping these together
all times (and conceptually thinking of them as inseparable) is
probably the only way to avoid ambiguity and errors.   An OCIO config
is not useful if it gets separated from its luts, and vice-versa.   In
fact, if it's possible to separate the two, it WILL happen, and only
bad will come of it.  It's hit us once this week, I'd like to prevent
anyone else from ever having to track down this type of issue.

* I'd like for an OCIO profile to be able to be distributed, and used,
either as a single self contained file or as a working directory.

* Consider, what would the OCIO world look like if we 'locked down' a
fixed directory layout that all profiles had to conform to?

Fixed OCIO profile (in directory form)

rootdir/
         config.txt
         filetransforms/
               a.lut, b.lut, etc.

Point $OCIO at the root directory, (not the config,txt file), and
everything "just works". All search paths, resource paths, go away.  A
bit inconvenient? Perhaps.   But unambiguous and simple?  Definitely.

* With this model, the "single file" profile + luts representation
becomes trivial:  we would just support pointing $OCIO at a compressed
version of the .ocio directory layout (zip, tgz, etc).

Want to know what's inside your .ocio.zip config?  Easy, just unzip it
using the normal os tools.  You can also unzip a profile, drag and
drop a new lut internally, and then re-zip it for distribution. Super
easy, and no lut transcoding needed.

But what would the API look like if we went with this approach?   It
would be more directory / file based, and less stream centric.
Would the "write" call write a directory instead?  (and copy all
referenced luts into a fixed local lut dir?) Probably.

As foreign as this idea sounds, putting the OCIO configuration on
rails and locking down a fixed directory layout may really help to
minimize errors and confusion going forward.

On a slighly different topic, I think it will be preferable when
possible to minimize the use of envvars. Consider the case where the
'global config' (created with $OCIO) is not used. (where config
objects are instead manually initialized). Any options specified with
envvars, such as search paths, will by definition be 'global' and will
have to be worked around (unset) in the the other profiles.

-- Jeremy


---------- Forwarded message ----------
From: Malcolm Humphreys <malcolmh...@mac.com>
Date: Mon, Oct 25, 2010 at 3:29 AM
Subject: Re: [ocio-dev] Review: Inital pass at searchpath impl
To: ocio...@googlegroups.com


- fixed a bug with abs searchpath entries
- empty '::' searchpaths now resolve to the profile cwd

http://github.com/malcolmhumphreys/OpenColorIO/commit/5d922362259e86073592481d428926de1054414d

.malcolm

On 25/10/2010, at 6:59 PM, Malcolm Humphreys wrote:

- removed the EnvExpand recusion limits as this didn't work in the production env which
has more than 30 env vars. might want to add limits in later if this becomes a problem.

http://github.com/malcolmhumphreys/OpenColorIO/commit/a172426e59ddebc4ccd0691b796bfe493677ec9f

.malcolm

On 25/10/2010, at 6:17 PM, Malcolm Humphreys wrote:

Inital pass at searchpath impl
- added environment variable string expansion code in PathUtils.h (GetEnvMap() & EnvExpand())
- added FileExists() function to PathUtils.h
- added searchpath functionality into Config:: class (setSearchPath(path), getSearchPath(expand), findFile(filename)) this hasn't be plugged into anything yet.
- added ocio_core_tests.sh.in so we can pass in some env vars for the unit tests

http://github.com/malcolmhumphreys/OpenColorIO/commit/80bfd194bd7fd3d787e7a8795ad63be6050702c6

.malcolm


Malcolm Humphreys <malcolmh...@...>
 

We've just started testing the current version of OCIO at Imageworks,
and already we've run into some major confusion regarding lut paths.
This past week's experiences (with Joseph playing the kind role of
guinea pig) illuminates this discussion, and makes me think that some
backtracking may be in order on this topic.
Can you elaborate a little on what was confusing.

* An OCIO profile is not just a single configuration file in isolation
- it's a config AND a set of associated luts. Keeping these together
all times (and conceptually thinking of them as inseparable) is
probably the only way to avoid ambiguity and errors. An OCIO config
is not useful if it gets separated from its luts, and vice-versa. In
fact, if it's possible to separate the two, it WILL happen, and only
bad will come of it. It's hit us once this week, I'd like to prevent
anyone else from ever having to track down this type of issue.

* I'd like for an OCIO profile to be able to be distributed, and used,
either as a single self contained file or as a working directory.
+1 to a self contained file

* Consider, what would the OCIO world look like if we 'locked down' a
fixed directory layout that all profiles had to conform to?

Fixed OCIO profile (in directory form)

rootdir/
config.txt
filetransforms/
a.lut, b.lut, etc.

Point $OCIO at the root directory, (not the config,txt file), and
everything "just works". All search paths, resource paths, go away. A
bit inconvenient? Perhaps. But unambiguous and simple? Definitely.
What about per shot balance grades and look / creative grades, I would cry if I needed to create a zip ocio profile per shot to do this. Then re-syncing these profiles would be a nightmare.

* With this model, the "single file" profile + luts representation
becomes trivial: we would just support pointing $OCIO at a compressed
version of the .ocio directory layout (zip, tgz, etc).
Let me describe how I would like to see OCIO setup on a show.

* central show OCIO profile describing
- working/reference space
- display devices
- colorspace definitions
- luts (relative to this profile)

* show/seq/shot (cdl) balance grades as separate xml on disk

* show/seq/shot (cdl/lut/spline) look / creative grades as separate files on disk

I could see this setup being forked and lock at different stages of production (prepro/onset, trailer, theatrical).

Onset when tech and human resources are limited, being able to setup this up with minimal hassle would be ideal.

For simplicity sake you had some cdl balance grades layout like this.
/job/config/ocio/balance_approved.xml
/job/myseq/config/ocio/balance_approved.xml
/job/myseq/someshot/config/ocio/balance_approved.xml
/job/myseq/othershot/config/ocio/balance_approved.xml

Your search path would be in the central config "/job/$SEQ/$SHOT/config/ocio:/job/$SEQ/config/ocio:/job/config/ocio"

Changing your $SEQ and $SHOT would allow switching between grades.

This is not to say that you might merge this into a more involved system when in house, maybe without using env vars. eg. a Transform plugin which picks up the approved grade from a database.

I think it's important to support both ways of working.

But you do want to make it possible to lock down which files are loaded relative to the profile and which use the searchpath.

I agree this could causes issues.

I would look at having some kind of prefix to filenames to mark them as 'only look in the resource path' this could be by prefixing the paths with an '@' eg. !<FileTransform> {src: @/lg10.spi1d} this would resolve to only paths in the resource path anything without an '@' would look through the search path (in order).

* resourcepath - is relative to the profile, either embedded or on disk
* searchpath - is a ':' ordered list of directories to search for file references
* file references - are absolute paths / '@' in the resourcepath / or somewhere in the searchpath

Want to know what's inside your .ocio.zip config? Easy, just unzip it
using the normal os tools. You can also unzip a profile, drag and
drop a new lut internally, and then re-zip it for distribution. Super
easy, and no lut transcoding needed.

But what would the API look like if we went with this approach? It
would be more directory / file based, and less stream centric.
Would the "write" call write a directory instead? (and copy all
referenced luts into a fixed local lut dir?) Probably.

As foreign as this idea sounds, putting the OCIO configuration on
rails and locking down a fixed directory layout may really help to
minimize errors and confusion going forward.
-1 for fixed directory layout, I think it's important that this is configurable. I can't say how different places will want to setup their pipe. Configurable decision points should be left configurable ( with sensible defaults and examples ;) ).

I think it's important for a process to validate the packaging of a profile. I was hoping for something like:

Package up a profile
$> ociopacakge source.ocio packaged.ocio

This would package up all the absolute and resourcepaths into a packaged ocio profile.

Package up a shot profile
$> ociopackage --resolve-searchpath source.ocio packaged.ocio

This would resolve the searchpath and package up any found files and these would be given '@' references in the packaged.ocio

Unpack a profile
$> ociopackage -x packaged.ocio ~/tmp/

I'm a lot more in favour of a single file with embedded ascii or binary blobs at the bottom. It's quite easy for someone unzip a dir remove a lut and re-zip it and circumvent any validation process.

I also I think people will be checking these into version control systems (aka easy setup publish system) being able to open profiles in a text editor / web browser and reading the header would be really nice rather than having to unzip to check stuff out.

I'm happy to code up a prototype of this in a branch if people are keen to see that idea.

On a slighly different topic, I think it will be preferable when
possible to minimize the use of envvars. Consider the case where the
'global config' (created with $OCIO) is not used. (where config
objects are instead manually initialized). Any options specified with
envvars, such as search paths, will by definition be 'global' and will
have to be worked around (unset) in the the other profiles.
Can you elaborate on this? I don't completely get what you mean, does the '@' in file reference help distinguish between local and global?

.malcolm


-- Jeremy


---------- Forwarded message ----------
From: Malcolm Humphreys <malcolmh...@mac.com>
Date: Mon, Oct 25, 2010 at 3:29 AM
Subject: Re: [ocio-dev] Review: Inital pass at searchpath impl
To: ocio...@googlegroups.com


- fixed a bug with abs searchpath entries
- empty '::' searchpaths now resolve to the profile cwd

http://github.com/malcolmhumphreys/OpenColorIO/commit/5d922362259e86073592481d428926de1054414d

.malcolm

On 25/10/2010, at 6:59 PM, Malcolm Humphreys wrote:

- removed the EnvExpand recusion limits as this didn't work in the production env which
has more than 30 env vars. might want to add limits in later if this becomes a problem.

http://github.com/malcolmhumphreys/OpenColorIO/commit/a172426e59ddebc4ccd0691b796bfe493677ec9f

.malcolm

On 25/10/2010, at 6:17 PM, Malcolm Humphreys wrote:

Inital pass at searchpath impl
- added environment variable string expansion code in PathUtils.h (GetEnvMap() & EnvExpand())
- added FileExists() function to PathUtils.h
- added searchpath functionality into Config:: class (setSearchPath(path), getSearchPath(expand), findFile(filename)) this hasn't be plugged into anything yet.
- added ocio_core_tests.sh.in so we can pass in some env vars for the unit tests

http://github.com/malcolmhumphreys/OpenColorIO/commit/80bfd194bd7fd3d787e7a8795ad63be6050702c6

.malcolm


Jeremy Selan <jeremy...@...>
 

There's a lot of good ideas here, it's taken me a bit of time to think
through the implications.

I'm going to be out of town for the remainder of the week, perhaps we
could try an internet call early next week to discuss these specifics
further? (Skype, iChat, etc). Many of these points can probably be
hashed out easier live, and then we can summarize our thoughts back to
the list.

What about per shot balance grades and look / creative grades, I would cry if I needed to create a zip ocio profile per shot to do this. > Then re-syncing these profiles would be a nightmare.
Agreed. Under no circumstances are we going to make per-shot looks
complicated to use. We're in the same boat here.

JS: I think it will be preferable when possible to minimize the use of envvars.
MH: Can you elaborate on this?
Your approach to implementing per-shot looks - using environment
variables within search paths - is clever. However, I'm not sold.
The portion I'm concerned about is the use of environment variables as
a communications mechanism.

For an application (such as a compositor) where you're not changing
show / shot / sequence at runtime the use of envvars is no big deal.
For example, say you have a per-shot display lut
(/job/$SHOT/.../shotgrade.3dl). You could run a script which sets up
the shot environment, launch the APP, and everything just works.

However, consider an image-viewing tool which needs to flipbook
multiple shots (or whole sequences) back to back. I.e., the display
transform, as queried from OCIO, would be different based on the shot
name. Would this approach be possible using an envvar mechanism? I
dont believe so. Envvars are commonly locked down at launch time.
And, even if you could modify it at runtime as playback was in
progress, would you want to? How would you notifiy OCIO that it had
changed? This seems like a tricky implementation, and strikes me as
going down the path of evil.

I would prefer to promote this concept of 'variables' to be an
explicit communications mechanism, rather then using envvars as a
back-door. Alternate implementation options include:

* a config->setVariable option, where variables were available for
use in color transforms. (They could even retain the $SHOW $SHOT
syntax). This way the OCIO plugin could set it as needed, and with
the python API exposed in client apps the list of variables could even
be amended with facility code, as needed at runtime!

* Alternatively, with a less stateful implementation the variables /
arguments could be passed along in the config->getProcessor call. This
may be preferable to those computer scientists among us.

Either way, this passing of variables as arguments to a transforms is
half of the work towards allowing fully dynamic colorspaces. (The
other 1/2 being the plugin API to define the lists of transforms at
runtime).

Perhaps this is good reason to move forward on both ASAP?

-- Jeremy


Malcolm Humphreys <malcolmh...@...>
 

Hi,

I'm going to be out of town for the remainder of the week, perhaps we
could try an internet call early next week to discuss these specifics
further? (Skype, iChat, etc). Many of these points can probably be
hashed out easier live, and then we can summarize our thoughts back to
the list.
 
It might have to wait a few weeks, I'm in Hong Kong right now and I'm going to be MIA in asia for a bit.

But yes a chat would be good when we are both free.

Your approach to implementing per-shot looks - using environment
variables within search paths - is clever. However, I'm not sold
The portion I'm concerned about is the use of environment variables as
a communications mechanism.
 
They are a common mechanism for setting up initial state for a lot of apps. This being similar is a nice entry point for integration with existing systems.

But I agree the env vars should only be used to setup the initial state.

Even if it is possible to change this state with setenv()'s I would discourage this use with a better interface for changing state during execution, similar to what you have suggested.

Most places have some form of setshot system which bootstraps the env for a shot, not just for color but things like the required toolchain for that artist session doing task A (I'm guessing you have something similar at SPI).

I can guarantee that I would need to write something that grabbed env's and setup an initial state through which ever interface we settle on. I would prefer that this ocio env wrapper type code was internal rather than each shop doing it slightly differently.

For an application (such as a compositor) where you're not changing
show / shot / sequence at runtime the use of envvars is no big deal.
For example, say you have a per-shot display lut
(/job/$SHOT/.../shotgrade.3dl). You could run a script which sets up
the shot environment, launch the APP, and everything just works.
 
+1

Being able to setup/enfoce the initial state like this is important for multiple dept / group workflows. Who are most likely using diffrent toolchains but require to start with the same consistent viewing env between them.

In this style of setup each person gets pushed the currently approved viewing env for that session and are not concerned with modification of it.

However, consider an image-viewing tool which needs to flipbook
multiple shots (or whole sequences) back to back. I.e., the display
transform, as queried from OCIO, would be different based on the shot
name Would this approach be possible using an envvar mechanism? I
dont believe so.

Neither do I, that would be a bit clumsy for that specific problem.

Envvars are commonly locked down at launch time.
And, even if you could modify it at runtime as playback was in
progress, would you want to? How would you notifiy OCIO that it had
changed? This seems like a tricky implementation, and strikes me as
going down the path of evil.
 
I think a mixture of the two ideas could work well (see below).

I would prefer to promote this concept of 'variables' to be an
explicit communications mechanism, rather then using envvars as a
back-door. Alternate implementation options include:

* a config->setVariable option, where variables were available for
use in color transforms. (They could even retain the $SHOW $SHOT
syntax). This way the OCIO plugin could set it as needed, and with
the python API exposed in client apps the list of variables could even
be amended with facility code, as needed at runtime!
 
I was thinking the same kind of thing, but with the possibility of these grabbing their initial value from the env (if it exists).

I'm also thinking it might be better to be explicit with globals eg. config->addglobal(name) these would end up as a list in the yaml profile. Only the $globals in this list would be expanded and also the only ones initialized from the env. (This does resolve my niggles with the current expand code which has the potential to be a bit slow, if used in many)

These globals could be set/get dynamically at run-time ie config->getglobal(), config->setglobal(). I would prefer to call these globals over variables as it suggests their scope.

Right now I only see a direct use for globals with string parameters and would only want to add support for other types if we had a strong need for them.

As an example this would be nearly a complete list of globals I could see a use for ($JOB, $SCENE, $SEQ, $SHOT, $TASK, $USERGROUP, $BALLANCE_GRADE, $CREATIVE_GRADE, $EDIT_GRADE). I'm sure these have slightly different names per shop but represent the same concepts.

* Alternatively, with a less stateful implementation the variables /
arguments could be passed along in the config->getProcessor call. This
may be preferable to those computer scientists among us.
 
I definitely would like some form of dynamic parameters/variables, but these should be something different to globals.

Either way, this passing of variables as arguments to a transforms is
half of the work towards allowing fully dynamic colorspaces. (The
other 1/2 being the plugin API to define the lists of transforms at
runtime).
 
With the prospect of multiple image sequences / clips with transforms, configs and or grades. For these case are you proposing that we setup a single config and then store some kind of ocio processor relationship per clip?

Perhaps this is good reason to move forward on both ASAP?
 
It would be great to flesh this out as it would be good for client apps not to need to manage the details of this per clip state storage if this is the way we are thinking of going.

.malcolm


Jeremy Selan <jeremy...@...>
 

Malcolm, I like your suggestions.

My thoughts:

* The name 'globals' is good.

* I like pre-declaring the list of allowed variables beforehand for the config.

* I like the use of env-vars as defaults. This seems very user-friendly.

* I like supporting only strings for now. In the future, I can also
see numbers being supported (such as allowing for CDL blocks that rely
on envvars), but this can be added as needed.

* The FileTransform will probably have to get a bit smarter. Say
you're using a pershot lut, and for some shots you dont want any such
table. This could be solved by either not installing a lut in the
expected location, or by installing an identity lut. Which is
preferable? In the current implementation, if a lut is not found
this is always an error condition. Maybe we should introduce on the
FilmTransform a 'silent error mode', where this would just skip the
application of that specific LUT?

* The "globals" pre-declaration could also define an optional default
(used if the envvar is not set). If the envvar is not set, and no
default is specified, any references to the global variable would
probably be an error.

* I have some apprehensions about the config->getGlobal,
config->setGlobal implementation. My primary concern is that the
GlobalConfig() is const (by design), so clients couldnt actually call
setGlobals on the config in use. Instead, they would have to do
something like:

c = GetGlobalConfig().createEditableCopy()
c.setGlobals(...)
SetGlobalConfig( c )

This is a lot of copying, particularly as the globals often are tied
to the clip. If you're A/B-ing two clips in an image viewer,
conceptually there really is only one config, and two sets of globals.
Why force the client to copy and have duplicate configs?

I will mockup some ideas which solve this and see what the possible
solutions look like.

-- Jeremy


Malcolm Humphreys <malcolmh...@...>
 

On 09 Nov, 2010,at 06:53 AM, Jeremy Selan <jere...@...> wrote:

* I like supporting only strings for now. In the future, I can also
see numbers being supported (such as allowing for CDL blocks that rely
on envvars), but this can be added as needed.
 
+1

* The FileTransform will probably have to get a bit smarter. Say
you're using a pershot lut, and for some shots you dont want any such
table. This could be solved by either not installing a lut in the
expected location, or by installing an identity lut. Which is
preferable? In the current implementation, if a lut is not found
this is always an error condition. Maybe we should introduce on the
FilmTransform a 'silent error mode', where this would just skip the
application of that specific LUT?
 
I was tempted to always have a lut that would match in the searchpath, say a global identity lut.
As this is possible to circumvent this way I would fix this later. The identity lut option should be
optimized out so it seems like a good option. But some explicit flag saying apply this lut when
the file is missing seems like a good way to try first.

* The "globals" pre-declaration could also define an optional default
(used if the envvar is not set). If the envvar is not set, and no
default is specified, any references to the global variable would
probably be an error.
 
+1

* I have some apprehensions about the config->getGlobal,
config->setGlobal implementation. My primary concern is that the
GlobalConfig() is const (by design), so clients couldnt actually call
setGlobals on the config in use. Instead, they would have to do
something like:

c = GetGlobalConfig().createEditableCopy()
c.setGlobals(...)
SetGlobalConfig( c )

This is a lot of copying, particularly as the globals often are tied
to the clip. If you're A/B-ing two clips in an image viewer,
conceptually there really is only one config, and two sets of globals.
Why force the client to copy and have duplicate configs?
 
What are your thoughts on having a config per clip and for each config to maintain each state.
I know this breaks the current concept of the config. Would be nice to re-confirm this is the way
we should go.

.malcolm


Malcolm Humphreys <malcolmh...@...>
 

Hi Jeremy,

On 04/11/2010, at 3:32 PM, Malcolm Humphreys wrote:
I'm going to be out of town for the remainder of the week, perhaps we
could try an internet call early next week to discuss these specifics
further? (Skype, iChat, etc). Many of these points can probably be
hashed out easier live, and then we can summarize our thoughts back to
the list.
 
It might have to wait a few weeks, I'm in Hong Kong right now and I'm going to be MIA in asia for a bit.

But yes a chat would be good when we are both free.

Just letting you know I'm back from Holidays now, I'll try and go back through
and review what I have missed.

And we should line up a chat maybe next week if you are around.

.malcolm



Jeremy Selan <jeremy...@...>
 

Next week will be fantastic. Welcome back.
-- Jeremy

On Fri, Dec 3, 2010 at 8:05 PM, Malcolm Humphreys
<malcolmh...@mac.com> wrote:
Hi Jeremy,
On 04/11/2010, at 3:32 PM, Malcolm Humphreys wrote:

I'm going to be out of town for the remainder of the week, perhaps we
could try an internet call early next week to discuss these specifics
further? (Skype, iChat, etc). Many of these points can probably be
hashed out easier live, and then we can summarize our thoughts back to
the list.


It might have to wait a few weeks, I'm in Hong Kong right now and I'm going
to be MIA in asia for a bit.

But yes a chat would be good when we are both free.

Just letting you know I'm back from Holidays now, I'll try and go back
through
and review what I have missed.
And we should line up a chat maybe next week if you are around.
.malcolm


Jeremy Selan <jeremy...@...>
 

What's your schedule like this week? Up for a conference call? I'd
like to primarily discuss the env-vars / 'globals' / per-shot cc look
options.

-- Jeremy


Malcolm Humphreys <malcolmh...@...>
 

* I like supporting only strings for now. In the future, I can also
see numbers being supported (such as allowing for CDL blocks that rely
on envvars), but this can be added as needed.
It would also be super nice if the CDLTransform supported the full Color Correction Collection (CCC) in the CDL spec, where you could have something along the lines of '!<CDLTransform> { src: '/path/to/basegrades.ccc', id: '${SEQ}_${SHOT}_balance' }'.
* The FileTransform will probably have to get a bit smarter. Say
you're using a pershot lut, and for some shots you dont want any such
table. This could be solved by either not installing a lut in the
expected location, or by installing an identity lut. Which is
preferable? In the current implementation, if a lut is not found
this is always an error condition. Maybe we should introduce on the
FilmTransform a 'silent error mode', where this would just skip the
application of that specific LUT?
Keep the system simple let it error, it should be up to the user to setup the searchpath so that an identity lut is found if other paths fail to resolve to a lut.

It might be an idea to make this case not an exception, but a error which gets sent through some kind or error handler.

* The "globals" pre-declaration could also define an optional default
(used if the envvar is not set). If the envvar is not set, and no
default is specified, any references to the global variable would
probably be an error.
+1

.malcolm


Malcolm Humphreys <malcolmh...@...>
 

How about we shoot for tomorrow, I'm still getting backup to speed with things.

.malcolm

On 07 Dec, 2010,at 09:08 AM, Jeremy Selan <jere...@...> wrote:

What's your schedule like this week? Up for a conference call? I'd
like to primarily discuss the env-vars / 'globals' / per-shot cc look
options.

-- Jeremy


Tobias Pfeiffer <pixeld...@...>
 

Hi guys,
I do understand the limitations of the use of env vars to point to shot grades, but dont know how to deal with it (even 4 years later) in case of flipbooking with RV.
Do you have any suggestions / Best Practices for that?
Thank you.
Tobi

Am Freitag, 29. Oktober 2010 00:24:03 UTC+2 schrieb Jeremy Selan:

There's a lot of good ideas here, it's taken me a bit of time to think
through the implications.

I'm going to be out of town for the remainder of the week, perhaps we
could try an internet call early next week to discuss these specifics
further?  (Skype, iChat, etc).  Many of these points can probably be
hashed out easier live, and then we can summarize our thoughts back to
the list.

> What about per shot balance grades and look / creative grades, I would cry if I needed to create a zip ocio profile per shot to do this. > Then re-syncing these profiles would be a nightmare.

Agreed.  Under no circumstances are we going to make per-shot looks
complicated to use. We're in the same boat here.

>> JS:  I think it will be preferable when possible to minimize the use of envvars.
> MH: Can you elaborate on this?

Your approach to implementing per-shot looks - using environment
variables within search paths - is clever.  However, I'm not sold.
The portion I'm concerned about is the use of environment variables as
a communications mechanism.

For an application (such as a compositor) where you're not changing
show / shot / sequence at runtime the use of envvars is no big deal.
For example, say you have a per-shot display lut
(/job/$SHOT/.../shotgrade.3dl).   You could run a script which sets up
the shot environment, launch the APP, and everything just works.

However, consider an image-viewing tool which needs to flipbook
multiple shots (or whole sequences) back to back. I.e., the display
transform, as queried from OCIO, would be different based on the shot
name. Would this approach be possible using an envvar mechanism?  I
dont believe so.   Envvars are commonly locked down at launch time.
And, even if you could modify it at runtime as playback was in
progress, would you want to? How would you notifiy OCIO that it had
changed?  This seems like a tricky implementation, and strikes me as
going down the path of evil.

I would prefer to promote this concept of 'variables' to be an
explicit communications mechanism, rather then using envvars as a
back-door.  Alternate implementation options include:

* a config->setVariable option,  where variables were available for
use in color transforms.  (They could even retain the $SHOW $SHOT
syntax).  This way the OCIO plugin could set it as needed, and with
the python API exposed in client apps the list of variables could even
be amended with facility code, as needed at runtime!

* Alternatively,  with a less stateful implementation the variables /
arguments could be passed along in the config->getProcessor call. This
may be preferable to those computer scientists among us.

Either way, this passing of variables as arguments to a transforms is
half of the work towards allowing fully dynamic colorspaces. (The
other 1/2 being the plugin API to define the lists of transforms at
runtime).

Perhaps this is good reason to move forward on both ASAP?

-- Jeremy


Kevin Wheatley <kevin.j....@...>
 

When I looked at trying to use per-shot grades last, I ended up with the limitation that you couldn't dynamically determine the representation which meant you could not have say a 3D LUT for one shot and a CDL for another.

Something that I did manage to make work was a layering approach allowing a top level look to be published, followed by more specific sequence and under that shot grades, although that didn't scale well it involved well structuring paths and names of files and using the search path capability to find the right one - needless to say this was a very twisted approach to achieving this - it didn't handle multiple versions of looks/grades, which combined with the fact that vendor supplied applications don't necessarily fully expose the OCIO capabilities meant I gave up and did something  less twisted.

As for per shot variables with edited timelines, I was utilizing the config overrides to replace environment variable settings, this worked great in Nuke and other similar applications where this is exposed, but lots of tools don't expose the feature for instance heiro/heiroplayer would be a suitable place to do this.

Kevin