|
3DL cube format.
bsloan <bsl...@...>
Regarding 3dl --
The lore I've been fed is that Mitch Bogdonowicz formerly of Kodak wrote (invented? burped?) the 3dl "specification" when he was at Kodak as a quick and dirty solution of 3D LUT ASCII serialization. Various interpretations have since been adopted (supported?) by Arri, Autodesk, Assimilate and others. Yes, the initial 1D monochrome shaper array is in some cases in a different bit depth from the 3D rgb array. What's more, various applications choke if the header comment fields (those preceded with a"#") are not 'just so' (eg. truelight's tl-utils). I guess the point is that there isn't just one spec for it. Maybe the way to deal with 3dl is to treat different interpretations as different formats: *arri.3dl *autodesk.3dl *scratch.3dl ugly -r- us. :) -blake |
|
|
|
.3DL format support
Jeremy Selan <jeremy...@...>
I'm fleshing out support for additional lut formats, and have a
question on 3d. There's a 1D shaper lut at the top of the file, and the 3D component further on, all of which use integer types. Simple enough. However, the range of the integers appears not to be explicitly specified. In the examples I've found thus far, all values appear to be either 10, 12, or 16 bit int. (and in a few cases the shaper lut and the 3d component do not appear to have the same bit depth). This means we'll need to be clever (bad clever, not good clever) in determining bit depth. GetBitDepth(maxValue): if maxValue > 32768: return 16 if maxValue > 2048: return 12 if maxValue > 512: return 10 This would mean for a max value [513,2047] -> 10 bits, [2048,32768] -> 12 bits, etc. (I'm allowing for a bit of overshoot in the luts, presuming 11 / 13 bit luts aren't allowed). The obvious downside (other than ugliness) is that depending on the thresholds, you wouldnt be able to make a really dark lut which didnt use the full range of output values. (Consider a 12 bit lut where you really did want to darken the image below 512). This is unlikely to happen in practice, but a bit unsettling. How do other people handle this issue? Should we only handle 10,12,16 bit luts, or all even bit depths >=8? |
|
|
|
Re: Lut format chart
Jeremy Selan <jeremy...@...>
An additional clarification...
toggle quoted message
Show quoted text
There does need need to be some end-user ui for image display. (I didnt want to imply no UI was necessary). You just dont need lut- directory / browser / setup interface). For a display application, the most common ui would be to have two popdowns. One would be the ColorSpace corresponding to image state. (Such as identifying the file as genesis log, film log, scene linear, etc). The other popdown would be to specify the output display profile (such as rec709, p3 dlp, srgb, etc). -- Jeremy On May 25, 3:42 pm, Jeremy Selan <jeremy...@...> wrote:
Exactly. Clients apps will automatically have support for all lut |
|
|
|
ftp site ready
Jeremy Selan <jeremy...@...>
We've now got an ftp site setup at spi for large file posting. If you
want to post files > 10MB email me and I'll send you the login info. |
|
|
|
Re: Lut format chart
Jeremy Selan <jeremy...@...>
Exactly. Clients apps will automatically have support for all lut
toggle quoted message
Show quoted text
formats. (It's about time!) To an end user, this should feel pretty automated. Apps shouldn't need ebd user options / dialogs, all of the lut-location mojo is internal to the ColorConfiguration file. Specifying which configuration to use is controlled via a single environment variable (which points at an ocs file). (Explicit initialization is also allowed, but this will probably be less common). The internals of the .ocs file are still in flux, but the rough idea is that there will be two flavors. One flavor will be a human- readable xml file, of which one element lut dir. The alternative will be a binary ocs 'package', where the lut dir is self contained in the file. (Think tgz of a config / lut dir). In the super common newbie-case, downloading a single .ocs binary file, and pointing to it with OCS_CONFIG will be all that's needed. -- Jeremy On May 25, 3:09 pm, Larry Gritz <l....@...> wrote:
Neat. So just to clarify, the big idea is that any app that uses your color library properly will, by extension, understand ALL of these luts (whichever happens to be available/used at the facility using the app)? |
|
|
|
Re: [ocs-dev] Lut format chart
Larry Gritz <l...@...>
Neat. So just to clarify, the big idea is that any app that uses your color library properly will, by extension, understand ALL of these luts (whichever happens to be available/used at the facility using the app)?
Just how automated can this be? I'm a little unclear on how much extra homework a display program needs to do. Will your library... um... just find the lut specs in whatever standard places they are stored and know how to correct images for proper display on that device? Are there standard places for the luts to be stored? Will an app need various user options/dialogs to set this up? Is this too much of a newbie question? On May 25, 2010, at 2:51 PM, Jeremy Selan wrote: Blake was kind enough to a google doc spreadsheet of lut formats:-- Larry Gritz l...@... |
|
|
|
Lut format chart
Jeremy Selan <jeremy...@...>
Blake was kind enough to a google doc spreadsheet of lut formats:
http://spreadsheets.google.com/ccc?key=0Avn2cZhY9jaYdF9qTHYteU1MUDYtdnFkSDJQdHpaU1E&hl=en I'll try to edit it to keep it up to date. (what formats we support, are planning to support, have examples of, etc). And eventually it'll probably turn into part of the ocs website. |
|
|
|
Re: Got Luts?
Jeremy Selan <jeremy...@...>
Thanks!
toggle quoted message
Show quoted text
I do have the houdini docs, but if you could send me known good examples luts that would be helpful as well. If the files are small (under 10 MB) just post them to this as files to this group. I am setting up an ftp site today, which will have a first code drop as well as allow folks to post larger luts / images as needed. -- Jeremy On May 25, 7:55 am, Mark Alexander <malex...@...> wrote:
I can send the specs for the Houdini .lut (ASCII) and .blut (bin) |
|
|
|
[ocs-dev] Re: Got Luts?
Mark Alexander <malex...@...>
I can send the specs for the Houdini .lut (ASCII) and .blut (bin)
formats, if you need that. They're also in the HDK documentation, if you have access to that. |
|
|
|
[ocs-dev] Re: Got Luts?
Jeremy Selan <jeremy...@...>
Ah, there is a 10MB per-file limit to uploads on google groups. Let
me whip up an alternative... On May 24, 9:15 am, Bruno Nicoletti <bruno.j....@...> wrote: 80 Megs or so of data seems to have confused Google. |
|
|
|
Re: [ocs-dev] Got Luts?
Bruno Nicoletti <bruno.j....@...>
80 Megs or so of data seems to have confused Google.
toggle quoted message
Show quoted text
Anyother way? b On 24 May 2010 10:59, Bruno Nicoletti <bruno.j....@...> wrote:
Hi Jeremy, --
Bruno Nicoletti |
|
|
|
Re: [ocs-dev] Got Luts?
Bruno Nicoletti <bruno.j....@...>
Hi Jeremy,
toggle quoted message
Show quoted text
these are the ones we support in Nuke... .vf Nuke native vector field vfz gzip compressed .vf files .cub(e) Truelight 1D or 3D LUT .cube Iridas LUT .csp Cinespace LUT .3dl Autodesk/Scratch LUT .blut Houdini mplay blut I can get you example files b On 21 May 2010 22:35, Jeremy Selan <jeremy...@...> wrote:
Hello! --
Bruno Nicoletti |
|
|
|
Re: [ocs-dev] Got Luts?
"Nathaniel Hoffman" <na...@...>
Jeremy,
toggle quoted message
Show quoted text
There are two LUT formats I know of that are used in game development. One is a 2D image format (any lossless image format will do - we've used BMP, TGA and PNG) with a 2D representation of the LUT where the planes along the 3rd axis have been placed next to each other. So a 32x32x32 LUT would turn into a 1024x32 2D image. A common usage is for an identity LUT in this format to be placed next to an ungraded screenshot, both manipulated in Photoshop or some other color manipulation package, and then the colorized LUT is extracted. The other format is a DDS (Microsoft DirectDraw Surface) file with a 3D texture in it, typically uncompressed. This is usually loaded directly into the game engine. These are both a bit ad-hoc and not really standardized, so I don't know if they are relevant for OCS. If they are, let me know and I can try to work up something more like an actual spec for each of these. Thanks, Naty Hello! |
|
|
|
[ocs-dev] Got Luts?
Jeremy Selan <jeremy...@...>
Hello!
I'm at the stage where I would like to get a survey of lut file formats (1d, 3d, matrix, etc) that folks actually use in the wild. If you commonly use a lut format at your facility, or define one in your software package, I would hugely appreciate it if you could upload example files and/or spec documents so I can begin work on the importers. Even if it's a proprietary format, I'd love to take a peek so I can get a sense of the scope of formats out in the wild. (I need to make sure the internal API is rich enough to encompass current use cases). Many formats have been mentioned previously, including: - Truelight ASCII .cub 3D lut - Assimilate Scratch (Arri /Kodak .3dl) - Iridas Framecycler (Iridas .cube) - Autodesk (MESH3D, .lut, etc.) For the majority of these, I do not have either example files or specifications. Please help! :) Also, does anyone know if the majority of lut formats identifiable by their extension? Are there common extension conflicts? Ideally, I'd like to try and have format reader registered based on file extension, and only if that fails give each lut loader a chance to read it. (Similar to how reader plugins work in nuke). (Note that I'm not assuming 1 file == 1 lut. (We will support readers where 1 file can generate multiple transforms, such as a 1d shaper lut -> 3d lut -> 1d shaper lut.)) |
|
|
|
[ocs-dev] Re: Foundry Questions
Jeremy Selan <jeremy...@...>
Sorry for the delay!
Progress on the library is going well, I hope to have a code drop ready in the next few weeks. To answer your latest few questions... These reference various .spiXXX files, are you going to open up theYes and no. We're going to open up the formats in the sense that the library will have a built-in reader support (The format details are incredibly uninteresting; just another brain-dead ascii text format). However, we're not going to push for anyone else to adopt them. Internally, once we get support for commercial lut formats we'll probably try to retire the spi formats. I'm not really familiar with how small houses (where someone is tasked only part time) deal with the issue of color management. Maybe we would ask some contacts at smaller houses for more detail about their 'off the shelf' color pipelines? I think this route would particularly be worth the effort, as the small houses (who dont have someone tasked full time to color) would have the most to gain in having this library succeed. -- Jeremy |
|
|
|
Re: [ocs-dev] Re: Foundry Questions
Bruno Nicoletti <bruno.j....@...>
Hi Jeremy,
thanks for the reply. A few more questions.... These reference various .spiXXX files, are you going to open up the- how do you representPlease see the file config_v0.ocs, you will see examples of all 3. format of those files? Little CMS was the one mumbled about here. Not sure how important this- any open source ones?What open source lut formats are you referring to? is, need to get back to our folks. By some sort of plugin method, you mentioned this in your reply withAre you looking at making it extensible?Not sure what you mean here. Anyone can add or create a custom respect to proprietary CMS systems. I'd simply say you couldn't process while manipulating the the configuration.What level of thread safety is there?The library is fully thread safe. (and efficient in a multi-thread We are playing with OpenCL and it might be useful to have that at some- have you thought about OpenCL code generation?Not yet. Are you interested in opengl? point. Not a high priority for us at the moment. Sorry, 'XFM' is my bizarre short hand for 'transform', but that stillAre you shipping it with any vanilla XFMs that you have created atWhat's an XFM? A configuration I presume? (im now calling them .ocs answers it. I suppose what we are missing are use cases of the library in practice. I have a pretty good guess as to how it would work at big film houses, film houses collaborating and film houses sharing work with smaller shops, but clear descriptions of that would be great. I guess my major concern is that if The Foundry is using this as the basis of a cross product CMS, we still need to make it work for the small shops in isolation, and they need tools to do at least the basic stuff that the big boys have in-house specialist do. So display calibration profiles and so on. Have you had any thoughts as to that? -- Bruno Nicoletti |
|
|
|
[ocs-dev] Re: Foundry Questions
Jeremy Selan <jeremy...@...>
Oh, and one thing to keep in mind; the current version of this library
already works great in Nuke. :) We've exposed 3 new nodes: ColorSpaceConvert / LogConvert / DisplayTransform. The 1st two are used in the comp graph natively, and the final one is used in the Input Process group to do filmlook conversions. We also have a custom read / write nodes that uses our color processing instead of the nuke-native processing. All of these plugins are planned on being open-sourced as well, so right off the bat we'll have a pretty good workflow example. (And clean example code on how to do plugin client integration). Obviously, native application support is the direction we're looking to head in the long term, but I wanted to clarify that direct manufacturer support is by no means required for Open Color Space to be useful at launch. -- Jeremy |
|
|
|
[ocs-dev] Re: Foundry Questions
Jeremy Selan <jeremy...@...>
On May 5, 2:30 pm, brunobignose <bruno.j....@...>
wrote: What is you internal representation?My discussion of the internals as a 'black' box are meant to be from the perspective of a simple client application, not from folks interested in color management or to 'authoring' apps. (Authoring apps being applications that can generate / introspect into color configurations). - what are the scope of the manipulations, are they all 3D ? Mixed 1D and 3D?The internal transformations currently include 1D luts / (normal + log allocation), matrix operations, gamma, 3D lut. They can be arbitrarily chained and used in the forward or inverse directions (with the exception of 3d luts, where it is left to use the user to provide an inverse). We also are adding a group transform, which will be useful in the API. - how do you representPlease see the file config_v0.ocs, you will see examples of all 3. ColorSpaces are ordered lists of atomic transforms (1d luts, mtx, gamma, 3d lut, etc). They can be used in any order as many times as necessary. DisplayDevices are just labels that define additional blocks of transforms. "Looks" are currently a bit in flux. If a look is constant across a show (even if there are multiple looks), it is typically rolled into the DisplayDevice block as this is more convenient. Examples of this would be show-wide 'Warm Look', or "DI Gamut check', etc. If a look is shot-varying we can use either a single 3dlut or asc color correction. (However, before release i expect this will get more general and allow for an arbitrary transform type). What third party LUTs do you manage?Currently the list is small, just some internal formats and .3dl files. But, we hope to support all commercial ascii lut formats before 1.0 - any open source ones?What open source lut formats are you referring to? - what happens if a user doesn't have a license for a proprietary lutIt is currently not obvious how we will add support for proprietary (encrypted) lut formats. Example, truelight. Some companies allow for the export of ascii luts, but others dont. One workaround would be to provide a lut plugin interface, so 3rd party color management companies could write OCS plugins that do the appropriate licence checks before running a lut transform. A plugin API is probably outside the scope of a 1.0 release, but could be added to 1.1 without effecting binary compatibility. How does one go about generating...Any ways you want. This is not a problem we're looking to solve for 1.0, and is treading deep into philosophical / 'secret sauce' issues. Internally at SPI, we have a bunch of raw measurement data (such as film stock profiles, camera exposure sweeps, etc) that guide this practice. You could make a whole class on this, I'm sure. There are also a bunch of different philosophies on how this should be done, many of which are equally valid. My gut feel is that if this project succeeds, we'll end up with a whole mailing list devoted to this topic. Please read the color space comments in this file, config_v0.ocs it provides a bit more info. - display profiles?Same as above. - looks?These are typically delivered by DI houses, and are director / DP guided. If a user is not provided a look by their client, they probably dont have to worry about one. - anything else in there...Remember that we'll provide a few good default configurations, but I expect all major facilities to define their own. And, we shouldnt dictate how they do this. The default configurations wont suck either, they're really what we use as the starting point for films. Are you looking at making it extensible?Not sure what you mean here. Anyone can add or create a custom colorspace configuration. Anyone can take one of the posted configurations and add new color spaces or devices to it. And the whole library will be open source so the internal details wont be secret. So,... "yes?" We hope to add the ability to export ICC profiles, for lut preview in tools such as photoshop. Other than that, they are not closely related. We currently have no intent to add support for reading ICC profiles. The library is fully thread safe. (and efficient in a multi-thread context). I am currently adding the color configuration mutable API, and am grappling with a few corner issues such as ' what if one thread manipulates the color configuration while another is processing pixels?' but this is a mental correctness issue, not a crashing issue. Color configurations specify whether each operation is invertible or not. An 'auto' mode is provided for types that are easy to perfectly invert, but we dont handle 'non-obvious' inversion cases. (i.e., 3d luts). Note that our 1D lut processing is really pretty clever (clever in a good way), and guaranteed to have perfect invertibility. For example, you can specify a 1d lut that goes from a low dynamic range allocation to an arbitrary high-dynamic range allocation, and still get perfect invertibility. We also have a validation script that prints out info about the current color configuration, this has a step thats walks through each colorspace and validates perfect invertibility (flagging otherwise). - can you flag invertible transforms in any way?Yes. Distributing colour management XML and LUTSYes, we're starting to think about it. Internally a directory works great but we're open to a single file as well. zip or tgz sound great, we have no preference. I'd probably do a timing check and pick whichever is faster to decompress live. - zip/tgz/voodoo?Only a single 3d lut is needed client side for the shader. The client is responsible for uploading to the card as necessary, though we provide a cacheid (cheap to compute) so it can be recomputed / uploaded as infrequently as possible. - have you thought about OpenCL code generation?Not yet. Are you interested in opengl? - we want to be able to supply our own names for objects in generatedYes, this is already part of the base api. please see the header. Note that sidefx has already pointed out that we're going to have to have finer granularity in our specification of glsl profiles (better glsl versions), this has yet to be added but will soon. On going supportIt's an open source project, so at a minimum I'm responsible for support. But my hope is that before 1.0 we develop a critical mass community of those interested who can all work on this 'as developer equals'. (multiple people having 'root' checkin privileges). I always get suspicious of open source projects where one person approves all checkins. My hope is that a few other reputable color folks will step up and we can all be responsible for validating new code checkins, etc. Of course, i'll be reading every line of code, but having more folks on board helps prevent 'hit by bus' scenarios. (or more likely, 'caught in a hard production crunch-time' scenario). Support on the artist / usage side will be open to a much broader community, just as nuke-users is. Are you shipping it with any vanilla XFMs that you have created atWhat's an XFM? A configuration I presume? (im now calling them .ocs files) Yes, we're gonna ship all input device profiles we've got, not just log/lin. The only camera we've never characterized is RED, so we'd have to leave that to someone else. (Or could adapt the nuke one for use in our environment). For device profiles the only ones we ever use are srgb, r709, and p3 dlp. These will be included.
|
|
|
|
[ocs-dev] Foundry Questions
brunobignose <bruno.j....@...>
Hi All,
this is a collection of questions that we came up with at The Foundry after sitting down with various product manager and techy types.... What is you internal representation? - we know you want to present this as a black box but we would like to know some technical details - what are the scope of the manipulations, are they all 3D ? Mixed 1D and 3D? - how do you represent - transforms - display devices - looks What third party LUTs do you manage? - any open source ones? - what happens if a user doesn't have a license for a proprietary lut library? How does one go about generating... - LUTs? - display profiles? - looks? - anything else in there... Are you looking at making it extensible? How does this all play with ICC proiles? What level of thread safety is there? Invertibility, - how guaranteed is that for the general colour transforms? - can you flag invertible transforms in any way? Distributing colour management XML and LUTS - have you thought of how to pack XML + LUTS + whatever into a single file for distribution? - zip/tgz/voodoo? OpenGL fragment shaders - how do we/you manage any LUTs that need to go along with the shaders you create? - have you thought about OpenCL code generation? - we want to be able to supply our own names for objects in generated code so as to avoid name collision, eg: the function name, On going support - who is responsible for support? - how do we manage folding in fixes? Are you shipping it with any vanilla XFMs that you have created at Sony? - and not just log/lin - device profiles? thanks Bruno NIcoletti |
|
|
|
New header, 0.5.1, now posted
Jeremy Selan <jeremy...@...>
Addresses Larry + Rods comments (mostly).
Coming soon, a mutable ColorConfig + better GLSL version-ness. (And after that, a library that builds) -- Subscription settings: http://groups.google.com/group/ocs-dev/subscribe?hl=en |
|
|