parsing configs with non-C locale


Paul Miller <pa...@...>
 

I think I found a bug in either my code or OCIO. When our software is run on a Ubuntu 12.04 machine with a German locale, parsing of srgb.spi1d fails and we get an "Invalid 'From' Tag" exception. I think the sscanf() there is attempting to parse the string using the German locale. Should I be forcing the C locale before calling into OCIO, or should OCIO be doing this itself?


Jeremy Selan <jeremy...@...>
 

Hm...

I have to admit I dont have much experience with locales. Can someone
with experience on such issues please comment? On first glance, it
feels like it should certainly be fixed inside OCIO, but perhaps I am
mistaken?

-- Jeremy

On Thu, Nov 15, 2012 at 7:01 AM, Paul Miller <pa...@fxtech.com> wrote:
I think I found a bug in either my code or OCIO. When our software is run on
a Ubuntu 12.04 machine with a German locale, parsing of srgb.spi1d fails and
we get an "Invalid 'From' Tag" exception. I think the sscanf() there is
attempting to parse the string using the German locale. Should I be forcing
the C locale before calling into OCIO, or should OCIO be doing this itself?


dbr/Ben <dbr....@...>
 

I've not had much experience with locales either, but it seemed like an interesting thing to read up on.. so..

With LANG=de_DE, sscanf expects to see floats like "0,12" (compared to 0.12 as required in the LUT file)

Doesn't happen often because the locale defaults to "C", and the user's environment is ignored unless setlocale() has been called elsewhere in the process:


I think OCIO should definitely set the locale to "C" temporarily while parsing/writing files, otherwise it might end up with invalid LUT's that happen to work for the current user but no-one else..

Based on this page..
..I think the tidiest way to fix this would be:

- Replace the few usages of the C string-parsing methods (sscanf/atof/etc) with the C++ stdlib equivalents (the *stream stuff)
- Modify FileTransform.cpp:GetFile to imbue the filestream with the "C" locale (as described in the above apache.org link)
- Same imbue'ing wherever else std::ofstream or std::ifstream is used (not many places - main ones are FileTransform, ociobakelut, and the CDLTransform)

I think that should make the reading/writing of files locale-independant, but there might still be other issues..? (e.g the stringstream used to construct exception messages)

Python has a kind of similar problem, seems like their approach was to write their own atof/stdtod methods, but I don't think this will work for OCIO, since it mainly uses the C++ *stream stuff..

On 17/11/2012, at 2:46 AM, Jeremy Selan wrote:

Hm...

I have to admit I dont have much experience with locales.  Can someone
with experience on such issues please comment?  On first glance, it
feels like it should certainly be fixed inside OCIO, but perhaps I am
mistaken?

-- Jeremy

On Thu, Nov 15, 2012 at 7:01 AM, Paul Miller <pa...@...> wrote:
I think I found a bug in either my code or OCIO. When our software is run on
a Ubuntu 12.04 machine with a German locale, parsing of srgb.spi1d fails and
we get an "Invalid 'From' Tag" exception. I think the sscanf() there is
attempting to parse the string using the German locale. Should I be forcing
the C locale before calling into OCIO, or should OCIO be doing this itself?


Dithermaster <dither...@...>
 

Ben's description aligns with my own experiences. We had a similar bug reading/writing a text file with decimals, and fixed it in a similar way (by saving/setting/restoring the locale setting).

///d@


On Nov 19, 2012, at 6:54 AM, dbr/Ben <dbr....@...> wrote:

I've not had much experience with locales either, but it seemed like an interesting thing to read up on.. so..

With LANG=de_DE, sscanf expects to see floats like "0,12" (compared to 0.12 as required in the LUT file)

Doesn't happen often because the locale defaults to "C", and the user's environment is ignored unless setlocale() has been called elsewhere in the process:


I think OCIO should definitely set the locale to "C" temporarily while parsing/writing files, otherwise it might end up with invalid LUT's that happen to work for the current user but no-one else..

Based on this page..
..I think the tidiest way to fix this would be:

- Replace the few usages of the C string-parsing methods (sscanf/atof/etc) with the C++ stdlib equivalents (the *stream stuff)
- Modify FileTransform.cpp:GetFile to imbue the filestream with the "C" locale (as described in the above apache.org link)
- Same imbue'ing wherever else std::ofstream or std::ifstream is used (not many places - main ones are FileTransform, ociobakelut, and the CDLTransform)

I think that should make the reading/writing of files locale-independant, but there might still be other issues..? (e.g the stringstream used to construct exception messages)

Python has a kind of similar problem, seems like their approach was to write their own atof/stdtod methods, but I don't think this will work for OCIO, since it mainly uses the C++ *stream stuff..

On 17/11/2012, at 2:46 AM, Jeremy Selan wrote:

Hm...

I have to admit I dont have much experience with locales.  Can someone
with experience on such issues please comment?  On first glance, it
feels like it should certainly be fixed inside OCIO, but perhaps I am
mistaken?

-- Jeremy

On Thu, Nov 15, 2012 at 7:01 AM, Paul Miller <pa...@...> wrote:
I think I found a bug in either my code or OCIO. When our software is run on
a Ubuntu 12.04 machine with a German locale, parsing of srgb.spi1d fails and
we get an "Invalid 'From' Tag" exception. I think the sscanf() there is
attempting to parse the string using the German locale. Should I be forcing
the C locale before calling into OCIO, or should OCIO be doing this itself?


Paul Miller <pa...@...>
 

On 11/19/2012 8:28 AM, Dithermaster wrote:
Ben's description aligns with my own experiences. We had a similar bug
reading/writing a text file with decimals, and fixed it in a similar way
(by saving/setting/restoring the locale setting).
Same here. But in that case, rather than fiddling with the locale (I wasn't sure of any unintended consequences of doing that), I swapped in a custom sscanf() that assumed decimal separators.

Still, I'll manually swap into the C locale when using OCIO in my imminent update, but I agree it should probably be done inside OCIO at some point. One less implementation detail to worry about.


Jeremy Selan <jeremy...@...>
 

I've opened up an issue on github so we dont forget about fixing this.
http://github.com/imageworks/OpenColorIO/issues/297

Anyone up for taking a stab at implementing the C-Locale fix?

-- Jeremy

On Mon, Nov 19, 2012 at 6:43 AM, Paul Miller <pa...@fxtech.com> wrote:
On 11/19/2012 8:28 AM, Dithermaster wrote:

Ben's description aligns with my own experiences. We had a similar bug
reading/writing a text file with decimals, and fixed it in a similar way
(by saving/setting/restoring the locale setting).

Same here. But in that case, rather than fiddling with the locale (I wasn't
sure of any unintended consequences of doing that), I swapped in a custom
sscanf() that assumed decimal separators.

Still, I'll manually swap into the C locale when using OCIO in my imminent
update, but I agree it should probably be done inside OCIO at some point.
One less implementation detail to worry about.


Paul Miller <pa...@...>
 

On 12/11/2012 6:09 PM, Jeremy Selan wrote:
I've opened up an issue on github so we dont forget about fixing this.
http://github.com/imageworks/OpenColorIO/issues/297

Anyone up for taking a stab at implementing the C-Locale fix?
I already have a fix. But I've never committed anything to github. If someone will walk me through the process, I'll be happy to commit my fix.


Jeremy Selan <jeremy...@...>
 

Is this your first stab at using git as well?

We've the very few first steps outlined on the ocio website,
http://opencolorio.org/developers/getting_started.html

The rough steps are:

make a github account, get the permissions working
fork OpenColorIO to your private repo
clone your private repo
add spi as a remote
checkout a new branch for your development task
modify code, build, test
commit code changes
push commit(s) to github
from the github website, submit a pull request for the developement branch

If this is too daunting, please feel free to just send me either a
patch or the update files; I'll be happy to submit a pull request on
my branch.

-- Jeremy

On Tue, Dec 11, 2012 at 5:24 PM, Paul Miller <pa...@fxtech.com> wrote:
I already have a fix. But I've never committed anything to github. If
someone will walk me through the process, I'll be happy to commit my fix.


Paul Miller <pa...@...>
 

On 12/11/2012 7:43 PM, Jeremy Selan wrote:
Is this your first stab at using git as well?
I use git here but not as part of a public project.

We've the very few first steps outlined on the ocio website,
http://opencolorio.org/developers/getting_started.html
Oh - I hadn't seen that. Yeah I'll take a look - thanks for the pointer.

The rough steps are:

make a github account, get the permissions working
fork OpenColorIO to your private repo
clone your private repo
add spi as a remote
checkout a new branch for your development task
modify code, build, test
commit code changes
push commit(s) to github
from the github website, submit a pull request for the developement branch

If this is too daunting, please feel free to just send me either a
patch or the update files; I'll be happy to submit a pull request on
my branch.
It's pretty daunting but probably well worth me trying it out. I'll give her the old college try.

Thanks!

-Paul