Date   

Re: Should OSL read non-texture files?

Larry Gritz
 

What use case did you have in mind? What type of data and how much are you imagining is a typical case? Upper bound?

Are tables all one data type? Only float/int, or also string?


On May 7, 2020, at 5:55 PM, Mitch Prater <mprater@...> wrote:

Not sure my answer went to the right place...

1-3 are what I'd like to avoid. I don't think the dict() functions are implemented in prman (although I haven't tested that).

Yes, your table() suggestion is much more like what I'm after.
_._,_._,_

--
Larry Gritz





Re: Should OSL read non-texture files?

Stephen Friedman
 

The dict() functions are internal to OSL and I don't think they require support form the renderer, and a quick code inspection in our SIMD branch of the code in prman also looks like it supports them as well.  That being said, I'm not aware of wide usage of them, so there may be some rough edges.  If you give it a try, and it doesn't work out well, it would be at least good to get that concrete example of why it doesn't work well for your case.

If we were to support something like table() OSL, one of my main concerns would be keeping it scalable, and so would want it to have these properties
 a) support a limited pre-defined set of data layouts/file formats
 b) the data layouts should be randomly accessible (facilitate caching/sparse reading)
 c) the shadeop should support a search-path abstraction to allow portability to the renderfarm/cloud

In particular, b) coupled with being able to have rows/columns support different data types mean you need a reasonable descriptor in the header to allow for directly seeking to the right place in the file.  

Also, if lookups can be either 1D or 2D  its worth considering throwing in 3D...

--Stephen


Upcoming Events #cal-summary

osl-dev@lists.aswf.io Calendar <osl-dev@...>
 

Open Shading Language discussion list Upcoming Events

OSL TSC meeting ( every other week )

When:
Thursday, 14 May 2020, 2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer: Chris Kulla ckulla@...

Details:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR

View Event


OSL TSC meeting ( every other week )

When:
Thursday, 28 May 2020, 2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer: Chris Kulla ckulla@...

Details:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR

View Event


OSL TSC meeting ( every other week )

When:
Thursday, 11 June 2020, 2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer: Chris Kulla ckulla@...

Details:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR

View Event


OSL TSC meeting ( every other week )

When:
Thursday, 25 June 2020, 2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer: Chris Kulla ckulla@...

Details:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR

View Event


OSL TSC meeting ( every other week )

When:
Thursday, 9 July 2020, 2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer: Chris Kulla ckulla@...

Details:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR

View Event


Re: Should OSL read non-texture files?

Mitch Prater
 

Not sure my answer went to the right place...

1-3 are what I'd like to avoid. I don't think the dict() functions are implemented in prman (although I haven't tested that).

Yes, your table() suggestion is much more like what I'm after.


Re: Should OSL read non-texture files?

Larry Gritz
 

Yeah, I've thought about that as well.

There are currently several ways to accomplish this, none of which are perfect or suited to all uses:

1. An array parameter:

    shader foo (float data[] = {0})
    ...
    float val = data[i];

  which the system can pass as an "instance parameter" at runtime (and will still get turned into a constant data array during runtime optimization of the shader).

2. Cram it into a 1D texture whose width is nvals, and look it up with interpolation disabled:

    val = texture("data.exr", (i+0.5)/nvals,, 0.5, "width", 0, "interp", "closest");

3. Put the values into a pointcloud format, read with pointcloud_search()/pointcloud_get().

4. Cram it into xml in such a way that you could use dict_find()/dict_value().

They each have their pros and cons, depending on the type of data, amount of data (15 values for a table is a different than a million entries), whether the data needs to be ordered, and the domain (is it inherently 1D index, 2D, sparse, or what?).

But at the end of the day, I think it could be useful to have a dedicated function that is a little like texture, but with integer indices and a 1D variant (which texture lacks) and data types that aren't restricted to float & color.

    val = table("data1.txt", i);
    coeff = table("data2.csv", row, column);

Presumably it would know how to read several interesting formats. And unlike texture, all rows/columns would not need to contain the same data type (though it would presumably be an error to assign to a variable of a type that could not accept a value of the data in that column or row).

Is that the kind of thing you had in mind?



On May 7, 2020, at 2:12 PM, Mitch Prater <mprater@...> wrote:

Hi All,

There are any number of scenarios in which a pattern shader might what to access some type of data or configuration from an external file that isn't an image. For example, I have a cloth shader plugin that reads an ascii text file that defines the warp/weft "pull" pattern for the cloth weave. This provides an easy means for a user to specify and edit the weave pattern(s), and makes it easy for the shader to read the weave pattern(s) and cache them for use during rendering. Using an image file for this purpose is far from optimal, both from the user's standpoint and probably also from a shader execution standpoint.

I'm not advocating for full c-style file i/o. But a simple form of external data input would be very useful.

mitch

--
Larry Gritz





Should OSL read non-texture files?

Mitch Prater
 

Hi All,

There are any number of scenarios in which a pattern shader might what to access some type of data or configuration from an external file that isn't an image. For example, I have a cloth shader plugin that reads an ascii text file that defines the warp/weft "pull" pattern for the cloth weave. This provides an easy means for a user to specify and edit the weave pattern(s), and makes it easy for the shader to read the weave pattern(s) and cache them for use during rendering. Using an image file for this purpose is far from optimal, both from the user's standpoint and probably also from a shader execution standpoint.

I'm not advocating for full c-style file i/o. But a simple form of external data input would be very useful.

mitch


Upcoming Event: OSL TSC meeting ( every other week ) - Thu, 05/14/2020 2:00pm-3:00pm #cal-reminder

osl-dev@lists.aswf.io Calendar <osl-dev@...>
 

Reminder: OSL TSC meeting ( every other week )

When: Thursday, 14 May 2020, 2:00pm to 3:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/100511909

View Event

Organizer: Chris Kulla ckulla@...

Description:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR


Upcoming Events #cal-summary

osl-dev@lists.aswf.io Calendar <osl-dev@...>
 

Open Shading Language discussion list Upcoming Events

OSL TSC meeting ( every other week )

When:
Thursday, 14 May 2020, 2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer: Chris Kulla ckulla@...

Details:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR

View Event


OSL TSC meeting ( every other week )

When:
Thursday, 28 May 2020, 2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer: Chris Kulla ckulla@...

Details:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR

View Event


Re: pointcloud.cpp -Werror=sign-compare

Mitch Prater <mitchpr...@...>
 

Sounds about right.


On Thursday, April 30, 2020 at 1:41:58 PM UTC-7, Larry Gritz wrote:
My bad.

That "size_t" I believe should be "int".

I'm not sure how that slipped through the CI.  Oh, maybe the CI does not have partio installed, and skipped this section?

-- lg


On Apr 30, 2020, at 1:19 PM, Mitch Prater <mit...@...> wrote:

I'm getting this compiler error from the current master:

/home/mprater/osl/src/liboslexec/pointcloud.cpp: In member function ‘virtual int OSL_v1_11::RendererServices::pointcloud_get(OSL_v1_11::ShaderGlobals*, OpenImageIO_v2_0::ustring, size_t*, int, OpenImageIO_v2_0::ustring, OpenImageIO_v2_0::TypeDesc, void*)’:
/home/mprater/osl/src/liboslexec/pointcloud.cpp:379:30: error: comparison of integer expressions of different signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘int’ [-Werror=sign-compare]
         for (size_t i = 0; i < count; ++i) {
                            ~~^~~~~~~
Looks like the change to pointcloud.cpp happened between 16 April and 21 April commits. It's a chunk of code, so I'm not sure about touching it myself.

mitch


--
Larry Gritz





OSL TSC meeting ( every other week ) - Thu, 04/30/2020 #cal-notice

osl-dev@lists.aswf.io Calendar <noreply@...>
 

OSL TSC meeting ( every other week )

When:
Thursday, 30 April 2020
2:00pm to 3:00pm
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/100511909

Organizer:
ckulla@...

Description:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR


Upcoming Event: OSL TSC meeting ( every other week ) - Thu, 04/30/2020 2:00pm-3:00pm #cal-reminder

osl-dev@lists.aswf.io Calendar <osl-dev@...>
 

Reminder: OSL TSC meeting ( every other week )

When: Thursday, 30 April 2020, 2:00pm to 3:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/100511909

View Event

Organizer: Chris Kulla ckulla@...

Description:

Every other week meeting of the OSL TSC.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR


Re: pointcloud.cpp -Werror=sign-compare

Larry Gritz <l...@...>
 

My bad.

That "size_t" I believe should be "int".

I'm not sure how that slipped through the CI.  Oh, maybe the CI does not have partio installed, and skipped this section?

-- lg


On Apr 30, 2020, at 1:19 PM, Mitch Prater <mitchpr...@...> wrote:

I'm getting this compiler error from the current master:

/home/mprater/osl/src/liboslexec/pointcloud.cpp: In member function ‘virtual int OSL_v1_11::RendererServices::pointcloud_get(OSL_v1_11::ShaderGlobals*, OpenImageIO_v2_0::ustring, size_t*, int, OpenImageIO_v2_0::ustring, OpenImageIO_v2_0::TypeDesc, void*)’:
/home/mprater/osl/src/liboslexec/pointcloud.cpp:379:30: error: comparison of integer expressions of different signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘int’ [-Werror=sign-compare]
         for (size_t i = 0; i < count; ++i) {
                            ~~^~~~~~~
Looks like the change to pointcloud.cpp happened between 16 April and 21 April commits. It's a chunk of code, so I'm not sure about touching it myself.

mitch


--
Larry Gritz





pointcloud.cpp -Werror=sign-compare

Mitch Prater <mitchpr...@...>
 

I'm getting this compiler error from the current master:

/home/mprater/osl/src/liboslexec/pointcloud.cpp: In member function ‘virtual int OSL_v1_11::RendererServices::pointcloud_get(OSL_v1_11::ShaderGlobals*, OpenImageIO_v2_0::ustring, size_t*, int, OpenImageIO_v2_0::ustring, OpenImageIO_v2_0::TypeDesc, void*)’:
/home/mprater/osl/src/liboslexec/pointcloud.cpp:379:30: error: comparison of integer expressions of different signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘int’ [-Werror=sign-compare]
         for (size_t i = 0; i < count; ++i) {
                            ~~^~~~~~~
Looks like the change to pointcloud.cpp happened between 16 April and 21 April commits. It's a chunk of code, so I'm not sure about touching it myself.

mitch


Upcoming Event: OSL TSC meeting ( every other week ) - Thu, 04/30/2020 2:00pm-3:00pm #cal-reminder

osl-dev@lists.aswf.io Calendar <osl-dev@...>
 

Reminder: OSL TSC meeting ( every other week )

When: Thursday, 30 April 2020, 2:00pm to 3:00pm, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/100511909

View Event

Organizer: Chris Kulla ckulla@...

Description:

Every other week meeting of the OSL TSC. Agenda available at https://github.com/imageworks/OpenShadingLanguage.

Meeting Agenda / Notes: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I/

Confirm this meeting invite is still valid by finding the meeting at https://lists.aswf.io/calendar.

Join Zoom Meeting https://zoom.us/j/100511909

Meeting ID: 100 511 909

One tap mobile +16465588656,,100511909# US (New York) +13126266799,,100511909# US (Chicago)

Dial by your location +1 646 558 8656 US (New York) +1 312 626 6799 US (Chicago) +1 669 900 6833 US (San Jose) +1 253 215 8782 US +1 301 715 8592 US +1 346 248 7799 US (Houston) 877 369 0926 US Toll-free 855 880 1246 US Toll-free +1 587 328 1099 Canada +1 647 374 4685 Canada +1 647 558 0588 Canada +1 778 907 2071 Canada +1 438 809 7799 Canada 855 703 8985 Canada Toll-free Meeting ID: 100 511 909 Find your local number: https://zoom.us/u/acBVrM6HWR


Re: [osl-dev] Is it time for LLVM minimum bump?

Larry Gritz
 

I believe that line with the error has been fixed in the current master.


On Apr 30, 2020, at 9:36 AM, Etienne Sandré-Chardonnal <etienne.sandre@...> wrote:

Hi,

I found a similar workaround, setting LLVM_CONFIG to /usr/bin/llvm-config-10 or whatever llvm-config version.
I also figured out that you need to use c++14 in order to compile with LLVM10 by passing CXX_STANDARD=14

Now I get many errors due to clang-10 such as:
> error: implicit conversion from 'unsigned int' to 'float' changes value from 4294967295 to 4294967296
> >     return bits * (1.0f / std::numeric_limits<unsigned int>::max());

It seems that this -Wimplicit-int-float-conversion is a new feature of clang 10. And since OSL uses -Werror -Wall it won't compile with clang 10.

I think I'll stick to LLVM9 / clang 9 for this time... Since I upgrade from 6 it's still a big step :)

Cheers




Le jeu. 30 avr. 2020 à 18:22, Thiago Ize <thiago.ize@...> a écrit :
Did you try setting the osl's Makefile LLVM_DIRECTORY to point to your llvm 10?

On Thu, Apr 30, 2020 at 9:34 AM Etienne Sandré-Chardonnal <etienne.sandre@...> wrote:
Hi,

I just figured out I am using LLVM 6.0

The reason is dumb : when building under ubuntu, LLVM is not detected when configuring the project with cmake, with any version above 6.0

I get the following error message:

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find LLVM: Found unsuitable version "", but required is at least
  "6.0" (found )
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:391 (_FPHSA_FAILURE_MESSAGE)
  src/cmake/modules/FindLLVM.cmake:108 (find_package_handle_standard_args)
  src/cmake/externalpackages.cmake:89 (find_package)
  src/cmake/externalpackages.cmake:208 (checked_find_package)
  CMakeLists.txt:119 (include)

If I do apt install llvm-10 llvm-10-dev I get:
llvm-10 is already the newest version (1:10.0.0-4ubuntu1).
llvm-10-dev is already the newest version (1:10.0.0-4ubuntu1).

The only version that gets properly detected by the cmake run is 6.0

Any idea? I would love using a more recent LLVM version.

Cheers!



Le mar. 31 mars 2020 à 01:31, Larry Gritz <lg@...> a écrit :
Now that LLVM 10 is out, does anybody have strong feelings about what our minimum should be for OSL?

(I'm talking strictly about master. We never take away dependency support from a release branch.)

Currently we support LLVM 6, 7, 8, 9, 10. Five releases to continue support and test seems excessive. Can we at least bump to 7 minimum? Does anybody even need that?

So let's say, if you still require LLVM < 8, AND you will need to build OSL next/master with it, can you please chime in here or privately and let me know what version you are using and how long you expect to be prevented from upgrading to at least LLVM 8?

--
Larry Gritz
lg@...




-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/08144BAD-18F6-4690-AE6B-C4035F362C73%40larrygritz.com.



--
Larry Gritz





Re: [osl-dev] Is it time for LLVM minimum bump?

Etienne Sandré-Chardonnal
 

Hi,

I found a similar workaround, setting LLVM_CONFIG to /usr/bin/llvm-config-10 or whatever llvm-config version.
I also figured out that you need to use c++14 in order to compile with LLVM10 by passing CXX_STANDARD=14

Now I get many errors due to clang-10 such as:
> error: implicit conversion from 'unsigned int' to 'float' changes value from 4294967295 to 4294967296
> >     return bits * (1.0f / std::numeric_limits<unsigned int>::max());

It seems that this -Wimplicit-int-float-conversion is a new feature of clang 10. And since OSL uses -Werror -Wall it won't compile with clang 10.

I think I'll stick to LLVM9 / clang 9 for this time... Since I upgrade from 6 it's still a big step :)

Cheers




Le jeu. 30 avr. 2020 à 18:22, Thiago Ize <thiago.ize@...> a écrit :
Did you try setting the osl's Makefile LLVM_DIRECTORY to point to your llvm 10?

On Thu, Apr 30, 2020 at 9:34 AM Etienne Sandré-Chardonnal <etienne.sandre@...> wrote:
Hi,

I just figured out I am using LLVM 6.0

The reason is dumb : when building under ubuntu, LLVM is not detected when configuring the project with cmake, with any version above 6.0

I get the following error message:

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find LLVM: Found unsuitable version "", but required is at least
  "6.0" (found )
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:391 (_FPHSA_FAILURE_MESSAGE)
  src/cmake/modules/FindLLVM.cmake:108 (find_package_handle_standard_args)
  src/cmake/externalpackages.cmake:89 (find_package)
  src/cmake/externalpackages.cmake:208 (checked_find_package)
  CMakeLists.txt:119 (include)

If I do apt install llvm-10 llvm-10-dev I get:
llvm-10 is already the newest version (1:10.0.0-4ubuntu1).
llvm-10-dev is already the newest version (1:10.0.0-4ubuntu1).

The only version that gets properly detected by the cmake run is 6.0

Any idea? I would love using a more recent LLVM version.

Cheers!



Le mar. 31 mars 2020 à 01:31, Larry Gritz <lg@...> a écrit :
Now that LLVM 10 is out, does anybody have strong feelings about what our minimum should be for OSL?

(I'm talking strictly about master. We never take away dependency support from a release branch.)

Currently we support LLVM 6, 7, 8, 9, 10. Five releases to continue support and test seems excessive. Can we at least bump to 7 minimum? Does anybody even need that?

So let's say, if you still require LLVM < 8, AND you will need to build OSL next/master with it, can you please chime in here or privately and let me know what version you are using and how long you expect to be prevented from upgrading to at least LLVM 8?

--
Larry Gritz
lg@...




--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/08144BAD-18F6-4690-AE6B-C4035F362C73%40larrygritz.com.


Re: [osl-dev] Is it time for LLVM minimum bump?

Thiago Ize
 

Did you try setting the osl's Makefile LLVM_DIRECTORY to point to your llvm 10?

On Thu, Apr 30, 2020 at 9:34 AM Etienne Sandré-Chardonnal <etienne.sandre@...> wrote:
Hi,

I just figured out I am using LLVM 6.0

The reason is dumb : when building under ubuntu, LLVM is not detected when configuring the project with cmake, with any version above 6.0

I get the following error message:

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find LLVM: Found unsuitable version "", but required is at least
  "6.0" (found )
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:391 (_FPHSA_FAILURE_MESSAGE)
  src/cmake/modules/FindLLVM.cmake:108 (find_package_handle_standard_args)
  src/cmake/externalpackages.cmake:89 (find_package)
  src/cmake/externalpackages.cmake:208 (checked_find_package)
  CMakeLists.txt:119 (include)

If I do apt install llvm-10 llvm-10-dev I get:
llvm-10 is already the newest version (1:10.0.0-4ubuntu1).
llvm-10-dev is already the newest version (1:10.0.0-4ubuntu1).

The only version that gets properly detected by the cmake run is 6.0

Any idea? I would love using a more recent LLVM version.

Cheers!



Le mar. 31 mars 2020 à 01:31, Larry Gritz <lg@...> a écrit :
Now that LLVM 10 is out, does anybody have strong feelings about what our minimum should be for OSL?

(I'm talking strictly about master. We never take away dependency support from a release branch.)

Currently we support LLVM 6, 7, 8, 9, 10. Five releases to continue support and test seems excessive. Can we at least bump to 7 minimum? Does anybody even need that?

So let's say, if you still require LLVM < 8, AND you will need to build OSL next/master with it, can you please chime in here or privately and let me know what version you are using and how long you expect to be prevented from upgrading to at least LLVM 8?

--
Larry Gritz
lg@...




--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/08144BAD-18F6-4690-AE6B-C4035F362C73%40larrygritz.com.


Re: [osl-dev] Is it time for LLVM minimum bump?

Etienne Sandré-Chardonnal
 

Hi,

I just figured out I am using LLVM 6.0

The reason is dumb : when building under ubuntu, LLVM is not detected when configuring the project with cmake, with any version above 6.0

I get the following error message:

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find LLVM: Found unsuitable version "", but required is at least
  "6.0" (found )
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:391 (_FPHSA_FAILURE_MESSAGE)
  src/cmake/modules/FindLLVM.cmake:108 (find_package_handle_standard_args)
  src/cmake/externalpackages.cmake:89 (find_package)
  src/cmake/externalpackages.cmake:208 (checked_find_package)
  CMakeLists.txt:119 (include)

If I do apt install llvm-10 llvm-10-dev I get:
llvm-10 is already the newest version (1:10.0.0-4ubuntu1).
llvm-10-dev is already the newest version (1:10.0.0-4ubuntu1).

The only version that gets properly detected by the cmake run is 6.0

Any idea? I would love using a more recent LLVM version.

Cheers!



Le mar. 31 mars 2020 à 01:31, Larry Gritz <lg@...> a écrit :
Now that LLVM 10 is out, does anybody have strong feelings about what our minimum should be for OSL?

(I'm talking strictly about master. We never take away dependency support from a release branch.)

Currently we support LLVM 6, 7, 8, 9, 10. Five releases to continue support and test seems excessive. Can we at least bump to 7 minimum? Does anybody even need that?

So let's say, if you still require LLVM < 8, AND you will need to build OSL next/master with it, can you please chime in here or privately and let me know what version you are using and how long you expect to be prevented from upgrading to at least LLVM 8?

--
Larry Gritz
lg@...




--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl-dev+unsubscribe@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/08144BAD-18F6-4690-AE6B-C4035F362C73%40larrygritz.com.


Re: Warning about llvm 10 on Mac

Larry Gritz <l...@...>
 

Workaround:  https://github.com/imageworks/OpenShadingLanguage/pull/1162


On Apr 28, 2020, at 10:31 PM, Larry Gritz <l...@...> wrote:

Thanks, Solomon! Your prods definitely pointed me in a fruitful direction. This has been a low level annoyance to me for at least a couple weeks now.

-- lg


On Apr 28, 2020, at 10:30 PM, Larry Gritz <l...@...> wrote:

What a mess!

But after some experimentation, it seems that it can all be solved (or at least be asymptomatic?) if I link to the llvm libs statically rather than dynamically. Luckily, I already had a way to tell my FindLLVM.cmake to prefer the static libs. So I think the solution I'll use for now is just ot always set that to true on OSX and force static linkage of libclang.


On Apr 28, 2020, at 7:26 PM, Solomon Boulos <solomo...@...> wrote:

And the follow up in the -core repo:

 

seems like they just gave up :). This recent and open one suggests badness for your LLVM 10 upgrade:


So I dunno, don’t install LLVM via homebrew? (Meaning just build it yourself locally)

On Tue, Apr 28, 2020 at 18:59 Solomon Boulos <solomo...@...> wrote:
This terrifying github issue:

  

says that libc++ is/was fairly broken (hardcoded rpath in the cmake build). I can’t see from a simple skim if homebrew gave up on it permanently, but obviously a lot can change in five-ish years.

Either way, I think that this implies you’re getting an old libc++ from your system when running with an empty DYLD path, and that whatever you need for this code to work is only in the newer one. (Which is a surprising behavioral difference, maybe an initializer of some sort that doesn’t break the ABI but does change behavior?)



On Tue, Apr 28, 2020 at 18:43 Larry Gritz <l...@...> wrote:
works:  oslc -IincA -IincB blah.osl
broken: DYLD_LIBRARY_PATH= oslc -IincA -IincB blah.osl

The only thing in my DYLD_LIBRARY_PATH was /usr/local/opt/llvm

DYLD_PRINT_LIBRARIES=1 is helpful

The broken one seems to pull in TWO different copies of libc++.1.dylib, one from /usr/local/opt/llvm (as expected) but also a second one from /usr/lib.

So I think... that maybe this is an rpath botch somewhere, or perhaps a sign that I should always try hard to link statically against the llvm components?


On Apr 28, 2020, at 3:55 PM, Larry Gritz <l...@...> wrote:

If /usr/local/Cellar/llvm/10.0.0_3 is in my DYLD_LIBRARY_PATH, it works. If not, it doesn't.

What I don't understand yet is why. If it can't find a library, why does it not fail, but instead have this incredibly weird behavior?
Still digging.


On Apr 28, 2020, at 2:31 PM, Larry Gritz <l...@...> wrote:

Aha!?

if I make sure my .bashrc contents are skipped, I can get the same behavior from a bare /bin/bash when I do 'oslc ...', no secondary shell needed. So.. something I'm setting up is somehow making this all work in my ordinary shell?

OK, will narrow it to the line. Stay tuned.


On Apr 28, 2020, at 2:25 PM, Solomon Boulos <bou...@...> wrote:

What's in your .bashrc and .bash_profile or .profile (or whatever your default shell reads from). What's curious is that assuming you ran /bin/bash manually like that it should inherit your environment variables from the interactive shell (while a direct invocation via subprocess.call would result in just executing the !login variants).

http://hayne.net/MacDev/Notes/unixFAQ.html#shellStartup has a good discussion, *except* it just dumps you out to the man page for bash for non-interactive shells. Which ... wow, has stuff I didn't know:

>     When bash is invoked as an interactive login shell, or as a  non-inter-
       active  shell with the --login option, it first reads and executes com-
       mands from the file /etc/profile, if that file exists.   After  reading
       that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile,
       in that order, and reads and executes commands from the first one  that
       exists  and  is  readable.  The --noprofile option may be used when the
       shell is started to inhibit this behavior.

       When an interactive login shell exits, or a non-interactive login shell
       executes  the  exit  builtin  command, bash reads and executes commands
       from the file ~/.bash_logout, if it exists.

       When an interactive shell that is not a login shell  is  started,  bash
       reads  and executes commands from ~/.bashrc, if that file exists.  This
       may be inhibited by using the --norc option.  The --rcfile file  option
       will  force  bash  to  read  and  execute commands from file instead of
       ~/.bashrc.

       When bash is started non-interactively, to  run  a  shell  script,  for
       example, it looks for the variable BASH_ENV in the environment, expands
       its value if it appears there, and uses the expanded value as the  name
       of  a  file to read and execute.  Bash behaves as if the following com-
       mand were executed:
              if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
       but the value of the PATH variable is not used to search for the  file-
       name.

       If  bash  is  invoked  with  the name sh, it tries to mimic the startup
       behavior of historical versions of sh as  closely  as  possible,  while
       conforming  to the POSIX standard as well.  When invoked as an interac-
       tive login shell, or a non-interactive shell with the  --login  option,
       it  first  attempts  to read and execute commands from /etc/profile and
       ~/.profile, in that order.  The  --noprofile  option  may  be  used  to
       inhibit  this  behavior.  When invoked as an interactive shell with the
       name sh, bash looks for the variable ENV, expands its value  if  it  is
       defined,  and uses the expanded value as the name of a file to read and
       execute.  Since a shell invoked as sh does not attempt to read and exe-
       cute  commands from any other startup files, the --rcfile option has no
       effect.  A non-interactive shell invoked with  the  name  sh  does  not
       attempt  to  read  any  other  startup files.  When invoked as sh, bash
       enters posix mode after the startup files are read.

       When bash is started in posix mode, as with the  --posix  command  line
       option, it follows the POSIX standard for startup files.  In this mode,
       interactive shells expand the ENV variable and commands  are  read  and
       executed  from  the  file  whose  name is the expanded value.  No other
       startup files are read.

On Tue, Apr 28, 2020 at 2:00 PM Larry Gritz <l...@...> wrote:
Ooh, you might be on to something!


$ oslc -IincA -IincB blah.osl
adding 'incA'        <-- my debug, shows that the '-IincA' was parsed on the command line
adding 'incB'        <-- my debug, shows that the '-IincB' was parsed on the command line
#include "..." search starts here:
#include <...> search starts here:
 incA             <-- LLVM diagnostics, shows I passed incA in includedirs list
 incB             <-- LLVM diagnostics, shows I passed incB in includedirs list
End of search list.
Compiled blah.osl -> blah.oso


$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
 incA
 incB
End of search list.
error: blah.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
#include "b.h"
         ^
FAILED blah.osl


Now let's try rebuilding all of OSL with llvm@9 from homebrew instead of llvm (which is llvm 10.0):

$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
 incA
 incB
End of search list.
Compiled blah.osl -> blah.oso


The mind reels. At least it's not python per se. But I don't have a working theory for what's going on here.



On Apr 28, 2020, at 11:50 AM, Solomon Boulos <bou...@...> wrote:

If you're not on Catalina, it's likely a false alarm. But for debugging, I meant to turn:

   subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)

into

   subprocess.call (['/bin/bash', 'oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)



On Mon, Apr 27, 2020 at 1:04 PM Larry Gritz <l...@...> wrote:
I'm not on Catalina yet.

I don't think it's my bashrc/etc, because I see the same problem on GitHub Actions CI for the Mac case.

I'm not quite sure I understand what you are suggesting in your second paragraph.

-- lg


On Apr 27, 2020, at 12:57 PM, Solomon Boulos <bou...@...> wrote:

Do you have stuff in your bashrc/similar? Assuming you upgraded to Catalina, there are a number of “ha ha, we break you” like bash => zsh, weird file permissions that control what programs can see what on your computer, etc.

As a simple test, instead of fixing it by running oslc as the binary with manual args, what happens with running bash explicitly and giving it those args instead?

On Mon, Apr 27, 2020 at 12:11 Larry Gritz <l...@...> wrote:
Ever since homebrew upgraded llvm to v10, I've been having some testsuite failures on Mac (both my laptop and GitHub CI) that basically boil down to the fact that in the testsuite, the oslc commands launched by runtest.py mysteriously only recognize the first directory in the include search path list, and therefore can fail to find some include files. This only happens when launched via runtest.py; the identical oslc command line run from the shell is fine.

Only happens on Mac. Only with llvm 10 Only when launched via python subprocess.call().

Here's the email I sent to the cfe-users list (dev mail list for the clang libraries, which we use for the preprocessor), just in case anybody here has a hypothesis about what's going on:

---

Excuse if this is a tricky explanation; I'm not sure I understand what's going on. 

I have a C-like language and compiler for which I use clang libraries to do the preprocessing. My compiler lets users specify `-I` directories for searchpaths for includes, per usual convention. I'm doing something like this:

   clang::HeaderSearchOptions &headerOpts = compilerinst.getHeaderSearchOpts();
   headerOpts.UseBuiltinIncludes = 0;
   headerOpts.UseStandardSystemIncludes = 0;
   headerOpts.UseStandardCXXIncludes = 0;
   for (auto&& inc : includepaths) {
       headerOpts.AddPath (inc, clang::frontend::Quoted,
                           false /* not a framework */,
                           true /* ignore sys root */);
   }


For the sake of a simple failure case, I have header a.h in directory incA/, and header b.h in incB/, and my test program just consists of

   #include "a.h"
   #include "b.h"

Also, I have set this to turn on some debugging:
   headerOpts.Verbose = 1; // DEBUGGING

Now, when I invoke my compiler from the command line, 

   oslc -IincA -IincB test.osl

I get this output:

   #include "..." search starts here:
   #include <...> search starts here:
    incA
    incB
   End of search list.

and my compile succeeds. As expected, and as it has for many many years.

But, as part of my compiler's test suite, there is a python script involved that boils down to:

   #!/usr/bin/env python
   import subprocess
   subprocess.call ('oslc -IincA -IincB test.osl', shell=True)

When I run the python program,

   python mytest.py

then I get this output:

   #include "..." search starts here:
   #include <...> search starts here:
    incA
    incB
   End of search list.
   error: test.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
   #include "b.h"
            ^
   FAILED test.osl

Wha? So I've poked around a bit with the behavior, and near as I can tell, even though the diagnostics say that both incA and incB are in the search list, it's only actually searching the first directory listed.

Now, this only happens on OSX, and only when I'm using clang 10 libraries (installed via Homebrew, though also when I build clang from scratch). Works fine on Linux. Works fine on all platforms for clang 9, 8, 7, 6, and I've been using this since back to 3.3 or so. Only had this problem after upgrading to clang/llvm 10, and only on OSX. Fails the same way for python 2.7 and 3.7.

If I change the subprocess.call to:

   subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)

it succeeds. (But in real life, this isn't an adequate workaround, because I want to use shell=True and keep the whole command line together, because it's really an arbitrary shell command that has output redirect.)

Does any of this ring a bell for anybody? Or does anyone have suggestions for what to try next?


--
Larry Gritz
l...@...




-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/F01DC5E9-CD9E-4D16-94A0-77DC0F1E537E%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2Ki3QiC2Z71b90xi1-%2B341tkV7QWtpnc7rZ3mgoH%3DyLQ1g%40mail.gmail.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/3B4C3231-74BF-4FE7-9AC4-40EEF4428888%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2KjhzmSRi2uzFdW1W1BX3SDERX-vrNPvd4sF3eu4HUX2KA%40mail.gmail.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/423D66A5-22F5-40BE-A0E7-BC5E78BEBD8B%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2KjtvehQTVg2xuJ85A2axHGNAHM9ujO%3DzojHjyhuNdqDOg%40mail.gmail.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/2400495B-5800-4A8D-ADAC-53C9C3661C9D%40larrygritz.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/EF6B4578-A930-4FDF-8C87-373D7608AD08%40larrygritz.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/07F2193E-3E2C-43B1-912F-DA28CD85B1F6%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2KiJkNvTukpxMMsmS6-ySnYWRKN7JS2R26%2Br4r1RE8MAmA%40mail.gmail.com.

--
Larry Gritz





--
Larry Gritz





--
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/7EBDDB80-2C3B-41B1-8F4E-442927CBB47B%40larrygritz.com.

--
Larry Gritz





Re: Warning about llvm 10 on Mac

Larry Gritz <l...@...>
 

Thanks, Solomon! Your prods definitely pointed me in a fruitful direction. This has been a low level annoyance to me for at least a couple weeks now.

-- lg


On Apr 28, 2020, at 10:30 PM, Larry Gritz <l...@...> wrote:

What a mess!

But after some experimentation, it seems that it can all be solved (or at least be asymptomatic?) if I link to the llvm libs statically rather than dynamically. Luckily, I already had a way to tell my FindLLVM.cmake to prefer the static libs. So I think the solution I'll use for now is just ot always set that to true on OSX and force static linkage of libclang.


On Apr 28, 2020, at 7:26 PM, Solomon Boulos <solomo...@...> wrote:

And the follow up in the -core repo:

 

seems like they just gave up :). This recent and open one suggests badness for your LLVM 10 upgrade:


So I dunno, don’t install LLVM via homebrew? (Meaning just build it yourself locally)

On Tue, Apr 28, 2020 at 18:59 Solomon Boulos <solomo...@...> wrote:
This terrifying github issue:

  

says that libc++ is/was fairly broken (hardcoded rpath in the cmake build). I can’t see from a simple skim if homebrew gave up on it permanently, but obviously a lot can change in five-ish years.

Either way, I think that this implies you’re getting an old libc++ from your system when running with an empty DYLD path, and that whatever you need for this code to work is only in the newer one. (Which is a surprising behavioral difference, maybe an initializer of some sort that doesn’t break the ABI but does change behavior?)



On Tue, Apr 28, 2020 at 18:43 Larry Gritz <l...@...> wrote:
works:  oslc -IincA -IincB blah.osl
broken: DYLD_LIBRARY_PATH= oslc -IincA -IincB blah.osl

The only thing in my DYLD_LIBRARY_PATH was /usr/local/opt/llvm

DYLD_PRINT_LIBRARIES=1 is helpful

The broken one seems to pull in TWO different copies of libc++.1.dylib, one from /usr/local/opt/llvm (as expected) but also a second one from /usr/lib.

So I think... that maybe this is an rpath botch somewhere, or perhaps a sign that I should always try hard to link statically against the llvm components?


On Apr 28, 2020, at 3:55 PM, Larry Gritz <l...@...> wrote:

If /usr/local/Cellar/llvm/10.0.0_3 is in my DYLD_LIBRARY_PATH, it works. If not, it doesn't.

What I don't understand yet is why. If it can't find a library, why does it not fail, but instead have this incredibly weird behavior?
Still digging.


On Apr 28, 2020, at 2:31 PM, Larry Gritz <l...@...> wrote:

Aha!?

if I make sure my .bashrc contents are skipped, I can get the same behavior from a bare /bin/bash when I do 'oslc ...', no secondary shell needed. So.. something I'm setting up is somehow making this all work in my ordinary shell?

OK, will narrow it to the line. Stay tuned.


On Apr 28, 2020, at 2:25 PM, Solomon Boulos <bou...@...> wrote:

What's in your .bashrc and .bash_profile or .profile (or whatever your default shell reads from). What's curious is that assuming you ran /bin/bash manually like that it should inherit your environment variables from the interactive shell (while a direct invocation via subprocess.call would result in just executing the !login variants).

http://hayne.net/MacDev/Notes/unixFAQ.html#shellStartup has a good discussion, *except* it just dumps you out to the man page for bash for non-interactive shells. Which ... wow, has stuff I didn't know:

>     When bash is invoked as an interactive login shell, or as a  non-inter-
       active  shell with the --login option, it first reads and executes com-
       mands from the file /etc/profile, if that file exists.   After  reading
       that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile,
       in that order, and reads and executes commands from the first one  that
       exists  and  is  readable.  The --noprofile option may be used when the
       shell is started to inhibit this behavior.

       When an interactive login shell exits, or a non-interactive login shell
       executes  the  exit  builtin  command, bash reads and executes commands
       from the file ~/.bash_logout, if it exists.

       When an interactive shell that is not a login shell  is  started,  bash
       reads  and executes commands from ~/.bashrc, if that file exists.  This
       may be inhibited by using the --norc option.  The --rcfile file  option
       will  force  bash  to  read  and  execute commands from file instead of
       ~/.bashrc.

       When bash is started non-interactively, to  run  a  shell  script,  for
       example, it looks for the variable BASH_ENV in the environment, expands
       its value if it appears there, and uses the expanded value as the  name
       of  a  file to read and execute.  Bash behaves as if the following com-
       mand were executed:
              if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
       but the value of the PATH variable is not used to search for the  file-
       name.

       If  bash  is  invoked  with  the name sh, it tries to mimic the startup
       behavior of historical versions of sh as  closely  as  possible,  while
       conforming  to the POSIX standard as well.  When invoked as an interac-
       tive login shell, or a non-interactive shell with the  --login  option,
       it  first  attempts  to read and execute commands from /etc/profile and
       ~/.profile, in that order.  The  --noprofile  option  may  be  used  to
       inhibit  this  behavior.  When invoked as an interactive shell with the
       name sh, bash looks for the variable ENV, expands its value  if  it  is
       defined,  and uses the expanded value as the name of a file to read and
       execute.  Since a shell invoked as sh does not attempt to read and exe-
       cute  commands from any other startup files, the --rcfile option has no
       effect.  A non-interactive shell invoked with  the  name  sh  does  not
       attempt  to  read  any  other  startup files.  When invoked as sh, bash
       enters posix mode after the startup files are read.

       When bash is started in posix mode, as with the  --posix  command  line
       option, it follows the POSIX standard for startup files.  In this mode,
       interactive shells expand the ENV variable and commands  are  read  and
       executed  from  the  file  whose  name is the expanded value.  No other
       startup files are read.

On Tue, Apr 28, 2020 at 2:00 PM Larry Gritz <l...@...> wrote:
Ooh, you might be on to something!


$ oslc -IincA -IincB blah.osl
adding 'incA'        <-- my debug, shows that the '-IincA' was parsed on the command line
adding 'incB'        <-- my debug, shows that the '-IincB' was parsed on the command line
#include "..." search starts here:
#include <...> search starts here:
 incA             <-- LLVM diagnostics, shows I passed incA in includedirs list
 incB             <-- LLVM diagnostics, shows I passed incB in includedirs list
End of search list.
Compiled blah.osl -> blah.oso


$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
 incA
 incB
End of search list.
error: blah.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
#include "b.h"
         ^
FAILED blah.osl


Now let's try rebuilding all of OSL with llvm@9 from homebrew instead of llvm (which is llvm 10.0):

$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
 incA
 incB
End of search list.
Compiled blah.osl -> blah.oso


The mind reels. At least it's not python per se. But I don't have a working theory for what's going on here.



On Apr 28, 2020, at 11:50 AM, Solomon Boulos <bou...@...> wrote:

If you're not on Catalina, it's likely a false alarm. But for debugging, I meant to turn:

   subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)

into

   subprocess.call (['/bin/bash', 'oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)



On Mon, Apr 27, 2020 at 1:04 PM Larry Gritz <l...@...> wrote:
I'm not on Catalina yet.

I don't think it's my bashrc/etc, because I see the same problem on GitHub Actions CI for the Mac case.

I'm not quite sure I understand what you are suggesting in your second paragraph.

-- lg


On Apr 27, 2020, at 12:57 PM, Solomon Boulos <bou...@...> wrote:

Do you have stuff in your bashrc/similar? Assuming you upgraded to Catalina, there are a number of “ha ha, we break you” like bash => zsh, weird file permissions that control what programs can see what on your computer, etc.

As a simple test, instead of fixing it by running oslc as the binary with manual args, what happens with running bash explicitly and giving it those args instead?

On Mon, Apr 27, 2020 at 12:11 Larry Gritz <l...@...> wrote:
Ever since homebrew upgraded llvm to v10, I've been having some testsuite failures on Mac (both my laptop and GitHub CI) that basically boil down to the fact that in the testsuite, the oslc commands launched by runtest.py mysteriously only recognize the first directory in the include search path list, and therefore can fail to find some include files. This only happens when launched via runtest.py; the identical oslc command line run from the shell is fine.

Only happens on Mac. Only with llvm 10 Only when launched via python subprocess.call().

Here's the email I sent to the cfe-users list (dev mail list for the clang libraries, which we use for the preprocessor), just in case anybody here has a hypothesis about what's going on:

---

Excuse if this is a tricky explanation; I'm not sure I understand what's going on. 

I have a C-like language and compiler for which I use clang libraries to do the preprocessing. My compiler lets users specify `-I` directories for searchpaths for includes, per usual convention. I'm doing something like this:

   clang::HeaderSearchOptions &headerOpts = compilerinst.getHeaderSearchOpts();
   headerOpts.UseBuiltinIncludes = 0;
   headerOpts.UseStandardSystemIncludes = 0;
   headerOpts.UseStandardCXXIncludes = 0;
   for (auto&& inc : includepaths) {
       headerOpts.AddPath (inc, clang::frontend::Quoted,
                           false /* not a framework */,
                           true /* ignore sys root */);
   }


For the sake of a simple failure case, I have header a.h in directory incA/, and header b.h in incB/, and my test program just consists of

   #include "a.h"
   #include "b.h"

Also, I have set this to turn on some debugging:
   headerOpts.Verbose = 1; // DEBUGGING

Now, when I invoke my compiler from the command line, 

   oslc -IincA -IincB test.osl

I get this output:

   #include "..." search starts here:
   #include <...> search starts here:
    incA
    incB
   End of search list.

and my compile succeeds. As expected, and as it has for many many years.

But, as part of my compiler's test suite, there is a python script involved that boils down to:

   #!/usr/bin/env python
   import subprocess
   subprocess.call ('oslc -IincA -IincB test.osl', shell=True)

When I run the python program,

   python mytest.py

then I get this output:

   #include "..." search starts here:
   #include <...> search starts here:
    incA
    incB
   End of search list.
   error: test.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
   #include "b.h"
            ^
   FAILED test.osl

Wha? So I've poked around a bit with the behavior, and near as I can tell, even though the diagnostics say that both incA and incB are in the search list, it's only actually searching the first directory listed.

Now, this only happens on OSX, and only when I'm using clang 10 libraries (installed via Homebrew, though also when I build clang from scratch). Works fine on Linux. Works fine on all platforms for clang 9, 8, 7, 6, and I've been using this since back to 3.3 or so. Only had this problem after upgrading to clang/llvm 10, and only on OSX. Fails the same way for python 2.7 and 3.7.

If I change the subprocess.call to:

   subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)

it succeeds. (But in real life, this isn't an adequate workaround, because I want to use shell=True and keep the whole command line together, because it's really an arbitrary shell command that has output redirect.)

Does any of this ring a bell for anybody? Or does anyone have suggestions for what to try next?


--
Larry Gritz
l...@...




-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/F01DC5E9-CD9E-4D16-94A0-77DC0F1E537E%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2Ki3QiC2Z71b90xi1-%2B341tkV7QWtpnc7rZ3mgoH%3DyLQ1g%40mail.gmail.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/3B4C3231-74BF-4FE7-9AC4-40EEF4428888%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2KjhzmSRi2uzFdW1W1BX3SDERX-vrNPvd4sF3eu4HUX2KA%40mail.gmail.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/423D66A5-22F5-40BE-A0E7-BC5E78BEBD8B%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2KjtvehQTVg2xuJ85A2axHGNAHM9ujO%3DzojHjyhuNdqDOg%40mail.gmail.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/2400495B-5800-4A8D-ADAC-53C9C3661C9D%40larrygritz.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/EF6B4578-A930-4FDF-8C87-373D7608AD08%40larrygritz.com.

--
Larry Gritz





-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/07F2193E-3E2C-43B1-912F-DA28CD85B1F6%40larrygritz.com.

-- 
You received this message because you are subscribed to the Google Groups "OSL Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osl...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/osl-dev/CANkN2KiJkNvTukpxMMsmS6-ySnYWRKN7JS2R26%2Br4r1RE8MAmA%40mail.gmail.com.

--
Larry Gritz





--
Larry Gritz




881 - 900 of 5005