first post here, not sure if this is the correct list, please point me into the correct direction.
In the email I got from the list server, I can read that the email of the discussion ist is: osl-discussion@..., but an email to this list is returned with the hint that such a group does not exist..

I'm using Maya 2022 with Arnold 6.2 and try to write a mask shader.
The goal of the shader is to mask objects which are inside a closed surface.
The idea is to trace a ray in the normal direction (or a randomized normal) and check if the normal at the hitpoint points in the same direction or not. If it points in the same direction (dot product > 0) we are inside the mask geometry, if it points in the opposite direction, we are outside. The trace command is supported by Arnold as well as getting the Normals via getmessage().

shader tracemask(

    float maxDistance = 2000.0,

    output color outColor = color(0.0,0.0,0.0)

)

{

    normal hitN = normal(0, 1, 0);

    int hit = trace(P, N, "maxdist", maxDistance, "traceset", "mask");

    if (hit)

    {

        getmessage("trace", "N" , hitN);

        float weight = dot(hitN, N);

        if(weight>0.0)

        {

            outColor = color(1, 0, 0);

        }else{

            outColor = color(0, 1, 0);

        }

    } else {

        outColor = color(0, 0, 1);

    }

 

}

 

This can work in special cases as we can see in the first image where the green area is inside the einclosing sphere:


But with the normals something is wrong. Have a look at the second image:



Here the grid on the left side uses the shader. If rays are shoot in the direction of the normals, they should never hit anything because they are pointing to the left. But the green area shows that the rays hit the other surface.
And since the result is green, the hit normal seems to point to the direction of the tracing ray, not in the direction of the surface normal as visible in the image.

So I seem to miss some basic shading concepts, maybe anyone can point me into the correct direction.

The questions are:
1. Why does the traced ray hit anything at all if it is shoot in the direction of the normal?
2. Why does the normal of the hit point always point in the direction of the traced ray, not in the normal of the hit geometry?

Thanks for any help.

haggi