brdf | BRDF Parameters window , each BRDF has a colored background

Starting from the Introduction in the OSL spec,

OSL’s surface and volume shaders compute an explicit symbolic description, called a ”closure”, of the way a surface or volume scatters light, in units of radiance.

In blender, the cycles nodes with a green output socket are the "closure" nodes, they provide the BRDF/BSDF calculations that give the appearance in the render. Cycles provides the type of closures that are available, OSL cannot implement new closure types.

So yes, you would have to alter blenders code to offer different closure types, but an OSL script can add plenty of customisation to a cycles node tree without you having to alter blenders code. The OSL script is compiled using LLVM's JIT compiler to run on the current CPU being used. So an OSL script does "inject" new code into the render engine, it is just limited in how it can alter the final result.

Note that an OSL script does not have to provide a closure output, it may output float or colour values which allows it to provide intermediate nodes within a node tree.

As example OSL scripts, the LGHexTiles.osl provides a complete node that can be used by itself as a material. While the MAscales.osl provides colour and float outputs that make only one node in the material.

And like many things, OSL scripts can be abused. Here you will find info about the old Amiga Juggler being implemented in an OSL script. Yes, a simple render engine, written in OSL. I'm pretty sure he continued to develop the raytracer further.

Stackoverflow error usually occurs when there are too many recursive calls. In a nutshell, in most programming languages there is a limit on how many recursive calls you can make.

In your example, the trace functions calls itself recursively and it can cause an stack overflow. You have a parameters maxDepth that can limit that. Setting it to a lower value will probably solve this particular issue.

There are 2 function declarations in the middle of the function BRDF. This is not allowed. In GLSL a function definition has to be global.

See GLSL - The OpenGL Shading Language 4.6; 6.1. Function Definitions ; page 122:

As indicated by the grammar above, a valid shader is a sequence of global declarations and function definitions.

Move the declarations of Fresnel and ggx_visib before the declaration of BRDF to solve your issue.

The major issue is, that the BRDF explorer requires the keyword analytic at the begin of the file, to identify the file as a light model.

After the keyword is add add the file reloaded, the application will trace error cause by your GLSL code.

Fixing the errors results in the following code:

The tangent and the biTangent have to be transformed by the normal matrix as you do it with the normal vector:

I recommend to read about Vertex Specification and to use the state of the art way of Vertex Array Objects for drawing.

But, if you draw the objects in to deprecated Fixed Function Pipeline style, then geometric objects are drawn by enclosing a series of vertex coordinates between glBegin/glEnd pairs.

You have to finish the drawing sequence by glEnd, before you can change or manipulate the framebuffer.

Move the glEnd instruction in the inner loop and your code should work

The correct formula goes like this (in my opinion):

vec3 irradiance = sunDiffuseRadiance * NdotL;