lwjgl3-demos | Demo suite for LWJGL | Learning library

There's a bunch of problems in your code, and that's without even seeing the core of the problem yet, since it's in the // ... part:

• You don't use glfwSwapBuffers when you're trying to render in a headless setup, that's only needed when you render to the context back buffer and want to swap it to screen.

• You don't want both glGetTexImage and glReadPixels, they both read data from graphics memory to system memory. In your case you're reading the texture data in buffer then overwriting it with the backbuffer data, which should be empty since you never wrote to it.

• You definitely do not want glBlitFramebuffer, that's for copying from video memory to video memory. You seriously need to stop and read the documentation, throwing random functions at your video driver is a sure way to make it crash.

• You need to enable OpenGL's debug layer validation, especially while trying all these random functions. And related, you need to check your frame buffer setup (glCheckFramebufferStatusEXT), I'm willing to bet money you didn't, but you don't show your code.

• You don't show your frame buffer bindings, so I can't tell if you're doing this for sure or not, but you need to bind the frame buffer as "read" (GL_READ_FRAMEBUFFER) before reading from it. I only see a "draw" frame buffer binding, and you're only randomly clearing it.

• And lastly, you should never stall the CPU on a read operation from video memory, especially when you have things you should be doing, like video encoding. Use pixel buffer objects in a round-robin fashion to double or triple buffer the transfer, so there's no stall.