Per-view depth map format and build question

[dknyxh]

Hi,

This is a really nice work! I tried to use my own global mesh/images for ULR rendeirng and it works fine. Now I want to supply depth map to get the per-view mesh, I am having trouble finding the format for the correct depth map format. They are saved as 8-bit png files when I ran the acquisition package. Do you have any documents stating what the format it should be?

The other issue is when I tried to build a final binary, i don't think the data are copied over and when i launch it just shows a empty scene. Is this expected?

Thanks!

[DinechinGreg]

Thanks for your feedback!

Ah, it seems that the documentation is indeed lacking in this regard. The depth maps are basically encoded using Unity's built-in shader method EncodeFloatRGBA, using only the RGB channels. When launching acquisition, the 0-1 float values in the virtual camera's depth texture are thus encoded as an RGB color using the method Encode01AsPreciseColor (used here: https://github.com/CAOR-MINES-ParisTech/colibri-vr-unity-package/blob/master/Shaders/Acquisition/Convert01ToColor.shader, defined here: https://github.com/CAOR-MINES-ParisTech/colibri-vr-unity-package/blob/master/Shaders/CGIncludes/ColorCG.cginc).

I haven't been able to extensively test building projects using the package, so it is quite possible that the paths to the data get broken during the process. I'd have to work on this question, but I'm not sure that I'll be able to allocate time to this issue, and I'm currently out of office in any case. Feel free to investigate though, and to submit a pull request if you find a fix, it would indeed be great to have builds up and running!

[dknyxh]

Cool. Thanks for the response!

[dknyxh]

Hi, I tried to supply the depth map. I convert the depth locally first using python and put them in the Data/stereo folder.

I did the following:

1. Convert the depth z to non-linear 0 to 1 value. : x = (1.0/x - 1.0/min_v)/(1.0/max_v - 1.0/min_v). Here I use min_v = 0.3, max_v=1000 same as the default range.
2. Convert the 0-1 value into RGBA by mimic the Encode01AsPreciseColor with the followings:

    x = np.clip(x, 0, 1.0 - 1/16581375.0)
    x = x.reshape(-1, 1) #N x 1
    kEncodeMul = np.array([1.0, 255.0, 65025.0, 16581375.0]).reshape(4,1) # 4 x 1
    kEncodeBit = 1.0/255.0;
    enc = np.dot(kEncodeMul, x.T) #  4 x N 
    enc = np.modf(enc)[0]
    yzww = enc[(1,2,3,3),:] #4xN
    enc -= yzww * kEncodeBit; #4xN

And I only output the RGB channel as uint8 into a image.

However, in the end I get something like this:

I'm not sure what's wrong with my conversion, the depth map looks something like this:

[DinechinGreg]

Sorry about the late reply. The approach looks good, but I agree that the output depth map is not right, so we must be missing something somewhere in the details.

Could you please try testing your version of Encode01AsPreciseColor for sample values?
The way the algorithm works, you should be able to try it out as follows:

r = 0.0 #values between 0.0 and 254.0 should be fine
g = 0.0 #same
b = 0.0 #same
output = Encode01AsPreciseColor(r/255.0 + g/65025.0 + b/16581375.0)
#save output as image

So for instance, if 'r=100.0' and 'g=0.0, b=0.0', the output should be a shade of red. Is this the case for red, green, and blue?

Thanks again for trying this out. I hope it will work out, it's a great idea to create a python script to convert numpy arrays of depth values into the depth map format we use here in Unity!