Skip to content

The model is not convergent. #7

@TrackingBird

Description

@TrackingBird

When I fine tuning the network with my dataset, The logs are as follows. I think the model is not convergent. The disparity computed by the fine-tuned model is a white image with a little texture.
Can you give me some suggestion about how to train the model? Or how to get a real reliable training dataset. Thanks.

I0508 15:53:59.874593  6385 solver.cpp:245]     Train net output #73719: blob65 = 74.7856
I0508 15:53:59.874598  6385 solver.cpp:245]     Train net output #73720: blob65 = 74.7856
I0508 15:53:59.874603  6385 solver.cpp:245]     Train net output #73721: blob65 = 74.7857
I0508 15:53:59.874608  6385 solver.cpp:245]     Train net output #73722: blob65 = 74.7857
I0508 15:53:59.874614  6385 solver.cpp:245]     Train net output #73723: blob65 = 74.7857
I0508 15:53:59.874619  6385 solver.cpp:245]     Train net output #73724: blob65 = 74.7856
I0508 15:53:59.874624  6385 solver.cpp:245]     Train net output #73725: blob65 = 74.7856
I0508 15:53:59.874629  6385 solver.cpp:245]     Train net output #73726: blob65 = 74.7856
I0508 15:53:59.874634  6385 solver.cpp:245]     Train net output #73727: blob65 = 74.7856
I0508 15:53:59.874646  6385 solver.cpp:245]     Train net output #73728: flow_loss1 = 0.0113061 (* 1 = 0.0113061 loss)
I0508 15:53:59.874652  6385 solver.cpp:245]     Train net output #73729: flow_loss2 = 174.297
I0508 15:53:59.874658  6385 solver.cpp:245]     Train net output #73730: flow_loss3 = 170.482
I0508 15:53:59.874665  6385 solver.cpp:245]     Train net output #73731: flow_loss4 = 163.814
I0508 15:53:59.874670  6385 solver.cpp:245]     Train net output #73732: flow_loss5 = 154.755
I0508 15:53:59.874675  6385 solver.cpp:245]     Train net output #73733: flow_loss6 = 141.625
I0508 15:53:59.874682  6385 solver.cpp:245]     Train net output #73734: loss_down = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874689  6385 solver.cpp:245]     Train net output #73735: loss_est = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874701  6385 solver.cpp:245]     Train net output #73736: loss_up = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874707  6385 solver.cpp:245]     Train net output #73737: loss_west = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874717  6385 sgd_solver.cpp:106] Iteration 11950, lr = 3.125e-06
I0508 15:55:18.302011  6385 solver.cpp:456] Snapshotting to binary proto file disp_iter_12000.caffemodel
I0508 15:55:18.302299  6385 net.cpp:867] Serializing 137 layers
I0508 15:55:21.718309  6385 sgd_solver.cpp:273] Snapshotting solver state to binary proto file disp_iter_12000.solverstate
I0508 15:55:29.855378  6385 solver.cpp:318] Iteration 12000, loss = 0.00952777
I0508 15:55:29.855406  6385 solver.cpp:323] Optimization Done.
I0508 15:55:29.855428  6385 caffe.cpp:222] Optimization Done.````

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions