Trial run to investigate error effects of non-zero delta

[greenTara]

We will use two scripts to investigate the effects of delta:

1. the "MM" script (which runs faster than the QMM script) to get heatmaps, and corresponding numerical values with taumeta on the horizontal axis, with values of just error (not performance), and just for bayes (not for naive), averaged over numruns runs.
2. the QMM script to get parallel results to the first, but for delta = 1/8

[greenTara]

The purpose is to determine what numruns needs to be to distinguish delta=0 from delta = 1/8.

[greenTara]

First we print out the errors for the MM scripts after averageing over 2, 4 and 8 runs. We can use this to evaluate how close we are to the actual expectation of the error, and to predict what the error would be for 16 runs.

[greenTara]

Second we print out the parallel errors for the QMM scripts with delta = 1/8. This should give us enough information to select a value of numruns that will give us sufficiently accurate computation of the expectation of the error, so that we can compare the effect of changing delta.

[alexlafleur]

Average Errors Run 1-2:
[array([[-3.47353914, -4.45701716, -5.13950525],
[-3.97987578, -5.02509966, -5.44087834],
[-4.47290525, -5.57498663, -5.80400199]]), array([[-3.45201229, -4.42831078, -5.12893552],
[-3.97987578, -5.02509966, -5.44087834],
[-4.48758795, -5.58262855, -5.80293912]]), array([[-2.88837582, -3.74672001, -4.61105779],
[-3.41353809, -4.3673471 , -5.23309221],
[-3.9909425 , -4.98843404, -5.71775497]])]
Average Errors Run 1-4:
[array([[-3.36643247, -4.09021552, -4.92575815],
[-3.8716255 , -4.60920369, -5.24532033],
[-4.35712849, -5.09488531, -5.47841397]]), array([[-3.34063415, -4.06403929, -4.90725631],
[-3.8716255 , -4.60920369, -5.24532033],
[-4.37134826, -5.10382162, -5.48322792]]), array([[-2.76058521, -3.54037912, -4.41049051],
[-3.30974249, -4.0918174 , -5.04491004],
[-3.87862866, -4.77148279, -5.58563878]])]
Average Errors Run 1-8:
[array([[-3.22223761, -4.03251531, -5.03787018],
[-3.71168053, -4.53556666, -5.37636787],
[-4.1659853 , -4.96390431, -5.59587952]]), array([[-3.1949843 , -4.00695361, -5.01645588],
[-3.71168053, -4.53556666, -5.37636787],
[-4.17670124, -4.9678966 , -5.61611683]]), array([[-2.69137939, -3.52381605, -4.37071104],
[-3.21742656, -4.04663648, -5.05207792],
[-3.71112431, -4.65122694, -5.60605293]])]

[greenTara]

These errors indicate that the accuracy is about 10% with 8 runs, as in all
entries, the value appears to have two significant figures. We can expect
the expectation to converge as the square root of the number of runs, so
that means double the runs has a multiplier of 1/squareroot(2). To get
another significant figure, we need about 5 such multipliers, so that means
a factor of 32 on the runs (256 runs).

On Tue, Nov 15, 2016 at 9:43 AM, Alexandra La Fleur <
notifications@github.com> wrote:

Average Errors Run 1-2:
[array([[-1.88093458, -2.71686497, -3.60755385],
[-1.91357182, -2.74574782, -3.6551762 ],
[-1.92765883, -2.7580714 , -3.68488187]]), array([[-1.87799924,
-2.71596494, -3.60113799],
[-1.91357182, -2.74574782, -3.6551762 ],
[-1.92781358, -2.75790778, -3.68358005]]), array([[-1.79948771,
-2.64111551, -3.50863986],
[-1.86683799, -2.71856082, -3.57676251],
[-1.92068519, -2.76510473, -3.68955744]])]

Average Errors Run 1-4:
[array([[-1.79463279, -2.60093794, -3.54837208],
[-1.83963474, -2.63129166, -3.59782583],
[-1.86559807, -2.64440289, -3.62798217]]), array([[-1.79123639,
-2.59900763, -3.54310805],
[-1.83963474, -2.63129166, -3.59782583],
[-1.86615054, -2.64411639, -3.62679358]]), array([[-1.71341504,
-2.52645986, -3.42589508],
[-1.78298235, -2.61703027, -3.5439346 ],
[-1.83540226, -2.63508334, -3.59925761]])]

Average Errors Run 1-8:
[array([[-1.80306574, -2.60908356, -3.54753706],
[-1.86936242, -2.66617601, -3.61087225],
[-1.90097156, -2.69888865, -3.64969267]]), array([[-1.79932429,
-2.60714174, -3.54045783],
[-1.86936242, -2.66617601, -3.61087225],
[-1.90177187, -2.69878874, -3.64981979]]), array([[-1.70647284,
-2.51479206, -3.31557204],
[-1.81694475, -2.61188808, -3.56503651],
[-1.87082977, -2.7155192 , -3.60628365]])]

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#25 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABMMVQIM7yqjAe1oWjfUl1qUzBQ-5gomks5q-cURgaJpZM4Kyf4j
.

[greenTara]

How long did it take to do these 8 runs with the MM script?

[alexlafleur]

It took about 2 minutes

[alexlafleur]

MM Delta=0

MM Delta=0 Error Values

Average Errors Run 1-2:
[array([[-4.06213075, -4.91986446, -5.9423074 ],
[-4.49030057, -5.34600912, -6.38005423],
[-4.89197218, -5.71637183, -6.76279817]]), array([[-4.03666902, -4.89278453, -5.91098235],
[-4.49030057, -5.34600912, -6.38005423],
[-4.897968 , -5.72638015, -6.75866188]]), array([[-3.45419691, -4.450008 , -5.17642037],
[-4.03666902, -4.89278453, -5.91098235],
[-4.62381138, -5.16156467, -6.27132667]])]

[alexlafleur]

Parameter Overview for ETA:
taumeta: 2
eta: 8
scale_window : 16
shift: 16
window_size: 256
num_estimations: 256
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 65792
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for ETA:
taumeta: 2
eta: 16
scale_window : 16
shift: 32
window_size: 512
num_estimations: 120
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 61952
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for ETA:
taumeta: 2
eta: 32
scale_window : 16
shift: 64
window_size: 1024
num_estimations: 52
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 54272
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for ETA:
taumeta: 4
eta: 8
scale_window : 16
shift: 32
window_size: 512
num_estimations: 120
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 61952
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 3. ..., 3. 3. 3.]]
Parameter Overview for ETA:
taumeta: 4
eta: 16
scale_window : 16
shift: 64
window_size: 1024
num_estimations: 52
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 54272
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 3. ..., 3. 3. 3.]]
Parameter Overview for ETA:
taumeta: 4
eta: 32
scale_window : 16
shift: 128
window_size: 2048
num_estimations: 18
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 38912
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 3. ..., 3. 3. 3.]]
Parameter Overview for ETA:
taumeta: 8
eta: 8
scale_window : 16
shift: 64
window_size: 1024
num_estimations: 52
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 54272
BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for ETA:
taumeta: 8
eta: 16
scale_window : 16
shift: 128
window_size: 2048
num_estimations: 18
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 38912
BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for ETA:
taumeta: 8
eta: 32
scale_window : 16
shift: 256
window_size: 4096
num_estimations: 1
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 8192
BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for scale_window:
taumeta: 2
eta: 16
scale_window : 8
shift: 32
window_size: 256
num_estimations: 128
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 33024
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for scale_window:
taumeta: 2
eta: 16
scale_window : 16
shift: 32
window_size: 512
num_estimations: 120
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 61952
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for scale_window:
taumeta: 2
eta: 16
scale_window : 32
shift: 32
window_size: 1024
num_estimations: 104
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 107520
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for scale_window:
taumeta: 4
eta: 16
scale_window : 8
shift: 64
window_size: 512
num_estimations: 60
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 31232
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 3. ..., 3. 3. 3.]]
Parameter Overview for scale_window:
taumeta: 4
eta: 16
scale_window : 16
shift: 64
window_size: 1024
num_estimations: 52
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 54272
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 3. ..., 3. 3. 3.]]
Parameter Overview for scale_window:
taumeta: 4
eta: 16
scale_window : 32
shift: 64
window_size: 2048
num_estimations: 36
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 75776
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 3. ..., 3. 3. 3.]]
Parameter Overview for scale_window:
taumeta: 8
eta: 16
scale_window : 8
shift: 128
window_size: 1024
num_estimations: 26
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 27648
BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for scale_window:
taumeta: 8
eta: 16
scale_window : 16
shift: 128
window_size: 2048
num_estimations: 18
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 38912
BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for scale_window:
taumeta: 8
eta: 16
scale_window : 32
shift: 128
window_size: 4096
num_estimations: 2
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 12288
BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for NUM_TRAJ:
taumeta: 2
eta: 16
scale_window : 16
shift: 32
window_size: 256
num_estimations: 128
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 33024
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 1. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for NUM_TRAJ:
taumeta: 2
eta: 16
scale_window : 16
shift: 32
window_size: 256
num_estimations: 60
len_trajectory: 2177
num_trajectories: 4
num_trajectories_len_trajectory: 8708
NAIVE window_size * num_estimations 15616
BAYES window_size + num_estimations_shift 2176

[[ 1. 1. 1. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 3. 3. 2.]
[ 1. 0. 0. ..., 2. 3. 1.]
[ 2. 2. 3. ..., 2. 2. 2.]]
Parameter Overview for NUM_TRAJ:
taumeta: 2
eta: 16
scale_window : 16
shift: 32
window_size: 256
num_estimations: 26
len_trajectory: 1089
num_trajectories: 8
num_trajectories_len_trajectory: 8712
NAIVE window_size * num_estimations 6912
BAYES window_size + num_estimations_shift 1088

[[ 1. 1. 1. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 2. 2. 2.]
[ 1. 0. 0. ..., 1. 1. 1.]
...,
[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 1. ..., 1. 3. 3.]
[ 0. 0. 0. ..., 2. 2. 3.]]
Parameter Overview for NUM_TRAJ:
taumeta: 4
eta: 16
scale_window : 16
shift: 64
window_size: 512
num_estimations: 60
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 31232
BAYES window_size + num_estimations_shift 4352

[[ 1. 1. 1. ..., 0. 0. 0.]
[ 3. 3. 3. ..., 3. 3. 3.]]
Parameter Overview for NUM_TRAJ:
taumeta: 4
eta: 16
scale_window : 16
shift: 64
window_size: 512
num_estimations: 26
len_trajectory: 2177
num_trajectories: 4
num_trajectories_len_trajectory: 8708
NAIVE window_size * num_estimations 13824
BAYES window_size + num_estimations_shift 2176

[[ 1. 1. 1. ..., 3. 3. 3.]
[ 3. 3. 3. ..., 2. 3. 3.]
[ 2. 2. 2. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 2. 2. 2.]]
Parameter Overview for NUM_TRAJ:
taumeta: 4
eta: 16
scale_window : 16
shift: 64
window_size: 512
num_estimations: 9
len_trajectory: 1089
num_trajectories: 8
num_trajectories_len_trajectory: 8712
NAIVE window_size * num_estimations 5120
BAYES window_size + num_estimations_shift 1088

[[ 1. 1. 1. ..., 3. 3. 2.]
[ 3. 3. 3. ..., 1. 1. 1.]
[ 2. 2. 2. ..., 2. 2. 1.]
...,
[ 0. 0. 2. ..., 1. 1. 1.]
[ 2. 2. 2. ..., 3. 3. 1.]
[ 1. 1. 1. ..., 1. 1. 1.]]
Parameter Overview for NUM_TRAJ:
taumeta: 8
eta: 16
scale_window : 16
shift: 128
window_size: 1024
num_estimations: 26
len_trajectory: 4353
num_trajectories: 2
num_trajectories_len_trajectory: 8706
NAIVE window_size * num_estimations 27648
BAYES window_size + num_estimations_shift 4352

[[ 3. 3. 3. ..., 3. 3. 3.]
[ 2. 2. 2. ..., 1. 1. 1.]]
Parameter Overview for NUM_TRAJ:
taumeta: 8
eta: 16
scale_window : 16
shift: 128
window_size: 1024
num_estimations: 9
len_trajectory: 2177
num_trajectories: 4
num_trajectories_len_trajectory: 8708
NAIVE window_size * num_estimations 10240
BAYES window_size + num_estimations_shift 2176

[[ 3. 3. 3. ..., 1. 1. 1.]
[ 2. 2. 2. ..., 1. 1. 1.]
[ 2. 2. 2. ..., 1. 1. 1.]
[ 1. 1. 1. ..., 2. 2. 3.]]
Parameter Overview for NUM_TRAJ:
taumeta: 8
eta: 16
scale_window : 16
shift: 128
window_size: 1024
num_estimations: 0
len_trajectory: 1089
num_trajectories: 8
num_trajectories_len_trajectory: 8712
NAIVE window_size * num_estimations 1024
BAYES window_size + num_estimations_shift 1024