Speed benchmarking

[tarunravi]

I'm interested in seeing generation speed benchmarks. For me generating the following text:

The fluorescent lights above hummed in unison with your racing thoughts as you sat across from Jax, his rugged face a mask of concentration

Took ~30 seconds on my NVIDIA 3060. This feels a bit slow. Im curious if yall are getting the same speeds or not.

[greatmate98]

36 seconds for me on 12GB 4080 laptop. Strangely, this is slower than using the main fork and pf32 model, which took 26 seconds. I am not sold on the benefits of this fork - i wanted to test it because of the built in speakers, but find that it doesnt adhere to the speaker (sometimes the voice for S1 or S2 just takes over and says everything rather than taking turns) and things like (laughs) (gasps) etc. are always cut short. Wonder if its the fp16 model causing this. But right now this is slower and poorer quality than the main fork, with the additional stuff not working as expected. Its a nice interface and was easy to install though.

Edit - i turned off breaking it down into chunks and it did 28 seconds, so similar to original repo. I figured breaking it into chunks was for performance.

[devnen]

Thank you for the update. The quality or speed should not be different than the official repo. If you keep the chunking/splitting option enabled, however, this may introduce additional delays due to the postprocessing like silence removal, merging chunks and similar. But, about 30 seconds for your sample sentence sounds about right.

[GoudaCouda]

I am having the same issues now after Dia made changes to their code it is much faster. On the original repo I get almost .8x rt but on this I am at about .35x. I am using a 3090 with cuda 12.4