Model.generate use_cache=True generates different results than use_cache=False

[John6666]

Similar case?

Transformer KV-Cache Produces Worse Output Than Normal Generation – Why?

Hi everyone, I implemented a transformer model from scratch based on Andrej Karpathy’s video. After that, I added some improvements, such as parallelized multi-head attention and KV-Cache for efficient generation. However, I am facing a problem: when I enable KV-Cache, the generated text is significantly worse than when KV-Cache is disabled. I expected KV-Cache to improve speed while keeping output quality the same, but my implementation seems to degrade output quality. I suspect there might…