Skip to content

IndexError: piece id is out of range. #12

@mxyttkx

Description

@mxyttkx

Thank you very much for your influential work, but I encountered the following error when running the chat file:
Traceback (most recent call last):
File "/data3/mxy/GeoPixel/chat.py", line 131, in
main(args)
File "/data3/mxy/GeoPixel/chat.py", line 82, in main
response, pred_masks = model.evaluate(tokenizer, query, images = image, max_new_tokens = 1000)
File "/data3/mxy/GeoPixel/model/geopixel.py", line 364, in evaluate
response = tokenizer.decode(output_ids[0].cpu().tolist(), skip_special_tokens=True)
File "/home/mxy/miniconda3/envs/geopixel/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 4004, in decode
return self._decode(
File "/home/mxy/miniconda3/envs/geopixel/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 1091, in _decode
filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
File "/home/mxy/miniconda3/envs/geopixel/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 1072, in convert_ids_to_tokens
tokens.append(self._convert_id_to_token(index))
File "/home/mxy/.cache/huggingface/modules/transformers_modules/GeoPixel/tokenization_internlm2.py", line 116, in _convert_id_to_token
token = self.sp_model.IdToPiece(index)
File "/home/mxy/miniconda3/envs/geopixel/lib/python3.10/site-packages/sentencepiece/init.py", line 1179, in _batched_func
return _func(self, arg)
File "/home/mxy/miniconda3/envs/geopixel/lib/python3.10/site-packages/sentencepiece/init.py", line 1172, in _func
raise IndexError('piece id is out of range.')
IndexError: piece id is out of range.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions