Item Embedding

[lms-design]

Hello, after studying your work, I have two questions regarding item embedding: 1. Does the quality of text embedding significantly affect subsequent training? 2. Is it necessary to fine-tune the embedding model based on domain-specific data to improve the quality of semantic IDs?

[AkaliKong]

Thank you for your questions! Here are the answers:

1. Improving the quality of text embeddings can indeed bring modest performance gains. In our experiments, as the capacity of the text encoder increases, the recommendation ability shows a mild scaling-law effect.

2. Regarding domain-specific fine-tuning:
   Some existing approaches concatenate text embeddings with collaborative-filtering embeddings before performing residual quantization (RQ). fine-tuning the embedding model on domain data will introduce additional computational cost, but it is very likely to improve recommendation performance by producing higher‑quality SIDs.
   One thing to keep in mind is that if you still adopt a two-stage pipeline (SID Construction → SID-based Generation), you will need a suitable intermediate objective to serve as the optimization target during embedding-model tuning.

Hope this helps! Feel free to reach out if you have more questions.

[lms-design]

thanks, your reply was very helpful to me. I am attempting to reproduce OneRec-v2 starting from pretraining. Based on your response, I have some speculation: For the lazy transformer pretrained solely from user sequences using only SIDs as tokens, the quality of SIDs should have a relatively significant impact on the training results. In comparison, in your work, the step of ALIGNMENT WITH LLMS integrates the knowledge from SIDs into the model’s world knowledge, so the requirement for SID quality is likely lower than in the former approach. I wonder if this speculation is correct.