Using ESM-C Original Weights or Re-Trained?
Great work! I had to create my own code patches over the original esm package in order to finetune it with PEFT. However, I've got a question. Per model card description, "ESM++ is a faithful implementation of ESMC (license) that allows for batching and standard Huggingface compatibility without requiring the ESM Python package", does implementation mean a wrapper over original ESM-C model weights, or the model was retrained? It might be a stupid question, but I just want to double check.
Hi @libokj ,
Not a stupid question! ESM++ uses exactly the same weights as ESMC. If you don't trust our Huggingface weights, you are more than welcome to load from ESM directly with from_pretrained_esm instead of from_pretrained. Just use paths ESMC-300 or ESMC-600 instead of our paths.
If you have any other questions please let me know.
LoRA fine tuning of protein language models will be easier than ever with our Protify package. ESM++ is offered as a base model here as well. Please check it out if you are interested, there will be more examples, docs, and features soon 🤗
Best,
Logan
Hi @libokj ,
Not a stupid question! ESM++ uses exactly the same weights as ESMC. If you don't trust our Huggingface weights, you are more than welcome to load from ESM directly with
from_pretrained_esminstead offrom_pretrained. Just use pathsESMC-300orESMC-600instead of our paths.If you have any other questions please let me know.
LoRA fine tuning of protein language models will be easier than ever with our Protify package. ESM++ is offered as a base model here as well. Please check it out if you are interested, there will be more examples, docs, and features soon 🤗
Best,
Logan
Thanks for the quick response, Logan! It’s great to see such active engagement in open-source communities. I’ll definitely give the Protify package a try. Keep up the fantastic work! I do have a follow-up question: Would it be possible to share a non-normalized version of the linear probing performance plot or a table with the raw results for all the PLMs tested? Additionally, are there plans to publish these findings in a journal or as a preprint in the future?
Would it be possible to share a non-normalized version of the linear probing performance plot or a table with the raw results for all the PLMs tested? Additionally, are there plans to publish these findings in a journal or as a preprint in the future?
This plot is actually from a much earlier version of Protify! I'm not sure if I have the raw data still but it will be easily reproduceable soon once the Protify kinks are sorted out.
There will be a conference paper submitted (likely) next month depicting results like this and the uses of Protify. If you would like to contribute to the package and get on the paper, feel free to utilize our bounty system on the Protify repo. If a non-normalized version is useful to you please open up an issue there asking for it - that way it is tracked in our system and we will get to it quicker (not all our scientists get Huggingface notifications).