@adiudiun, I had the same problem. The correct way is to first load the base_model using AutoModel.from_pretrained(). Then, load the adapter config using PeftConfig.from_pretrained('saved_dire'). Finally, load the peft_model as follows:
Load the entire model with adapters
peft_model = PeftModel.from_pretrained(base_model, saved_dire)
Load adapter1 and adapter2
peft_model.load_adapter(saved_dire + ‘/adapter1’, adapter_name=‘adapter1’)
peft_model.load_adapter(saved_dire + ‘/adapter2’, adapter_name=‘adapter2’)
Here is the screenshot of that part:
Also, here is the link to the turtorial,
https://github.com/akashghimireOfficial/HuggingFace_101/blob/master/Turtorial/12_more_about__get_peft_model.ipynb