How do I send system prompts using inference api serverless, llama3 8b instruct model

[marcoguerrap]

Hey I found this post that helped me:

Formatting Inference API call for LLama 2

I solved it by inputting a single string using the official Llama 2 format (see Llama 2 is here - get it on Hugging Face). I don’t know why the default Sagemaker Llama endpoint doesn’t work that way. But this works for me: import json import requests API_URL = "https://huggingface.co/proxy/api-inference.huggingface.co/models/meta-llama/Llama-2-7b-chat-hf" headers = {"Authorization": f"Bearer hf_XXXXXXXXXXXXXXXXXX", "Content-Type": "application/json",} def query(payload): json_body = { "inputs":…