I tried to construct a simple AI agent that would fetch the schema and then query the particular table for a customers detail! for this I used Langchain and a local LLM that I pulled from Ollama (llama3:8b), this is how I have created the AI Assistant using lancghain as suggested by fellow forum member!
import os
import django
from django.conf import settings
os.environ.setdefault(‘DJANGO_SETTINGS_MODULE’, ‘agent_project.settings’)
django.setup()
from langchain_community.chat_models import ChatOllama
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents import AgentExecutor, create_react_agent
from langchain.tools import StructuredTool
from langchain.memory import ConversationBufferMemory
from .schema_reader import schema_reader, SchemaReaderInput
from .indexed_reader import fetch_invoices_by_company_and_year, InvoiceFetchInput
from .query_runner import query_runner, QueryRunnerInput
from .file_reader import pdf_reader, FileReaderInput
from .fetchfile import file_path_fetcher, FilePathFetcherInput
from .modify_table_data import Modify_Table_Data_Input, modify_table_data
llm = ChatOllama(
model=“llama3:8b”,
temperature=0.0,
)
print(“Using ChatOllama model with create_react_agent.”)
tools=[
StructuredTool(
name=“schema_reader”,
description=“Use this to get the list of ALL tables and their columns in the database. This is your FIRST step to understand the database structure.”,
func=schema_reader,
args_schema=SchemaReaderInput,
return_direct=False
),
StructuredTool(
name=“query_runner”,
description=“Executes SQL queries on the SQLite database. Use this ONLY when you already know the correct SQL query after consulting the schema.”,
func=query_runner,
args_schema=QueryRunnerInput,
return_direct=False
),
]
system_prompt = “”"
You are a highly intelligent and versatile assistant with two primary areas of expertise: interacting with a SQLite database and processing PDF documents. Your responses must be accurate, efficient, and strictly based on the information retrieved from your tools.
Core Objective: To answer user questions by fetching information from a database, reading PDF files, or, when explicitly and carefully instructed, modifying the database.
MANDATORY WORKFLOWS & RULES:
You MUST follow the correct workflow based on the user’s request.
Workflow 1: Answering Questions About the Database
- Understand the Database Structure: If you do not know the table names and their columns, your absolute first step is to use the
schema_readertool. - Formulate a Query: Based on the schema, formulate a precise
SELECTSQL query to answer the user’s question. - Execute the Query: Use the
query_runnertool to execute theSELECTquery. - Provide the Answer: Analyze the result from
query_runnerand provide a concise answer to the user.
Workflow 2: Reading and Summarizing a PDF Document
- Find the Document: The user will provide a document name. Your first step is to use the
file_path_fetchertool to get the full URL for that document name. - Read the Content: Once you have the full URL from
file_path_fetcher, use thepdf_readertool with that URL to extract the text from the document. - Provide the Answer: Analyze the extracted text to answer the user’s question (e.g., provide a summary).
General Rules:
- NEVER invent information. If you cannot find the answer using your tools, state that clearly.
- NEVER use
database_modifierforSELECTqueries. Usequery_runner. - NEVER use
query_runnerforINSERT,UPDATE,DELETE, orALTERqueries. Usedatabase_modifier. - If a tool returns an error, report the error to the user and ask for clarification. Do not try the same failed action again unless you have a good reason.
“”"
prompt = ChatPromptTemplate.from_messages([
(“system”, system_prompt),
MessagesPlaceholder(variable_name=“chat_history”),
(“human”, “{input}”),
MessagesPlaceholder(variable_name=“agent_scratchpad”),
])
agent = create_react_agent(llm, tools, prompt)
memory = ConversationBufferMemory(memory_key=“chat_history”, return_messages=True)
agent_executor = AgentExecutor(
agent=agent,
tools=tools,
verbose=True,
handle_parsing_errors=True,
memory=memory
)
BUT STILL, when I ask a simple query like
“fetch the first name of the user whose email id is [email protected]”, the AI assistant enters the agent executor chain! but instead of getting the schema and querying the customer table, it just keeps saying 'INVALID TOOL", and just displays text like it Is using the tool, but it is actually not using the tool! and sometimes it even shows that Action, Observation, Action Input and stuff, but it does not use the proper tools and I have tried changing the prompt many times!
MY QUESTIONS
- Do local models(LLM’s) lack the tool/function calling ability?(gemini, ai studio, chatgpt and many other AI tools say that local models are not trained to recognize and use tools)
- am I doing something wrong in the above code!! I tried many combinations of agents along with the prompt types (hwchase17/{type_of_agent_name}) but still the local model does not work!!
- IS IT POSSIBLE TO TRAIN THIS MODEL TO MAKE IT GAIN TOOL/FUNCTION CALLING ABILITIES(I read somewhere that it is possible), kindly guide me on the same if that is possible!
NOTE:
I ALSO TRIED THE “LLAMA3-GROQ-TOOL-USE” MODEL THAT IS CLAIMED TO PERFORM WELL, BUT THAT ALSO GIVES THE SAME “INVALID TOOL” RESPONSE, SO I AM AT THE CONCLUSION THAT LOCAL LLM’s CANNOT BE USED FOR ANYTHING BUT SIMPLE GENERATIVE TASKS"
By the way, my job has nothing to do with AI.