Interact with Your Own Data Using Ollama Hosted Phi3

 Phi3 is a very small large language model, with only 4 billion parameters compared to ChatGPT's 175 billion. Despite its smaller size, Phi3 offers a unique advantage: it can directly interact with your data without needing an API. In contrast, ChatGPT requires an API, and the free access to this API expires after three months unless you upgrade to a paid plan at $25 per month.

One of the main benefits of Ollama hosted Phi3 is that it doesn't require internet access and is completely free. However, running Phi3 locally does have its drawbacks. For optimal performance, it requires a high-spec PC or Mac, particularly one with a powerful GPU. While it can run on lower-spec machines, the performance will be significantly slower.

To demonstrate how Phi3 works, I have prepared a simple program that uploads a text file, processes it, and allows you to ask questions about the data. For this example, I used data about the national anthem of the Philippines. When I asked the untrained Phi3 who wrote the national anthem, it provided an incorrect answer. This is due to the limited amount of training data available for Phi3, as mentioned earlier. The following picture shows the reply of Phi3(without embeddings) from the DOS prompt:

I obtained the data from Wikipedia by just copying a few paragraphs and pasting it to Notepad and saving it as .txt file.

Here is the program and some explanations:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

from langchain_chroma import Chroma from langchain_community.document_loaders import TextLoader from langchain_community.embeddings.sentence_transformer import ( SentenceTransformerEmbeddings, ) from langchain_text_splitters import CharacterTextSplitter from langchain_community.embeddings import OllamaEmbeddings from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_community.vectorstores import Chroma from langchain.prompts import ChatPromptTemplate, PromptTemplate from langchain_core.output_parsers import StrOutputParser from langchain_community.chat_models import ChatOllama from langchain_core.runnables import RunnablePassthrough from langchain.retrievers.multi_query import MultiQueryRetriever loader = TextLoader("lupanghinirang.txt") documents = loader.load() # Split the document into chunks text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) docs = text_splitter.split_documents(documents) # Create the open-source embedding function embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2") # Load it into Chroma db = Chroma.from_documents(docs, embedding_function) # Query the database query = "What is the National Anthem of the Philippines?" docs = db.similarity_search(query) # Split and chunk the documents text_splitter = RecursiveCharacterTextSplitter(chunk_size=7500, chunk_overlap=100) chunks = text_splitter.split_documents(docs) # Add to vector database vector_db = Chroma.from_documents( documents=chunks, embedding=OllamaEmbeddings(model="nomic-embed-text", show_progress=True), collection_name="local-rag" ) # LLM from Ollama local_model = "phi3" llm = ChatOllama(model=local_model) QUERY_PROMPT = PromptTemplate( input_variables=["question"], template="""You are an AI language model assistant. Your task is to generate five different versions of the given user question to retrieve relevant documents from a vector database. By generating multiple perspectives on the user question, your goal is to help the user overcome some of the limitations of the distance-based similarity search. Provide these alternative questions separated by newlines. Original question: {question}""", ) retriever = MultiQueryRetriever.from_llm( vector_db.as_retriever(), llm, prompt=QUERY_PROMPT ) # RAG prompt template = """Answer the question based ONLY on the following context: {context} Question: {question} """ prompt = ChatPromptTemplate.from_template(template) chain = ( {"context": retriever, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() ) chain.invoke("who composed the national anthem of the Philippines?")

Explanation

1. Loading and Splitting Documents:

   • The program starts by loading a text file (lupanghinirang.txt) using TextLoader.
   • The text is then split into chunks using CharacterTextSplitter.

2. Embedding and Storing Data:

   • An embedding function is created using the SentenceTransformerEmbeddings model.
   • The split documents are embedded and stored in a Chroma database.

3. Querying the Database:

   • A query is made to the database to find relevant documents.
   • The retrieved documents are further split into chunks using RecursiveCharacterTextSplitter.

4. Adding to Vector Database:

   • The chunks are embedded using OllamaEmbeddings and stored in another Chroma database.

5. Setting Up the Language Model:

   • The ChatOllama model (Phi3) is initialized.
   • A prompt template is created to generate multiple versions of a user query.

6. Retrieving Relevant Documents:

   • MultiQueryRetriever is used to retrieve relevant documents from the vector database.

7. Answering the Query:

   • A final prompt template is set up to answer the question based on the retrieved context.
   • The chain is executed to generate the answer to the question.

Here is the result when running the program:

This program demonstrates the capabilities of Phi3 in processing and interacting with local data. Despite its smaller size compared to ChatGPT, Phi3 can be a powerful tool for specific applications where internet access is limited or data privacy is a concern.

Upgrading the Ollama Hosted Phi3 Offline Chat Program

 In my previous post, "Chat with Ollama Phi3 in Python Offline," I introduced a basic program that allowed you to chat with Ollama's Phi3 language model offline using Python. Today, I’m excited to share an upgraded version of this program. This new version incorporates a graphical user interface (GUI) using PyQt6 and includes a text-to-speech (TTS) engine to read aloud Phi3's responses. The chat logs are displayed in a QTextEdit widget, making the interaction more user-friendly and accessible. This program is created in Python 3.10 same experience I encountered when I first create a chat program using Langchain and Openai. also a good pc/mac will deliver a better experience but it will still run on a Intel M3-100y but it will be very slow considering that I have downloaded the phi3 4b  quantized version.

Upgraded Features

1. Graphical User Interface (GUI): A chat window created using PyQt6.
2. Text-to-Speech: Utilizes pyttsx3 to read aloud Phi3's responses.
3. Chat Logs: Displays the conversation history in a QTextEdit widget.

Code Explanation

Below is the complete code for the upgraded program, followed by detailed explanations of each part.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

import pyttsx3 from PyQt6.QtGui import * from PyQt6.QtCore import * from PyQt6.QtWidgets import * from langchain_community.llms import Ollama class Window(QMainWindow): def __init__(self): super(Window, self).__init__() # Set up the layout layout = QVBoxLayout() hlayout = QHBoxLayout() central_widget = QWidget() central_widget.setLayout(layout) self.setCentralWidget(central_widget) # Initialize the language model self.llm = Ollama(model="phi3") # Add widgets to the layout layout.addLayout(hlayout) self.te = QLineEdit(self) self.le = QTextEdit(self) self.le.setReadOnly(True) self.btnStart = QPushButton("Ask", self) hlayout.addWidget(self.te) hlayout.addWidget(self.btnStart) layout.addWidget(self.le) # Connect button click to function self.btnStart.clicked.connect(self.onClickedStart) # Set window properties self.setGeometry(25, 45, 900, 500) self.setWindowTitle('Speak') def onClickedStart(self): global text # Get user input and display it in the chat log user_input = self.te.text() self.le.append(f'You: {user_input}') # Get response from the language model response = self.llm.invoke(user_input) # Display the response in the chat log self.le.append(f'Phi3: {response}') # Prepare text for text-to-speech text = response self.worker = WorkerThread() self.worker.start() text = '' engine = [] error = '' i = 0 class WorkerThread(QThread): def run(self): global text, engine, error, i try: i += 1 engine.append(pyttsx3.init()) engine[i-1].setProperty('rate', 120) engine[i-1].say(text) engine[i-1].startLoop() except Exception as err: error = str(err) if __name__ == "__main__": import sys app = QApplication(sys.argv) w = Window() w.show() sys.exit(app.exec())

Detailed Explanation

Imports

• pyttsx3: A text-to-speech conversion library in Python.
• PyQt6: A set of Python bindings for the Qt application framework, used to create the GUI.
• Ollama: A module from the langchain_community package for interacting with the Phi3 language model.

Window Class

• Initialization (__init__ method):

  • Sets up the main layout and widgets (QLineEdit, QTextEdit, QPushButton).
  • Initializes the Phi3 language model using Ollama.
  • Connects the "Ask" button to the onClickedStart method.
  • Configures the window size and title.

• onClickedStart Method:

  • Gets the user's input from the QLineEdit widget.
  • Appends the user's input to the QTextEdit widget for display.
  • Sends the input to the Phi3 language model and gets the response.
  • Appends the response to the QTextEdit widget.
  • Initiates a text-to-speech conversion of the response using the WorkerThread class.

WorkerThread Class

• A QThread subclass that handles the text-to-speech functionality.
• run Method:
  • Initializes a new pyttsx3 engine for each response.
  • Sets the speech rate and converts the response text to speech.
  • Starts the speech engine loop to read the response aloud.

Main Block

• Creates a QApplication instance and a Window instance.
• Displays the Window and starts the event loop.

Conclusion

This upgraded version of the Ollama Phi3 offline chat program provides a more interactive and user-friendly experience. The integration of a GUI and text-to-speech functionality makes it easier to interact with the Phi3 model without needing to modify the underlying code. Give it a try and see how these enhancements improve your interactions with Phi3!

Chat with Ollama Phi3 in Python Offline

 Ollama lets you download and run local versions of LLMs of your choice. In this article, I downloaded Phi3 with 8B parameters.

To begin, download the Ollama software from their website. Then, in the DOS prompt, type ollama run phi3. The download will begin, and after downloading, it will start Phi3 right away so you can chat with Phi3 immediately.

Install the required library.

pip install langchain_community

Next, I prepared this Python program to interact with Phi3 locally, i.e., offline:

1 2 3 4 5 6 7 8

from langchain_community.llms import Ollama llm = Ollama( model="phi3" ) # assuming you have Ollama installed and have the phi3 model pulled with `ollama pull phi3` x = llm.invoke("Tell me a joke") print(x)