Clelia Astra Bertelli

I am working on supporting compatibility with other embeddding models, and we will have that soon, for now I had to reduce the compatibility only to Sentence Transformers.
For what concerns page numbers, I am also working toward having better and more extensive metadata: everything is a big work-in-progress and will come in future releases!

So, there are two possibilities:

• If you mean customizing the embedder among the ones available within Sentence Transformers, it is very possible, you just have to change the embedding_model parameter when calling the ingest method
• If you mean that you have your own embedding model (like saved on your PC), that is a tad more difficult. I think Sentence Transformer might allow loading the model from your PC as long as it is compatible with the package. I think that this guide might be useful in that regard

For now the package only supports Sentence Transformers models, in the future it will probably extend its support to other embedding models as well :)

Hey @T-2000 , you're absolutely right! I'm in the process of making the application online so for now the repo got a bit messy, tomorrow it will be clean and ready to be spinned up also locally: sorry for the incovenient!

I used good old Canva (pro :)

Hi!

I generally use LangChain + PyPDF, I leave here a code snippet:

from langchain_community.document_loaders import PyPDFLoader

def preprocess(pdf: str) -> list:
    """
    Uses LangChain's PyPDFLoader to extract text.
    """
    loader = PyPDFLoader(pdf)
    documents = loader.load()
    for doc in documents:
        print(doc.page_content)    

This should give a more solid result :)

PS: Langchain is distributed under an MIT license, see their GitHub (https://github.com/langchain-ai/langchain)

Thank you so much for letting me know! This is indeed a very interesting role :)