Thai language RAG with Llamaindex + Weaviate + SeaLLM
Sa-Wad-Dee, Hello from Thailand.
Introduce
RAG is like this cool AI tool designed for developers and teams who want to bring LLMs into new features. But gotta admit, there's a small hiccup - we don't really have a ton of tutorial projects and this leads to a couple of big issues.
- Everything's pretty much in English. So for those who aren't native English speakers, they have to tweak their solutions and spend that extra time getting acquainted with stuff like Tokenizers or Text splitters.
- Everything's pretty much geared for OpenAI's API, which is the easiest and most efficient to use. But on the flip side, this makes other AI providers and embedding models feel left out. And let's not forget those developers who, trying to save some bucks, want to use other AI and embeddings - they're hit with crazy high switching costs.
These issues end up slowing the AI adoption process and bumping up costs, especially for firms not primarily using English.
So, the idea to create use cases with examples came about. It's pretty much a way to help developers learn faster and get AI applications up and running quicker, whether to boost existing features or whip up new ones.
Challenge
- Tiktoken counts each character in the Thai language as one token, whereas a local AI model could consider each word as one token.
- Thai language doesn't complete sentences with a "full stop".
- Integrate Huggingface Embedding and an OpenAI API-like with LlamaIndex.
- Highly customize the vector search using Weaviate.
Environment
- LlamaIndex (Data Framework)
- Weaviate (Vector Database)
- SeaLLM-7b (AI Model) Serving by Float16.cloud
- intfloat/multilingual-e5-large (Embedding Model)
The notebook
Float16-example/llamaindex/RAG at main · vultureprime/Float16-example
Contribute to vultureprime/Float16-example development by creating an account on GitHub.
vultureprimeWarning. I have commented the code in Thai language.
