Data Creation for RAG Optimization at HF-Space

AutoRAG HF-Space Tutorial #2. Data Creation

[embed]AutoRAG Data Creation - a Hugging Face Space by AutoRAG Make RAG evaluation dataset. 100% compatible to AutoRAGhuggingface.co

0. Basic

For RAG pipeline optimization, we need Corpus data and QA data for evaluation.

AutoRAG Data Creation Process

We have 3 step parsing, chunking and QA creation.

• Parsed result: Raw
• Chunked result: Corpus
• QA Creation result: QA

If you already have parsed results (=raw.parquet) or corpus data, uncheck the box to show a space for uploading files.

1. Parsing

🛠️ Note

To use llama parse and upstage, you need to get an API key and enter it.

2. Chunking

Chose chunking method and chunk parameters.

🛠️ Note

To use Semantic Chunking, we need an embedding model, so we need to put it in.

Embedding Model: OPENAI text-embedding-ada-002

3. QA Creation

3–1. Set OpenAI API Key

Select your language for QA data.

3–2. Run QA Creation

Sample Run QA Creation

4. Download your Corpus and QA

Don’t forget download created corpus and qa parquet file.