数据流

🛠️ 7.1 Environment Setup and Installation

DataFlow supports Python>=3.10 environments, tested passed on Windows, Linux, and MacOS with Python 3.10, 3.11, and 3.12.

Please use the following commands for environment setup and installation👇

We recommend use uv to install DataFlow for speed up.

pip install uv
uv pip install open-dataflow

If you want to use your own GPU for local inference, please use:

pip install uv
uv pip install open-dataflow[vllm]

After installation, you can use the following command to check if dataflow has been installed correctly:

dataflow -v

If installed correctly, you should see:

open-dataflow codebase version: 1.0.0
        Checking for updates...
        Local version:  1.0.0
        PyPI newest version:  1.0.0
You are using the latest version: 1.0.0.

🐳 7.2 Docker Installation (Alternative)

We also provide a Dockerfile for easy deployment and a pre-built Docker image for immediate use.

Option 1: Use Pre-built Docker Image

You can directly pull and use our pre-built Docker image:

```shell

Inside the container, verify installation

dataflow -v

##### Option 2: Build from Dockerfile

Alternatively, you can build the Docker image from the provided Dockerfile:

Build the Docker image

docker build -t dataflow:custom .

Inside the container, verify installation

dataflow -v ```

Note: The Docker image includes CUDA 12.4.1 support and comes with vLLM pre-installed for GPU acceleration. Make sure you have NVIDIA Container Toolkit installed to use GPU features.

🤖 6.3 Agent Guided Pipelines

• DataFlow Agent: An intelligent assistant that performs data analysis, writes custom operators, and automatically orchestrates them into pipelines based on specific task objectives.

• 
• [[HuggingFace🤗 demo input & output for DataFlow Agent]](https://huggingface.co/datasets/Open-Dataflow/dataflow-demo-Agent)

</details>

⚡ 7. Quick Start

🚀 7.3 Quick Start with Google Colab

You can start your first DataFlow translation project directly on Google Colab. By following the provided guidelines, you can seamlessly scale from a simple translation example to more complex DataFlow pipelines.

👉 Start DataFlow with Google Colab

configure LLM serving (e.g., OpenAI API)

api key needs to be set via `export DF_API_KEY=sk-xxx`

llm_serving = APILLMServing_request( api_url="https://api.openai.com/v1/chat/completions", )

prompted_generator = PromptedGenerator( llm_serving=llm_serving, # pre-configured LLM backend system_prompt="Please solve this math problem." )

prompted_generator.run( storage=self.storage.step(), # data management (details omitted) input_key="problem", # read from this column output_key="solution" # write to this column )

After running, the operator will append the generated results into output_key. For example, the output data (json/jsonl-style) becomes:

<details> <summary><h2>🛠️ 6. Pipelines (Click to expand)</h2></summary>

📖 7.4 Reference Project Documentation

For detailed usage instructions and getting started guide, please visit our DataFlow Documentation.

<a name="dfwebui"></a>

✅2.1 Ready-to-Use Data Synthesis and Cleaning Pipelines

• High-Quality Training Data Generation
• Text, Math, and Code data generation (see DataFlow-Instruct-10K for results)
• Data generation via tools like AgenticRAG and Text2SQL
• Structured Data Extraction
• Large-scale PDF → QA conversion
• Large-scale book PDF → Visual-QA conversion
• Scientific Data Workflow Management
• Text2SQL workflow management (Accepted by ICDE 2026)
• Math data workflows (Accepted by KDD 2026)

⚙️2.2 Flexible Custom Pipeline Orchestration

• 10+ core operators define interaction patterns and design principles
• 100+ pipeline-specific operators available for reuse or reference
• Full support for creating custom operators — plug-and-play, easily packaged and distributed via GitHub or PyPI

🔧 6.1 Ready-to-Use PipeLines

Current Pipelines in Dataflow are as follows:

• 📝 Text Pipeline: Mine question-answer pairs from large-scale plain-text data (mostly crawed from InterNet) for use in SFT and RL training.
• 
• [[HuggingFace🤗 demo input &amp; output for Text Pipeline]](https://huggingface.co/datasets/Open-Dataflow/dataflow-demo-Text)
• 🧠 Reasoning Pipeline: Enhances existing question–answer pairs with (1) extended chain-of-thought, (2) category classification, and (3) difficulty estimation.
• 
• [[HuggingFace🤗 demo input &amp; output for Reasoning Pipeline]](https://huggingface.co/datasets/Open-Dataflow/dataflow-demo-Reasonning)
• 🗃️ Text2SQL Pipeline: Translates natural language questions into SQL queries, supplemented with explanations, chain-of-thought reasoning, and contextual schema information.
• 
• [[HuggingFace🤗 demo input &amp; output for Text2SQL Pipeline]](https://huggingface.co/datasets/Open-Dataflow/dataflow-demo-Text2SQL)
• 📚 Knowlege Base Cleaning Pipeline: Extract and structure knowledge from unorganized sources like tables, PDFs, and Word documents into usable entries for downstream RAG or QA pair generation.
• 
• 🤖 Agentic RAG Pipeline: Identify and extract QA pairs from existing QA datasets or knowledge bases that require external knowledge to answer, for use in downstream training of Agnetic RAG tasks.
• 

⚙️ 6.2 Flexible Operator PipeLines

In this framework, operators are categorized into Fundamental Operators, Generic Operators, Domain-Specific Operators, and Evaluation Operators, etc., supporting data processing and evaluation functionalities. Please refer to the documentation for details.