Pathway

Getting started<a id="getting-started"></a>

Installation<a id="installation"></a>

Pathway Live Data Framework requires Python 3.10 or above.

You can install the current release of Pathway Live Data Framework using pip:

$ pip install -U pathway

⚠️ Pathway Live Data Framework is available on MacOS and Linux. Users of other systems should run Pathway Live Data Framework on a Virtual Machine.

Deployment<a id="deployment"></a>

Docker<a id="docker"></a>

You can easily run Pathway Live Data Framework using docker.

Pathway Live Data Framework image

You can use the Pathway Live Data Framework docker image, using a Dockerfile:

FROM pathwaycom/pathway:latest

WORKDIR /app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./your-script.py" ]

You can then build and run the Docker image:

docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app

Run a single Python script

When dealing with single-file projects, creating a full-fledged Dockerfile might seem unnecessary. In such scenarios, you can execute a Python script directly using the Pathway Live Data Framework Docker image. For example:

docker run -it --rm --name my-pathway-app -v "$PWD":/app pathwaycom/pathway:latest python my-pathway-app.py

Python docker image

You can also use a standard Python image and install Pathway Live Data Framework using pip with a Dockerfile:

FROM --platform=linux/x86_64 python:3.10

RUN pip install -U pathway
COPY ./pathway-script.py pathway-script.py

CMD ["python", "-u", "pathway-script.py"]

Use-cases and templates

Ready to see what Pathway Live Data Framework can do?

Try one of our easy-to-run examples!

Available in both notebook and docker formats, these ready-to-launch examples can be launched in just a few clicks. Pick one and start your hands-on experience with Pathway Live Data Framework today!

Example: computing the sum of positive values in real time.<a id="example"></a>

```python import pathway as pw

Contribution guidelines<a id="contribution-guidelines"></a>

If you develop a library or connector which you would like to integrate with this repo, we suggest releasing it first as a separate repo on a MIT/Apache 2.0 license.

For all concerns regarding core Pathway Live Data Framework functionalities, Issues are encouraged. For further information, don't hesitate to engage with Pathway's Discord community.

Define the schema of your data (Optional)

class InputSchema(pw.Schema): value: int

Event processing and real-time analytics pipelines

With its unified engine for batch and streaming and its full Python compatibility, Pathway Live Data Framework makes data processing as easy as possible. It's the ideal solution for a wide range of data processing pipelines, including:

• Showcase: Real-time ETL.
• Showcase: Event-driven pipelines with alerting.
• Showcase: Realtime analytics.
• Docs: Switch from batch to streaming.

AI Pipelines

Pathway Live Data Framework provides dedicated LLM tooling to build live LLM and RAG pipelines. Wrappers for most common LLM services and utilities are included, making working with LLMs and RAGs pipelines incredibly easy. Check out our LLM xpack documentation.

Don't hesitate to try one of our runnable examples featuring LLM tooling. You can find such examples here.

• Template: Unstructured data to SQL on-the-fly.
• Template: Private RAG with Ollama and Mistral AI
• Template: Adaptive RAG
• Template: Multimodal RAG with gpt-4o