Pathway: Real-time AI pipelines and a post-transformer future

60,000 messages per second from a pip install

Most Python developers have a complicated relationship with real-time data. You want streaming pipelines, live search, instant updates - but the moment someone mentions Apache Flink or Kafka Streams, you're suddenly reading Java docs and configuring JVM clusters. Pathway looked at that situation and said: what if real-time data processing was just... a Python library?

And it works. But that's only half the story. The same company is also building BDH - a brain-inspired neural architecture that might genuinely challenge the Transformer. Two very different products, one company, and both worth understanding.

What is Pathway?

Pathway is actually two separate things under one roof:

1. Pathway Framework - an open-source Python ETL framework for stream processing, real-time analytics, and RAG pipelines. Think of it as what you'd get if pandas and Apache Flink had a child that was actually pleasant to use.
2. Baby Dragon Hatchling (BDH) - a post-Transformer neural architecture that takes inspiration from how biological brains work. This is the research side.

The framework is the mature, production-ready product. BDH is the ambitious research bet. Let's start with what you can use today.

The framework

Pathway's core idea is simple: write Python, run Rust. Your pipeline code looks like regular Python - familiar syntax, pip install, notebook-friendly - but under the hood it compiles to a Rust engine based on Differential Dataflow that handles multithreading, incremental computation, and distributed processing.

The same code handles both batch and streaming data. No separate pipelines, no mode switching. When new data arrives, Pathway computes only the minimum delta needed - not the entire dataset from scratch.

import pathway as pw

class SensorSchema(pw.Schema):
    device_id: str
    temperature: float
    timestamp: str

# Read from Kafka (streaming) or CSV (batch) - same API
readings = pw.io.kafka.read(
    rdkafka_settings,
    topic="sensors",
    schema=SensorSchema,
    format="json"
)

# Filter anomalies
alerts = readings.filter(
    pw.this.temperature > 85.0
).select(pw.this.device_id, pw.this.temperature)

# Push results to another Kafka topic
pw.io.kafka.write(alerts, rdkafka_settings, topic_name="alerts")
pw.run()

That's a complete streaming pipeline. No cluster setup, no YAML configs, no Docker orchestration. Just Python.

Key numbers

Installation

pip install -U pathway

That's it. No JVM, no cluster manager, no Zookeeper.

Can a web developer actually use this?

Short answer: yes, and there are specific scenarios where it makes a lot of sense.

Pathway isn't a web framework - it won't replace your Nuxt, Next.js, or Django. But it can power the backend intelligence behind your web app. Here's where it fits:

Real-time RAG for your app

If you're building any kind of AI-powered search, chatbot, or document Q&A system, Pathway handles the entire pipeline: data ingestion, document parsing, embedding generation, vector indexing, and retrieval. All in real-time, all synchronized automatically.

No separate vector database needed. Pathway handles vector search natively. When a document changes in your Google Drive or S3 bucket, Pathway detects it, re-parses the content, updates embeddings, and refreshes the index - without you writing a single line of sync logic.

import pathway as pw
from pathway.xpacks.llm import embedders, splitters
from pathway.xpacks.llm.vector_store import VectorStoreServer

# Sync documents from multiple sources
docs = pw.io.fs.read("./documents/", format="binary", with_metadata=True)
gdrive_docs = pw.io.gdrive.read(object_id="folder_id", with_metadata=True)

all_docs = docs.concat(gdrive_docs)

# Split, embed, and serve - all reactive
text_splitter = splitters.TokenCountSplitter(max_tokens=400)
embedder = embedders.OpenAIEmbedder(model="text-embedding-3-small")

vector_server = VectorStoreServer(
    all_docs,
    embedder=embedder,
    splitter=text_splitter,
)
vector_server.run_server(host="0.0.0.0", port=8765)

Your frontend just hits localhost:8765 with a query. The index stays live.

REST API endpoints

Pathway can serve HTTP endpoints directly. You don't need Flask or FastAPI as a separate layer - Pathway has a built-in HTTP connector that accepts requests, processes them through your pipeline, and returns results:

import pathway as pw

webserver = pw.io.http.PathwayWebserver(
    host="0.0.0.0",
    port=9999,
    with_schema_endpoint=True  # auto-generates OpenAPI docs at /_schema
)

class QuerySchema(pw.Schema):
    text: str

queries, response_writer = pw.io.http.rest_connector(
    webserver=webserver,
    route="/search",
    schema=QuerySchema,
    methods=('POST',)
)

# Process queries through your pipeline
results = queries.select(
    query_id=queries.id,
    result=pw.apply(lambda x: search_index(x), pw.this.text)
)

response_writer(results)
pw.run()

Multiple endpoints, one server instance, auto-generated OpenAPI schema. Not bad for a data processing framework.

Live dashboards and monitoring

If your web app needs real-time analytics dashboards - think IoT monitoring, financial data, log analysis - Pathway processes the streams and exposes results that your frontend can consume via HTTP or WebSocket. Pairs well with tools like Streamlit for quick prototyping.

How it compares to the alternatives

The obvious comparison targets are Apache Flink, Kafka Streams, and for RAG specifically - LangChain and LlamaIndex.

Vs. Apache Flink

Flink is the industry standard for stream processing, but it's a Java-first framework. Its Python support (PyFlink) is a wrapper that lags behind the Java API in features. Setting up Flink means configuring a JVM environment, spinning up a cluster, and dealing with a compile-and-submit workflow. Pathway is pip install and you're done.

Performance-wise, benchmarks published by the Pathway team show Pathway outperforming Flink on iterative graph algorithms, with some tests showing 20x speed improvements in streaming scenarios.

Vs. Kafka Streams

Kafka Streams only supports Java and Scala. If you're a Python developer, it's not even an option. And it's tightly coupled to Kafka - your data has to be in Kafka to use it. Pathway connects to 300+ sources.

Vs. LangChain / LlamaIndex for RAG

LangChain and LlamaIndex are excellent for building RAG prototypes. But they're designed around static indexes. You build the index, query it, done. If your documents change, you rebuild. Pathway's RAG pipelines are reactive - changes propagate automatically. For production systems where data freshness matters, that's a significant advantage.

Who's already using it

Pathway isn't just a cool open-source project with GitHub stars. It's running in production at companies you've heard of - and in environments where "eventually consistent" isn't good enough.

Logistics and transport

This is where Pathway seems to have the deepest foothold. DB Schenker - one of the world's largest logistics providers - built a cloud-based application on Pathway for real-time insights on IoT and status data across their logistics fleet. La Poste (French postal service) cut IoT deployment costs by 50% using Pathway for container operations analytics. CMA CGM, the shipping giant, improved container gate-out ETA precision, directly reducing operational costs and handling times. Transdev uses it for real-time passenger information - bus deviations, arrival time estimates, the kind of data that needs to be accurate right now.

Formula 1

F1 uses Pathway to process real-time telemetry streams. The system is flexible enough to allow independent user-defined functions for different business needs - race strategy, broadcast data, performance analysis - all from the same streaming pipeline.

Defense

NATO/JSEC collaborated with Pathway during Steadfast Foxtrot 2024 for data processing and simulation capabilities in military operations across Eastern Europe. When NATO trusts your data processing framework for operational exercises, that's a different level of validation than GitHub stars.

Enterprise AI solutions

Beyond the framework itself, Pathway offers ready-made solutions: document answering systems for enterprise knowledge bases, RAG-powered slide search across SharePoint and Google Drive, and AI contract management tools. These aren't demos - they're products deployed at companies like Intel for internal document discovery.

Baby Dragon Hatchling - the post-Transformer bet

Now for the wild part. Pathway isn't just building data tools - they're trying to replace the Transformer architecture that powers GPT, Claude, and every major LLM.

What is BDH?

Baby Dragon Hatchling (BDH) is a neural architecture inspired by biological brains. The paper's title says it all: "The Missing Link Between the Transformer and Models of the Brain".

BDH is not a language model like GPT. It's an architecture - a fundamentally different way of organizing neural computation. GPT-4, Claude, Gemini - they're all products built on the Transformer architecture. BDH is proposing an alternative foundation.

How it works

In a traditional Transformer, attention is a global operation - every token can attend to every other token. It's powerful but computationally expensive and fundamentally limited in how it handles time.

BDH flips this. Instead of global attention, it uses a population of artificial neurons that interact locally:

• State lives on synapses, not neurons. The connections between neurons carry and update information dynamically
• Local interactions instead of global attention - each neuron mostly talks to its neighbors, and understanding builds step by step
• Hebbian learning during inference - "neurons that fire together, wire together." The network literally rewires itself as it processes data
• No fixed context length - because context emerges from network dynamics, not from position embeddings

The modular structure of the network isn't engineered - it emerges spontaneously during training. Pathway's researchers argue that this emergence is the key to genuine intelligence.

Why it matters

Three properties make BDH genuinely interesting:

Monosemantic synapses - individual connections consistently activate for specific concepts, even across languages. In Transformers, interpretability is a massive unsolved problem. BDH gets it for free.

Continual learning - Transformers are static after training. BDH's synapses update during inference, meaning the model can theoretically learn from new data without retraining. This is the "generalization over time" that Pathway keeps emphasizing.

Scale-free architecture - the network can grow and reason over extended periods predictably. Unlike Transformers where longer contexts degrade performance, BDH's local interaction model scales differently.

Current status

Let's be honest about where BDH is. The current benchmarks show GPT-2 scale parity - competitive loss-versus-parameters scaling at 10 million to 1 billion parameters. That's promising for a new architecture, but it's a long way from competing with GPT-4 or Claude at hundreds of billions of parameters.

BDH is available on GitHub, runs on NVIDIA GPUs, and has a partnership with AWS for cloud deployment. The team is backed by Łukasz Kaiser - one of the co-inventors of the original Transformer architecture ("Attention Is All You Need"). When someone who built the Transformer funds the thing trying to replace it, that's worth paying attention to.

The team behind it

Pathway's founding team has serious credentials:

• Zuzanna Stamirowska (CEO) - complex systems expert, recognized by the US National Academy of Sciences
• Jan Chorowski (CTO) - applied attention mechanisms to speech processing, collaborated with Geoffrey Hinton
• Adrian Kosowski (CSO) - theoretical computer scientist with 100+ published papers
• Backed by Łukasz Kaiser (Transformer co-inventor), TQ Ventures, and Kadmos Capital

This isn't a random startup making bold claims. The scientific bench is deep.

What should you actually do with this?

If you're a web developer who needs real-time data features - live search, synced document indexes, streaming analytics, or RAG pipelines - Pathway's framework is a genuinely useful tool. It's easier than Flink, more capable than LangChain for real-time use cases, and the Python API means you don't need to learn a new language.

If you're interested in AI architecture - keep an eye on BDH. The interpretability and continual learning properties are exactly what the field needs, even if it takes years to reach scale. The paper is worth reading, and the code is open source.

If you're neither - file Pathway away as "interesting company doing important things" and check back in a year. The framework will probably have even more connectors and better RAG tooling by then. BDH might have scaled to something more competitive. Either way, the trajectory is worth watching.