Back

Convoy v1.5

Convoy is a canvas-based data pipeline builder. You drag nodes onto a React Flow canvas (data sources, filters, group-bys, sorts, transforms, computed columns, reshapes, and chart outputs), wire them together, and the pipeline runs. Each node type corresponds to a discrete operation on tabular data: a filter node takes a column, an operator, and a value; a group-by node takes a column and an aggregation function; a chart node takes axis assignments and a chart type. Connections between nodes define the execution order. A data preview collapses in and out of each node so you can inspect what a step is actually producing without leaving the canvas.

A code panel alongside the canvas generates equivalent Python at every change. The output is a pandas script — or a Jupyter notebook if you prefer — that mirrors the node graph one step at a time. It stays in sync automatically, so the canvas and the code are always the same pipeline described two different ways. You can export either at any point, or paste existing Python back in and have the canvas reconstruct the graph from it.

Chart nodes render previews through a Python subprocess rather than anything running in the browser. The Express backend spawns python3 on demand, sends the chart configuration and row data as JSON over stdin, and reads a base64-encoded PNG back from stdout.

function runRenderChart(payload): Promise<{ image: string } | { error: string }> {
  return new Promise((resolve) => {
    const proc = spawn('python3', [scriptPath], { stdio: ['pipe', 'pipe', 'pipe'] });
    let stdout = '';

    proc.stdout?.on('data', (chunk) => { stdout += chunk; });
    proc.stdin?.write(JSON.stringify(payload));
    proc.stdin?.end();

    proc.on('close', (code) => {
      if (code !== 0) { resolve({ error: stderr }); return; }
      const result = JSON.parse(stdout);
      resolve(result.image ? { image: result.image } : { error: result.error });
    });
  });
}

The Python script uses matplotlib with the Agg backend (non-interactive, no display required) and writes the rendered chart back as a base64 string for the frontend to drop into an <img> tag. No Python server stays running between requests; each chart preview spawns and exits its own process. If Python or matplotlib isn't installed, the chart area shows an error, but nothing else in the pipeline breaks.

The AI layer, powered by the Anthropic API, runs through the same backend. It can suggest nodes to add to a partial pipeline, explain what any individual node does, edit a node's configuration from a natural language description, and diagnose why a node is in an error state. Each capability is a separate route with its own prompt; none of them share state with each other or with the chart renderer.