Skip to main content
02

Data & Pipelines

Ingestion, pgvector, warehousing, GDPR, anonymisation

An assistant is only ever as good as what it is allowed to read. Almost all of the work sits upstream of the model, in the unglamorous part nobody demos.

  • 5+enterprise systems wired as ingestion sources
  • 4products whose data I modelled for the group warehouse
  • MITAnonDocs, open source, with a public npm SDK

Ingest where the knowledge already lives

SharePoint, Google Drive, Jira, Airtable, CMS platforms. Extraction, chunking, embedding and incremental re-indexing, so an assistant answers from the company’s knowledge rather than from a copy of it that started drifting the day it was made.

Modelling for the people who ask questions later

At Redspher, hand in hand with the data analysts: schemas, extraction and data quality for the application data feeding the group’s BigQuery warehouse, across the four products I owned. At AI SmartTalk, the same discipline turned inward — conversation, usage and knowledge-base schemas structured for analytics from the start.

Contracts at the boundaries, not hope

Zod and JSON Schema validation where data enters, Redis and RabbitMQ where it queues. A pipeline that silently accepts a payload that changed shape will lie to you for a week, and the bug you eventually find will be in the wrong place.

The constraint is the architecture

EU hosting, GDPR alignment, PII stripped before inference, on-premise deployments for clients whose documents cannot reach a third-party API. Bolted on at the end, none of that works. Designed in, none of it costs anything.

Stack

  • PostgreSQL
  • pgvector
  • BigQuery
  • Redis
  • RabbitMQ
  • SharePoint · Drive · Jira · Airtable
  • Zod
  • JSON Schema
  • GDPR · EU hosting