When we started Stakari, we assumed transaction ingestion would be the boring part. Parse an email, extract the amount, store it. It took about three weeks before we realized how wrong we were.
The Problem With Polling
Our first architecture polled Gmail every five minutes. This sounds fine until your customers expect real-time anomaly alerts. A five-minute polling interval means a fraudulent charge could sit undetected for up to five minutes before it even enters our pipeline, let alone gets processed.
Event-Driven Ingestion
We moved to a push-based model using Gmail push notifications via Google Pub/Sub. When a new email arrives, Google pushes a lightweight notification to our endpoint. We then fetch only that message. This cut our average detection latency from 3.5 minutes to under 4 seconds.
Handling Backpressure
The tricky part is that push notifications don't guarantee ordering or deduplication. We buffer all incoming message IDs into a Kafka topic before processing, which gives us replay capability, ordering guarantees, and natural backpressure handling when the AI classification service is under load.
At peak, we process 800 messages per minute across all tenants. Our p99 end-to-end latency, from email arrival to anomaly evaluation, is 2.1 seconds.
Ndifoin Hilary
Founder