Market Demand Pipeline
A five-stage production ETL system that scrapes, normalises and tracks job market demand.
Scraping is easy to demonstrate and hard to run. Three job platforms are collected, LinkedIn, FreeWork and Welcome to the Jungle, through five stages: collect, clean, enrich, control, load. The stages are separated so a failure in one does not corrupt the others, every run is tracked, and the logs are structured because the question asked of a pipeline at three in the morning is always which run and which stage.
Version 2.0.1 went into production in March 2026 and it has run since, maintained and extended by other people. Handing a pipeline over is the test of whether its logs were written for someone else.
Five isolated stages rather than one script: a failure while collecting does not corrupt the warehouse, and every run carries an identifier that follows it through the logs end to end. That is what answers the only question anyone asks at three in the morning: which run, which stage.