Custom data scraping and analysis

Some data you need is not available in any convenient format. It sits inside a public catalogue, a portal, your competitors’ websites, and the only way to get it is to go and take it.

The hard part is not writing the extractor. It is keeping it working when the source changes, and making sure you find out immediately when it stops. An extractor that breaks is loud and you fix it. One that quietly returns half the rows ends up in your reports and stays there for months. So along with the pipeline I hand over the checks on the result: expected coverage, out-of-range values, rows that change shape.

Before extracting I survey the source with an MCP server I wrote for the purpose: structure, technologies, limits, what robots.txt says. It decides whether the job can be done well, before I estimate it.

Reading from the outside also verifies your own systems. To check the state of a catalogue I queried 2,393 products and 9,591 images by walking the site the way a visitor would, without touching the database. That check found a problem the database, internally consistent, could not show.

What I need from you is which source, how often, and what you will do with the data. I do not take work that requires bypassing authentication or anti-bot protection, and I do not work on sources exposing personal data without a legal basis that you state.