kingfisher-scrape for OCDS

kingfisher-scrape is a tool to download OCDS data from various sources, and to store it on disk and/or send it to an instance of kingfisher-process.

It is built using the Scrapy crawler framework.

It can be used standalone, for development or testing of scrapers. For production use, we recommend using scrapyd.

OCP operate a hosted instance of kingfisher-scrape, which is available to OCP staff and the OCDS Team. For information about how to access this, see the hosted kingfisher documentation

The developers have tested kingfisher-scrape on Scrapy Cloud and it works, but the limitations of the service (specifically, not being able to write to a readable file storage and send to the Process API at the same time) mean that it’s not suitable for OCP use. Community users may, however, find this a helpful service.