kingfisher-scrape for OCDS¶
kingfisher-scrape is a tool to download OCDS data from various sources, and to store it on disk and/or send it to an instance of kingfisher-process.
It is built using the Scrapy crawler framework.
It can be used standalone, for development or testing of scrapers. For production use, we recommend using scrapyd.
OCP operate a hosted instance of kingfisher-scrape, which is available to OCP staff and the OCDS Team. For information about how to access this, see the hosted kingfisher documentation
The developers have tested kingfisher-scrape on Scrapy Cloud and it works, but the limitations of the service (specifically, not being able to write to a readable file storage and send to the Process API at the same time) mean that it’s not suitable for OCP use. Community users may, however, find this a helpful service.
- Use - Standalone
- Use - Scrapyd
- Use - Hosted Kingfisher
- Run Report Guide
- Use - Scrapy Cloud (on Scrapinghub)
- Command-line tool
- Writing OCDS Kingfisher scrapers with Scrapy