kingfisher-scrape for OCDS¶
kingfisher-scrape is a tool to download OCDS data from various sources, and to store it on disk and/or send it to an instance of kingfisher-process.
It is built using the Scrapy crawler framework.
It can be used standalone, for development or testing of scrapers. For production use, we recommend using scrapyd. It is possible to use kingfisher-scrape on the hosted scrapycloud service, however the restrictions of this service mean that it’s not suitable for all scrapers.
OCP operate a hosted instance of kingfisher-scrape, which is available to OCP staff and the OCDS Team. For information about how to access this, see the hosted kingfisher documentation
- Use - Standalone
- Use - Scrapyd
- Use - Hosted Kingfisher
- Use - Scrapy Cloud (on Scrapinghub)
- Writing OCDS Kingfisher scrapers with Scrapy
- Using the pipeline