SlipStreamsBot
SlipStreamsBot is the crawler of SlipStreams, a market research tool for online retailers. It reads the product information that online stores publish openly, and nothing else.
It identifies itself as SlipStreamsBot/1.0 (+https://slipstreams.io/bot)
What it fetches
First /robots.txt, before every visit. If that allows it, only these public addresses:
/meta.jsonStore currency, country and shipping countries/products.jsonThe public product catalog, 250 products per page/collections.jsonThe list of collections/collections/<handle>/products.jsonThe order of products within a collection
We store product data: titles, prices, variants, availability and links to images. We link to images; we do not copy them.
What it never does
- Log in, or fetch anything behind a login or password page
- Add items to a cart or touch checkout
- Work around a bot check, a block or a rate limit
- Pretend to be a browser, rotate user agents or use proxies to hide
- Collect personal data about customers or anyone else
How carefully
- One request at a time per store, with at least 1.5 seconds between requests
- At least 1.5 seconds between any two of our requests, across all stores together
- On HTTP 429 or 430 (too many requests) we wait 5, then 20 seconds, or as long as your
Retry-Afterheader asks, up to one minute. After 2 retries we leave your store alone until the next round - When we meet a bot check, we stop all requests, to every store, for 30 minutes
- Most stores are visited about once a day
How to opt out
Add this to your robots.txt:
User-agent: SlipStreamsBot Disallow: /
We read robots.txt before every visit, so this takes effect the next time we come by. We then stop fetching your store, and we do not come back another way. A rule for User-agent: * is respected in the same way.