# SEO Site Extractor — self-hosted CORS proxy (Cloudflare Worker)

The [SEO Site Extractor](../../../../seoextractor.html) runs entirely in the
browser, so it can't fetch other origins directly — every request goes through
a CORS proxy. It ships pointed at a free public proxy
(`https://api.allorigins.win/raw?url=`), which is fine for light use but gets
rate-limited and can't report real per-page HTTP statuses.

`worker.js` in this folder is a small Cloudflare Worker you can deploy in a
couple of minutes to get reliable, higher-limit crawls with **accurate
statuses** and **gzipped-sitemap support**.

## What the Worker does

- **GET only**, requires an absolute `http`/`https` `?url=` target.
- **SSRF guard** — rejects `localhost`, `127.0.0.0/8`, `169.254.0.0/16`,
  `10/8`, `172.16–31/12`, `192.168/16` (plus `100.64/10` CGNAT, multicast, and
  IPv6 loopback/link-local/unique-local), and any non-`http(s)` scheme. It
  can't be used to probe internal hosts.
- Fetches with a browser-like `User-Agent` and a 20s timeout; **follows
  redirects**.
- Returns the upstream body with `Access-Control-Allow-Origin: *`, passes the
  upstream `Content-Type` through, and adds:
  - `X-Proxy-Status` — the upstream HTTP status (the tool reads this so per-page
    statuses in the report are accurate)
  - `X-Proxy-Final-URL` — the final URL after redirects
- **Gunzips `.gz` targets** with `DecompressionStream('gzip')`, so gzipped
  sitemaps (`sitemap.xml.gz`) work end-to-end.

## Deploy

### Option A — Wrangler (CLI)

```bash
npm install -g wrangler
wrangler login

mkdir seo-proxy && cd seo-proxy
# copy worker.js from this folder into ./worker.js
cat > wrangler.toml <<'TOML'
name = "seo-proxy"
main = "worker.js"
compatibility_date = "2024-11-01"
TOML

wrangler deploy
```

Wrangler prints the deployed URL, e.g. `https://seo-proxy.<you>.workers.dev`.

### Option B — Dashboard

1. Cloudflare dashboard → **Workers & Pages** → **Create** → **Create Worker**.
2. Replace the starter code with the contents of `worker.js` and **Deploy**.
3. Copy the worker's URL from the deployment screen.

## Wire it into the tool

Open `seoextractor.html` and set the `CORS_PROXY` constant near the top of the
page's `<script>`:

```js
var CORS_PROXY = 'https://seo-proxy.<you>.workers.dev/?url=';
```

The trailing `?url=` matters — the tool appends the URL-encoded target after it.

Once the Worker is in use, the tool automatically picks up `X-Proxy-Status` /
`X-Proxy-Final-URL`, so per-page statuses and redirect handling become reliable
with no further changes.

## Notes

- The Worker is **not** part of the static site deploy — it's an optional,
  separately deployed helper. Cloudflare's free tier (100k requests/day) is
  ample for crawling.
- It only proxies `GET` and never forwards request bodies or cookies.
- To lock it down further, add a check on `request.headers.get('Origin')` and
  allow only your own origin.