# penelope

is a web server to controll a firefox with playwright through a REST API.

Problems i was trying to solve with penelope:
- have reliable web scraping tool that can handle JS heavy sites
- have it passive (not polling)
- have it "flexible" enough to MAYBE... one day... eventually... plug an AI on top of it to make it smarter

To make it reliable:
- use a real browser (Firefox) with a persistent profile
- do NOT run it in headless mode (so that it's harder to detect it's a bot)
- access the internet throught my home connection (so that the requests seems to come from my home IP address)

That why i deploy penelope the following way:
- on a minimal server at home (spoiler: server = raspberry)
- connected to my wifi, with VNC server (as last resort if something goes wrong) and wireguard (to access it from outside my home network, tailscale + headscale) 

## To run it
```
cd into penelope
uv sync
uv run playwright install firefox
uv run python server.py
```

## Missing

- Tests
- Reliable deployment (dockerfile / systemd service / uvicorn)
- Authentication (API key?)
- Any safeguards (rate limiting, max payload size, etc)

## Expanding

Eventually, if someone need other "/tools" i guess the right way is to add a class inside /lib and add buttons + endpoints for it, script everything in the new class and voila.

## Notes

- It has a stupidly overengineered cookie banner detection system wrote by Claude.