Remember Your Web
Web scrappers are ideal for remembering the web sites you have visited,
Crawl4AI
- GitHub - unclecode/crawl4ai: 🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
- https://docs.crawl4ai.com/
- https://www.youtube.com/@unclecode788
Applications
Others
Jina Reader
Hacked self-hosted
- FYI jin.ai reader can now be self hosted · open-webui/open-webui · Discussion #5789 · GitHub
- GitHub - intergalacticalvariable/reader: 📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simple prefix http://127.0.0.1:3000/https://website-to-scrape.com/
Becareful: no sure how long non-official version will work, in any case do NOT use the official jina.ai version - it is a cloud service.
ScapeGraphAI
Official self-hosted
Anything LLM
They wrote their own web scrapping code e.g.