Web crawler and scraper for Rust
-
Updated
Jul 14, 2025 - Rust
#
Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
Here are 87 public repositories matching this topic...
An ergonomic Rust HTTP Client with TLS fingerprint
tls rust http crawler scraper https websocket http2 http-client fingerprint web-scraper requests web-scraping akamai impersonate ja3 tls-client tls-fingerprint curl-impersonate ja4
-
Updated
Jul 15, 2025 - Rust
A rock-solid cryptocurrency crawler library.
-
Updated
Mar 19, 2023 - Rust
🕷️ The pipeline for the OSCAR corpus
-
Updated
Dec 18, 2023 - Rust
Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.
-
Updated
Jun 15, 2025 - Rust
✴️ An experimental graph database
-
Updated
Jul 20, 2023 - Rust
Spider ported to Python
-
Updated
Jan 28, 2025 - Rust
A command line tool based on the crypto-crawler library.
-
Updated
Mar 19, 2023 - Rust
CLI to download all images/webms in a 4chan thread
-
Updated
Aug 22, 2022 - Rust
🎓a Better BlackBoard for PKUers. 北京大学教学网命令行工具（🖥️Win/🐧Linux/🍏Mac）, 支持查看/提交作业、下载课程回放.
-
Updated
Jun 29, 2025 - Rust
Spider ported to Node.js
-
Updated
Jan 28, 2025 - Rust
Rust Web Crawler saving pages on Redis
-
Updated
Jun 13, 2021 - Rust
⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations
rust crawler scanner bruteforce rust-lang subdomain brute-force pentesting pentest rust-crate pentest-tool subdomain-scanner subdomain-enumeration subdomain-bruteforcing zonetransfer pentesting-tool subdomain-finder searchengines
-
Updated
Jul 14, 2025 - Rust
Crawling and scraping the Web for fun and profit
-
Updated
Apr 13, 2023 - Rust
Download Webtoon
crawler scraper manga hentai manga-downloader manga-scraper hentai-downloader manhwa-scraper manhua-scraper doujin-downloader hentai-scraper tachiyomi-alternative
-
Updated
Mar 29, 2025 - Rust
decentralized web search engine, web archive and feed reader
-
Updated
Jul 13, 2025 - Rust
A smart web crawler built in Rust that uses Claude AI to select the most relevant URLs from website sitemaps based on crawling objectives.
-
Updated
Jul 9, 2025 - Rust
- Followers
- 491 followers
- Website
- github.com/topics/crawler
- Wikipedia
- Wikipedia