#

Crawler

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Here are 420 public repositories matching this topic...

code4craft / webmagic

A scalable web crawler framework for Java.

java crawler framework scraping

Updated Jul 4, 2025
Java

ssssssss-team / spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

crawler spider web-crawler jsoup xpath webcrawler webspider web-spider spider-flow

Updated Jun 14, 2023
Java

xtuhcy / gecco

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

java crawler dynamic jsoup gecco fastjson

Updated Feb 22, 2024
Java

TeamNewPipe / NewPipeExtractor

NewPipe's core library for extracting data from streaming sites

crawler scraper youtube extractor soundcloud bandcamp newpipe peertube mediaccc

Updated Jul 10, 2025
Java

dadoonet / fscrawler

Elasticsearch File System Crawler (FS Crawler)

java elasticsearch crawler tika

Updated Jul 10, 2025
Java

codelibs / fess

Fess is very powerful and easily deployable Enterprise Search Server.

search java search-engine elasticsearch crawler full-text-search lucene fulltext-search enterprise-search

Updated Jul 10, 2025
Java

apache / stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm

java crawler web-crawler distributed apache-storm stormcrawler

Updated Jul 7, 2025
Java

wycm / zhihu-crawler

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

java crawler spider zhihu

Updated Apr 2, 2019
Java

fanyong920 / jvppeteer

Java API For Chrome and Firefox

java crawler chrome scraper chrome-headless puppeteer jvppeteer

Updated Jun 24, 2025
Java

xuxueli / xxl-crawler

A lightweight web crawler framework.（Java爬虫框架）

java crawler web spider flexible distributed object-oriented xxl-crawler

Updated Jan 5, 2025
Java

fengzhizi715 / NetDiscovery

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

kotlin redis middleware crawler kafka spider dsl coroutines selenium rxjava2 lettuce disruptor htmlunit vertx3

Updated Nov 28, 2020
Java

crawljax / crawljax

Crawljax

javascript crawler dom dynamic crawling test-generation web-testing web-analysis event-driven-crawling

Updated Sep 18, 2023
Java

jaeksoft / opensearchserver

Open-source Enterprise Grade Search Engine Software

search java search-engine enterprise crawler ocr indexing synonyms lucene webcrawler custom-search webcrawling opensearchserver

Updated Sep 3, 2022
Java

smuyyh / CrawlerForReader

Android 本地网络小说爬虫，基于jsoup及xpath

android crawler jsoup xpath bookreader

Updated Sep 2, 2020
Java

commoncrawl / news-crawl

News crawling with StormCrawler - stores content as WARC

crawler news web-crawler apache-storm warc commoncrawl common-crawl storm-crawler

Updated Feb 19, 2025
Java

DragonKingpin / Hydra

Hydra九头龙，面向PB级别知识库取数、情报系统、数据平台、大规模控制调度系统。面向大规模数据采集、分析、智能取数。——以实现大规模分布式爬虫搜索引擎为例。

infrastructure search-engine distributed-systems crawler cloud microservice architecture bigdata operating-system knowledge-graph quant datawarehouse

Updated Jul 8, 2025
Java

yAnXImIN / weiboPicDownloader

免登录下载微博图片爬虫 Download Weibo Images without Logging-in

java crawler weibo

Updated Oct 3, 2024
Java

tim232385 / WebVideoBot

Web crawler.

crawler spider pornhub

Updated Dec 1, 2019
Java

Norconex / crawlers

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

java search-engine crawler flexible web-crawler crawlers filesystem-crawler collector-http collector-fs

Updated Jul 11, 2025
Java

codesofun / web-bee

🐝 Web vertical crawler framework for fun

java crawler framework java-8 webbee

Updated Dec 16, 2023
Java

Followers: 490 followers
Website: github.com/topics/crawler
Wikipedia: Wikipedia