有人懂编程吗？网络爬虫（采集器）怎么写？

emucxg

最近想做个网站，一开始想赚点pr


大概思路是，先采集文章，然后分别把标题和内容添加到sql数据库里

每隔一段时间更新一次

以前没弄过采集器，不太懂啊

PnZ.Billy_Cren

试试火车采集器

Bookends

IT码农真屌丝

热啊

我一个朋友做了个类似的电影网站，也没啥意思

xixilee

用php很简单，两个函数搞定

解晓东

呵呵，php可是做不了爬虫啊。要从dns上面下载网站地址列表，然后从各个网站下载html，然后迭代提取网站的链接。自己下载基本搜索引擎的书看看吧。

emucxg

解晓东 发表于 2012-5-22 09:30
呵呵，php可是做不了爬虫啊。要从dns上面下载网站地址列表，然后从各个网站下载html，然后迭代提取网站的链 ...

嗯，我有时间研究研究。谢谢

anomaly

wiki 列大把啊
Open-source crawlers
Aspseek is a crawler, indexer and a search engine written in C++ and licensed under the GPL
DataparkSearch is a crawler and search engine released under the GNU General Public License.
GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used to mirror Web and FTP sites.
GRUB is an open source distributed search crawler that Wikia Search used to crawl the web.
Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java.
ht://Dig includes a Web crawler in its indexing engine.
HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL.
ICDL Crawler is a cross-platform web crawler written in C++ and intended to crawl Web sites based on Web-site Parse Templates using computer's free CPU resources only.
mnoGoSearch is a crawler, indexer and a search engine written in C and licensed under the GPL (Linux machines only)
Nutch is a crawler written in Java and released under an Apache License. It can be used in conjunction with the Lucene text-indexing package.
Open Search Server is a search engine and web crawler software release under the GPL.
Pavuk is a command-line Web mirror tool with optional X11 GUI crawler and released under the GPL. It has bunch of advanced features compared to wget and httrack, e.g., regular expression based filtering and file creation rules.
PHP-Crawler is a simple PHP and MySQL based crawler released under the BSD. Easy to install it became popular for small MySQL-driven websites on shared hosting.
the tkWWW Robot, a crawler based on the tkWWW web browser (licensed under GPL).
YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL).
Seeks, a free distributed search engine (licensed under Affero General Public License).

不过这个年代还用 sql 处理 crawler 数据是不是太落伍了点

新建文件夹

我本科的时候就做过这个玩意。
网页先下载到本地，然后分析里面的链接，放入数据库列表中。
继续按列表中的链接下载。