步骤/目录： 1.DDNS简介 2.获取域名 3.配置路由器 4.配置端口转发 5.配置光猫 6.DDNS的安全性本文首发于个人博客https://lisper517.top/index.php/archives/54/，转载请注明出处。本文的目的是介绍DDNS，并在路由器上使用DDNS，以达到访问内网设备的目的。本文写作日期为2022年9月30日。1.DDNS简介想要从公网访问自己的电脑或其他设备，笔者之前的文章中曾详细介绍过frp内网穿透，它的使用流程大概是：购买一台有公网IP的云服务器，在云服务器上运行frps，在需要从公网访问的机器上运行frpc，运行frpc时是将一个或多个内网机器的端口投射到云服务器的端口上，比如多个内网机器的22端口都需要开放到公网，那云服务器自己占一个22端口，其他机器就用云服务器的122、222、322等端口。frp是中心化的，优点是你可以将多个不同局域网（比如你在两个地方各买了一套房子，或者其他情况下不得不把需要投射的机器放在几个不相连的局域网中）的机器端口投射到同一个公网ip，安全性也有保障（如果你会设置tls、双向认证的话）；缺点是带宽等于云服务器的带宽，而云服务器的带宽常常不够用，可能只有 5 Mbps 这种速度（ 625 KB/s ），想看看家里的监控都有点卡。本文介绍的DDNS则是另一种路线，使用流程为：向网络运营商申请公网ip（这个公网ip在每次拨号时都会变化，不是固定的。另外，据说电信的公网IP最容易申请，联通次之，移动基本不可能），到DDNS服务商处申请一个域名，在需要开放到公网的机器上安装DDNS客户端即可。它的原理是，每次装有DDNS客户端的内网机器联网时，都会要求DDNS服务商把你的域名解析到这台机器的公网ip上（所以虽然每次拨号时公网ip可能变化，但都能通过域名确定公网ip）。在实践中，一般这个开放到公网的机器是路由器，因为它开机时间长、不怕断电，能把连到路由器上的其他内网机器的端口投射到路由器上，而且现在很多路由器都自带DDNS客户端支持。DDNS是以域名为中心的，优点是网速可以很快，只和你的路由器速度、宽带（百兆还是千兆）相关；缺点是域名选择比较少（但是花一些钱，也可以把DDNS的域名设置为自己的域名，比如下文的no-ip是29.95$/年），所有要投射的机器都要在同一个局域网下，而且安全性上需要一点注意（在openwrt等固件上可以设置DDNS用HTTPS加密，但是路由器自带的固件很少有支持这个的）。申请公网ip这一步，你需要找到当初帮你装宽带的师傅，或者你这一片区的网络负责人，可以告诉他家里要装监控，想申请一个公网ip，如果是电信、联通的基本都会帮你开。另外据说尽量不要通过10000或10010找师傅，因为这样算投诉、要扣工资，而是应该留一个师傅的私人电话或微信联系。本文中，选择红米AX6S路由器，它自带支持花生壳（oray.com）、公云（3322.org）、 dyndns.com 和 no-ip.com 这四家DDNS服务商，其中笔者选择 no-ip.com ，很多路由器都支持 no-ip.com 。不过路由器支持最多的还是花生壳，可惜比较贵，而且用花生壳据说需要提交身份证信息。如果你想跟着本文实操，最好买一样的路由器，或者自己去找支持DDNS的路由器（同时还需要支持端口转发，一般支持DDNS的路由器都支持端口转发），查看该路由器支持哪些DDNS服务商；不要买太便宜的路由器了，两三百还是要出的，表示一下起码的尊重，如果一切顺利，你需要花钱的地方只有买路由器这一步，原本的路由器就支持DDNS的话甚至可以不用花钱。2.获取域名进入 https://www.noip.com/ ，右上角用自己的邮箱注册一个免费账号（高级账号有一些额外功能，但是这里就用免费账号演示一下），注册的时候可以先不填域名。登录后自己跳转到 https://my.noip.com/ ，点击左侧的 Dynamic DNS -> No-IP Hostnames ，点击 Create Hostname ，这里需要输入自己的域名、选择后缀域名，最后的样子是 自己的域名.后缀域名1.后缀域名2 ，意思就是比如 www.bing.com ，这3者当中你能随意改动的只有 www ； bing.com 则是固定的，不过还是可以有几种选择。自己的域名最好独特一点，否则路由器下线的时候这个域名可能被别人注册（虽然概率极低）；后缀域名最好先试试能否访问，有些后缀域名因没有备案而无法访问。接下来点击右侧的 Account -> Account Info ，在 Basic Info 里把Username改一下。这个用户名和密码要记住，设置路由器要用。然后在右侧点击 Account -> Security ，有个二维码，下面有这样一段文字：Account: No-IP.com: 你的邮箱 Key: 你的密钥 Type: time-based这个密钥最好记一下，本文中不需要用，但有些时候可能有用。最后提醒，由于这是no-ip的免费DDNS服务，每30天你会收到一封邮件要求再次激活一下，但这个激活不算太麻烦。如果免费DDNS玩熟练后需要额外功能，请升级到no-ip的高级账号（1.99$/月，后缀域名比免费账号的选择更多）或最高级账号（29.95$/年，可以完全用自己的域名）。3.路由器设置以红米AX6S为例，就用原版固件、不用openwrt等其他固件（目前AX6S的openwrt固件用来看视频还行，玩一些在线游戏就会偶有断网，而且笔者对AX6S开SSH、刷其他固件的需求不强），在 高级设置 -> DDNS 里点击 添加服务 ，选择 no-ip ，用户名密码就填刚才的Username和你自己的密码，主机名称为你选的域名，状态检查和强制更新随便，比如填成5分钟和6小时。最后确定，路由器会尝试连接，等待几秒后 状态 变为绿色的 连接正常 ，就行了。如果是红色的 连接失败 ，就检查一下有没有填错的地方，等一会儿再尝试连接一下，直到成功。另外在 常用设置 -> 局域网设置 -> DHCP静态IP分配 里，把连到路由器的所有机器的内网IP都固定一下，这样转发端口的时候不会乱。最后是一些AX6S的其他推荐设置。在 常用设置 -> 上网设置 -> 上网设置 里，选择 手动配置DNS ，填入Google的两个DNS： 8.8.8.8 和 8.8.4.4 ；开启 IPV6网络设置 ，同样 手动配置DNS ，填入Google的IPV6 DNS服务器，即 2001:4860:4860::8888 和 2001:4860:4860::8844 。然后看看配置的效果，访问 http://test-ipv6.com/ （这个网址挺不错，还可以检测自己能否访问ipv6网址），看一下自己的ipv4地址、ipv6地址。最后提一嘴，在现在的主流浏览器上，比如chrome、edge这些，要访问ipv6的IP而不是域名（解析到ipv6地址的域名）的话，用 https://[ipv6地址]:端口号 类似的格式，比如 http://[2001:4860:4860::8888]:80 （这个网址访问不了，只是举个例子）。4.配置端口转发配置到这里，理论上说已经可以通过域名访问自己的路由器及内网机器了（但是可能只在局域网内，这将在后面说明），我们下面来实验一下。首先是路由器的管理页，你访问 选择的域名 （比如 test.ddns.net ），应该就能访问到路由器登录页面，这已经说明成功了。如果需要进一步实验，把内网机器的端口开放到公网，就要配置端口转发，在AX6S的 高级设置 -> 端口转发 里。以树莓派为例，我想用realvnc访问树莓派桌面，realvnc使用的是5900端口，那么可以新建一个端口转发规则，名称可以写成 pi-realvnc ，外部端口比如是56789，内部ip地址为树莓派连到AX6S后的内网ip，内部端口为5900，然后把树莓派的5900端口打开，在realvnc的客户端里，ip填 选择的域名::56789 ，连接即可。再以笔者之前文章提到的samba（一种NAS的搭建方案）为例，它使用的是445端口，那么ip填运行samba的机器的ip，内部端口填445，外部端口也填445（你也可以填其他外部端口，只是用windows连NAS稍微麻烦一点），在samba客户端中，比如windows机器上，访问 //选择的域名/... 即可。其他就不再举例了。总之，搞清楚自己想开放的机器的ip，搞清楚需要开放哪一个或几个端口，就能非常轻松地从外网访问到自己的内网机器。5.配置光猫如果你家里是通过路由器拨号连到外网，那么就可以跳过这一步；如果你家是通过光纤连光猫（EPON、GPON），然后连到下级路由器上，则此时这个域名只能通过内网访问。具体可以查看AX6S的DDNS设置界面，如果 WAN口 IP 是 192.168. 开头的，那就说明只能内网访问；或者可以到 no-ip官网 ，登录自己的账号，可以看到自己的域名， IP / Target 对应的是 192.168. 开头。这是因为虽然你的光猫能获取一个动态的公网ip，但路由器拿到的是内网ip，在路由器上进行DDNS配置也是把域名解析到一个内网ip，自然无法在外网访问。接下来就需要对光猫和路由器再进行一些设置，但下面的步骤笔者并未进行实操。光猫的设置方法比较多变，建议自行搜索（搜索 光猫型号+超级密码 ），这里以TEWA-708E这个型号的光猫为例：首先去光猫背面查看用户名和密码（或者你自行更改过密码，则是自己改的密码），然后访问光猫的ip，比如 192.168.1.1 ，但是这个ip可能只是一些基础设置，对TEWA-708E这个型号需要访问 192.168.1.1:8080 ，用户名密码是一样的。接下来的操作需要你知道自己上网拨号的账号、密码。在 状态 -> 配置向导 里把光猫的工作模式从 自动拨号 改成 桥接模式 ，这里也能看到上网拨号的账号。然后是AX6S路由器，登录管理页面后，在 常用设置 -> 上网设置 -> 上网设置 里，把DHCP改成PPPoE，账号密码就是你上网拨号的账号密码。简单来说，DHCP是由上级的路由器或光猫等给自己分配网络，PPPoE则是自己拨号上网。成功配置后，在上面的上网信息里，应该能看到 IP地址 从 192.168 开头变成一个公网ip，你可以网页访问 https://www.whatismyip.com/ 或 https://www.ip138.com/ 或 http://www.test-ipv6.com/ 看看这三个网站显示的IPv4地址和AX6S路由器上显示的IPv4地址是否相同。在AX6S的DDNS设置里，也能看到 WAN口 IP 变成了这个公网ip。再等待3~10min（如果用的不是no-ip，有时可能要等30min），当DNS服务器里都把你的域名解析到路由器的这个动态公网ip后，就能通过域名访问到自己的内网。最后提一下，很多光猫有一些隐藏功能，比如TEWA-708E这个型号，你还可以访问 http://192.168.1.1:8080/scdmz.html 这个页面来配置DMZ。DMZ功能需要输入一个ip，光猫会把所有端口的流量都转发到这个ip上，相当于转发所有端口。还有开启telnet（ssh），详情参考 这篇文章 。6.DDNS的安全性前面也提到过DDNS的安全性上需要一些注意，下面主要给出4点建议：定期更换域名，或者定期更改路由器的登录密码（设置的复杂一点）。一般情况下，国内的网络会屏蔽80、443端口，大家可能会把路由器的管理页面放到其他端口。虽然概率不大，但路由器长时间开着，总有些人会用一些脚本扫描no-ip的各种域名或ip，万一不巧扫到你的，他又恰好扫描端口到路由器的管理页面，一个一个试你的密码，说不定就给你试出来了。从内网登录路由器后台。不要在公网登录路由器，因为用的http是不加密的。笔者的解决方法是在端口转发规则里常备一个树莓派的22端口、5900端口转发，需要的时候先ssh登录树莓派、临时打开5900端口，用realvnc的客户端登录树莓派的桌面，浏览器访问192.168.31.1登录小米路由器后台，从内网这里操作路由器页面。这也是对上面一条的补充，因为你并没有在外网改端口开放路由器的管理页面，更安全。使用非常规的端口。在转发端口时，内网机器的端口号可以不改，路由器开放的公网端口建议要改一下（前文中的5900改成56789），而且在1~65535这个范围内尽量大一点，这样别人扫到端口的几率会降低，而且就算扫到也猜不到是用来干什么的。最后，你转发端口时也要注意你的这些服务是否加密，比如SSH就是安全的，realvnc也在传输层有加密，samba用的是用户名密码认证，也算是安全的（只是建议定期修改密码）；如果是redis的6379端口，则不应该直接开放到公网，而是要设置一下redis的tls加密才敢放开。有些路由器的DDNS支持在最外层用https加密，也可以打开，但是需要在DDNS服务商处有一些其他设置或收费。
步骤/目录： 1.Redis简介与安装 2.redis的数据结构 （1）string （2）list （3）set （4）hash （5）zset （6）HLL 3.redis的一些命令 4.redis事务 5.TTL（time to live） 6.排序 7.发布/订阅 8.管道 9.redis GEO 10.其他本文首发于个人博客https://lisper517.top/index.php/archives/53/，转载请注明出处。本文的目的是初步介绍Redis、为爬虫使用redis打好基础，主要整理自 runoob教程 。这里介绍的内容非常浅显，更深入的内容请自行查询书籍。本文写作日期为2022年9月15日。运行的平台为win10和树莓派，编辑器为VS Code。1.Redis简介与安装Redis是一个由 Salvatore Sanfilippo 在2008年开始创造的 key-value 存储系统，是一种开源的非关系型（NoSql）数据库，它的特点有：基于内存，分布式，键值对，支持多种语言的API（Redis本身是由ANSI C写的）。虽然它主要在内存中运行，但也可以将数据保存在磁盘；它支持主从式的数据备份；它的读写性能极高（所以把Redis开放到公网上时还要注意安全性）；它支持事务，多个操作可以打包到一起，要么不执行，要么全部执行（all or nothing）；它支持5种数据类型（字符串，列表，hash散列，集合，有序集合）；它支持推送，key过期。总的来说Redis就是好。在使用Redis时，你可以把它用作主数据库，也可以用作辅数据库，不论何种方式，都是对关系型数据库功能的极大补充。由于redis的一些特性，它特别适合做存储标签这种一对多的工作，而存储标签在mysql这样的关系型数据库中比较麻烦；redis可以设置键的过期时间TTL（time to live），到期后自动过期，这是很多服务都需要的（比如登录cookie的过期）；redis的读写效率非常高，在轻度使用中很少会遇到redis是瓶颈的情况；redis的命令比sql要简单得多（因为它以键值对形式存储数据）。下面介绍三种安装redis的方法。需要注意，redis为了效率而牺牲了安全性，所以默认情况下redis几乎没有保护措施，不要把它开放到外网。在windows上安装redis：window版redis由热心网友维护，版本远落后于linux版或docker版（目前是5.0和7.0的差距），所以不推荐下载。需要的话，建议在windows上用docker安装redis，见 Docker入门（一） 。如果嫌麻烦的话，到 windows版redis下载网页 下载zip文件或msi文件，解压或安装即可。解压zip文件，或用msi文件安装后，在cmd中进入redis目录，运行 redis-server.exe redis.windows.conf 即可运行redis服务器端。然后在同样的目录下，运行 redis-cli.exe -h 127.0.0.1 -p 6379 即可连接到redis数据库。但是要注意，目前它的版本只有5.0，笔者在后续将演示redis使用tls双向验证，但这是redis 6.0后才加入的功能。所以有条件的话还是用docker装redis。在linux上：到 https://download.redis.io/releases/ 下载tar.gz文件，解压、 make 即可。具体过程为：mkdir /usr/local/redis cd /usr/local/redis wget https://download.redis.io/releases/redis-7.0.4.tar.gz #最新版本的redis tar -xzf redis-7.0.4.tar.gz cd redis-7.0.4 apt-get install pkg-config make gcc libssl-dev #一些依赖项 make BUILD_TLS=yesBUILD_TLS=yes 这个选项在redis 6.0以后可用，是用tls在传输层对redis通信加密的一种方式，在后续安全性的部分会提到。这里介绍一下如何用docker安装redis（安装docker、docker-compose请参考 Docker入门（一） 和 Docker入门（二））。在linux上使用如下命令（windows上也是一样的步骤，只是路径改一下）：mkdir -p /docker/redis/data/redis vim /docker/redis/docker-compose.yml写入如下内容：version: "3.9" services: redis: image: redis ports: - "56379:6379" volumes: - /docker/redis/data/redis:/data logging: driver: json-file restart: always然后运行：cd /docker/redis docker-compose config docker-compose up -d然后你可以使用redis客户端尝试连接，比如本文后面将用的python，但是首先需要 pip install redis 。然后写一个py文件：import redis, pprint r = redis.StrictRedis(host='127.0.0.1', port=56379, decode_responses=True, db=0) pprint.pprint(r.info())运行，如果成功连接的话它将会打印出redis服务器的一些信息。在python中，更好的连接方式是使用连接池：import redis pool = redis.ConnectionPool(host='127.0.0.1', port=56379, decode_responses=True, db=0) r = redis.Redis(connection_pool=pool)或者你也可以使用redis-cli连接，这是redis服务器端自带的软件。比如你可以进入docker-redis容器：cd /docker/redis docker-compose exec -it redis bash redis-cli -h localhost -p 6379 SAVESAVE 将返回一个 OK ，并且在/docker/redis/data下出现一个dump.rdb文件。实际上， SAVE 会把redis在内存中的数据写到磁盘中，这会在后面提及。更常用的测试连接命令为 ping ，服务器会返回一个 PONG 表示服务器已连接。另外， clear 可以清屏， exit 可以断开连接。最后，如果你想在redis中看到中文，使用 redis-cli --raw 来发起连接。除了redis-server、redis-cli，你还能找到redis-benchmark（性能测试工具，使用方法同redis-cli）、redis-check-aof（AOF文件修复）、redis-check-dump（RDB文件检查）。另外，如果想让redis把内存数据写到硬盘并关闭服务器，可使用 redis-cli SHUTDOWN 命令（由于docker-compose.yml里写了 restart: always ，所以它还会重启）。最后再提醒一下，目前的redis安装方式，效率极高，但安全性几乎没有，不要把它开放到外网了。后续将介绍如何通过一些方法提升redis的安全性。安装好redis后，使用 select 数据库编号 来选择一个数据库。默认情况下redis有16个database，从0~15编号，不支持自定义数据库名（你需要自己记住每个库里存了什么），不支持用户对单独一个数据库有权限。2.redis的数据结构正如前面提到过，redis中有5种数据结构：string字符串，list列表，hash散列，set集合，zset有序集合，接下来将一一介绍它们。下面介绍的操作是一小部分，更多操作参考 redis官网 。（1）string首先是字符串，它是redis最基本的类型，一般的格式就是一个键对应一个字符串。它还可以存储二进制数据，比如图片等，但它最大只能存储512MB（毕竟redis跑在内存里）。你可以在redis-cli尝试下面3种命令（set，get，del）：set test 123 OK get test "123" del test (integer) 1 get test (nil) del test (integer) 0set用于给键赋值，get用于取出键对应的值，del用于删除一个键。在上面的例子中，如果你在python中运行，代码如下：import redis r = redis.StrictRedis(host='localhost', port=536379, decode_responses=True, db=0) a = [] a.append(r.set('test', '123')) a.append(r.get('test')) a.append(r.delete('test')) a.append(r.get('test')) a.append(r.delete('test')) for i in a: print(i)这里把每个返回的值都打印了出来，注意观察对应关系。另外，如果字符串是纯数字的话，redis提供了 INCR ，用于让其自增1，这个函数返回自增后的数字（如果键不存在，会自动创建一个为 "0" 的值并自增为1）。常见的例子是用于存储阅读量，比如键为 page.view ，在里面存储阅读量。注意在redis键的命名中，要力求写详细、直观，一般用 : 或 . 来分隔。在mysql中对列可用指定 AUTO_INCREMENT 来让列值自增，在redis中则可以用 INCR 达到类似的效果。 INCRBY 可达到相同的效果，但是它可以指定增加多少。同系列的命令还有 DECR 、 DECRBY （decrby每太大用，因为incrby可以设置成负值）， INCRBYFLOAT （增加浮点数）。其他关于字符串的命令还有：APPEND ，用于向字符串的末尾追加字符串。STRLEN ，获取字符串长度。注意UTF-8编码中一个中文字符的长度为3。MSET 和 MGET 同时设置、获取多个键值。位操作，包括 GETBIT （获得字符串的一个bit位，超出范围时返回0）、 SETBIT （返回旧值）、 BITCOUNT （返回为1的bit个数）、 BITOP （进行位运算，包括 AND 、 OR 、XOR 、 NOT）。 使用位操作，可以让存储空间非常紧凑。最后，下面介绍的其他类型都是字符串衍生出来的，它们的键值只能是字符串，不能嵌套。（2）list下面介绍一下列表。列表是有序的，可以存储多个字符串，在redis-cli中它支持的方法有：lpush 在列表头部添加一个元素 rpush 在列表末尾添加一个元素 lrange 获取从头到尾、一定范围的元素（包含两端；支持负索引） ltrim 用法和lrange一样，但是它将会删除范围外的元素 lindex 获取从头到尾的第几个元素。支持负索引 lset 给出索引并赋值指定元素 lpop 从列表头部弹出一个或几个元素（弹出，意为返回这个元素的值，并且从列表中删除它） rpop 从列表末尾弹出一个或几个元素 llen 获取列表中元素个数 lrem 删除列表中的几个元素，这些元素的值需要指定。格式为 LREM key count value，count指定要删除几个，当count为正时从左边开始删，为负时从右边开始删，0时删除所有 linsert 在一个元素的前或后插入元素 rpoplpush 将旧列表转移到新列表 brpop 类似rpop，区别在于列表内无元素时会被阻塞（每当列表加入一个新元素时brpop就会返回这个元素）。常用于实现任务队列（可带优先级）如果熟悉数据结构的话，会发现这里的列表可以当作栈或队列使用。在redis-cli中实验如下：lpush test 123 (integer) 1 lpush test 234 (integer) 2 lpush test 345 (integer) 3 rpush test 456 (integer) 4 lrange test 0 -1 1) "345" 2) "234" 3) "123" 4) "456" lindex test 0 "345" lpop test "345" rpop test 2 1) "456" 2) "123" lrange test 0 -1 1) "234" del test (integer) 1除了这些命令，redis对列表还有其他操作，将在后面介绍。另外，一个列表最多可存储 2^32-1 个元素，即 4,294,967,295 。（3）set列表中可以出现相同的元素，且有序，但集合中每个元素不能相同，且set是无序的（set里的元素只有键没有值）。它支持的命令有：sadd 添加一个元素到集合 srem 从集合中移除一个元素 sismember 检查元素是否在集合中 smembers 返回集合的所有元素。这个命令在集合较大时运行很慢 sdiff 差集 sinter 交集 sunion 并集 sdiffstore 存储运算结果 sinterstore 同上 sunionstore 同上 scard 获得集合中元素个数 srandmember 随机获取一个元素。它可以添加count参数，当count为正时取出count个不重复元素，为负时表示可以重复。但是由于redis的底层机制，某些元素出现的概率可能更大 spop 随机弹出一个或多个元素使用例：sadd test 123 (integer) 1 sadd test 234 (integer) 1 sadd test 234 #添加重复元素时被忽略了 (integer) 0 sadd test 345 (integer) 1 sadd test 456 (integer) 1 srem test 123 (integer) 1 sismember test "123" (integer) 0 sismember test "234" (integer) 1 smembers test 1) "234" 2) "345" 3) "456" del test (integer) 1redis中的集合使用哈希表实现（只有键的hash），它增、删、查元素的复杂度都是O(1)。除了这些命令，redis对集合还有其他操作，将在后面介绍。另外，一个集合最多也可存储 2^32-1 个元素，即 4,294,967,295 。在一般的实践中，常用set来存储tag，或者用列表。比如，键名为 tag:标签名:posts ，值为set，存储带有该标签的文章id（4）hash散列是存储键值对的集合，它存储的值可以为字符串或数字（不能嵌套字符串以外的类型）。hash的命令有：hset 在hash中添加或修改一个键值对 hget 获取hash中指定键的值 hmset 添加或修改多个键值对 hmget 获取多个键值 hsetnx 仅当键不存在时新建一个键 hincrby 类似字符串的 incrby 操作 hgetall 获取hash中所有键值对 hexists 判断键是否存在 hdel 如果hash中存在一个或多个键，删除它 hkeys 获得所有键名 hvals 获得所有值 hlen 获得键值对数量使用例：hset test 1 111 (integer) 1 hset test 2 222 (integer) 1 hset test 3 333 (integer) 1 hset test 1 123 (integer) 0 hget test 1 "123" hgetall test 1) "1" 2) "123" 3) "2" 4) "222" 5) "3" 6) "333" hdel test 1 (integer) 1 hget test 1 (nil) del test (integer) 1一个hash可以看作mysql中的一行，不同之处在于每行包含的列可以不同。hash比较适合存储对象；它同样能存储 2^32-1 个键值对，即 4,294,967,295 。现在可用比较一下string和hash。比如你想存储一篇文章，需要存储的信息有文章名，文章url，文章内容等。那么你此时有两种选择，一是把文章对象序列化、转换为二进制数据，用string存储；二是用hash，把它当成mysql中的一行。用hash显得更直观，也不需要在读写是进行编解码。（5）zset有序集合，它可以看作是介于列表和集合之间，因为zset中的元素是唯一的，但它里面的元素有序；但它又像散列，存储键值对（但是zset中的键称为member，值称为score，值必须为浮点数）。zset支持的操作如下：zadd 添加键值对到zset（ +inf 和 -inf 表示正无穷和负无穷） zrange 取出范围内的成员值（键值），按score从小到大 zrevrange 同上，但是score从大到小 zrangebyscore 给定score范围，取出所有在范围内的元素（包含极值。如果不想包含，可加上 ( 如 "80 (100"） zrem 如果zset中存在一个或多个成员，删除它 zscore 获取元素的分数 zincrby 增加score zcard 获取元素个数 zcount 获取分数在范围之间的元素个数 zremrangebyrank 按照元素score从小到大的顺序删除范围内的元素 zremrangebyscore 删除指定分数范围的元素 zrank 获取元素的排名，从小到大排列 zrevrank 同上，从大到小 zinsertstore 计算并导出有序集合的交集。新zset中的分数会使用聚合函数，支持sum、min、max。还能设置权重 zunionstore 计算并导出并集使用例：zadd test 123 1 (integer) 1 zadd test 234 2 (integer) 1 zadd test 345 3 (integer) 1 zrange test 0 -1 1) "1" 2) "2" 3) "3" zrangebyscore test 124 345 1) "2" 2) "3" zrem test 1 (integer) 1 del test (integer) 1它也支持 2^32-1 个键值对。在应用场景上，一般string用于存储文件、文字；hash用于存储对象（描述对象的各种属性），可以当成mysql中的一行来使用；list用于列表、队列；set可用于去重；zset用于排行、权重等。（6）HLL在2.8.9版本中，redis添加了HyperLogLog，这种结构专用于数据量很大的集合，在一定误差内返回集合中元素的个数（不能返回元素本身），它主要用于计算视频点击量（上千万、上亿次）等。它的特点有：占用空间小，一个HLL键在redis中只占12KB，就能统计2^64量级的元素；使用了一些概率算法，标准差在 0.81% 左右（可以设置辅助因子降低，但要付出空间）。redis中HLL相关命令有：pfadd 添加元素到HLL中 pfcount 返回个数估算值 pfmerge 将多个HLL合并为一个HLL3.redis的一些命令除了前面提到的 ping ， exit ， clear ， save ， shutdown ， select ，数据类型相关的命令，还有一些命令：keys 模式 ，返回所有匹配模式的键。在模式中， ? 匹配一个字符， * 匹配0到任意个字符， [] 用于给出单个字符的范围， \ 用于转义。DEL ，删除一个或多个键。不能用通配符，但是可在linux命令行借助管道与xargs实现类似的功能： redis-cli KEYS "模式" | xargs redis-cli DEL 。DUMP ，把一个键序列化，类似python中decode的功能。EXISTS ，检查键是否存在。EXPIRE ，为键设定过期时间（单位为秒）。到过期时间后，键会被删除。EXPIREAT key timestamp ，和 EXPIRE 功能相同，但是使用UNIX时间戳（从1970年1月1日到现在的秒数）。PEXPIRE ，和 EXPIRE 功能相同，只是单位为毫秒。PERSIST ，取消键的过期时间。TTL （time to live），以秒为单位返回键的剩余时间。PTTL ，以毫秒为单位返回键的剩余时间。KEYS ，给出一个pattern，返回符合模式的键。MOVE ，把键转移到另一个库中（redis中默认有16个库）。RANDOMKEY ，返回一个随机键。RENAME ，修改键名。RENAMENX ，仅当要改成的键名不存在时，将旧键名改为新键名。SCAN ，返回可迭代的数据库键（数据库编号）。TYPE ，返回键对应值的类型。更多命令可参考 redis官网 。4.redis事务事务（transaction）是现在很流行的一个概念。设想我进行一个转账1000￥的操作，银行的数据库系统需要进行两步：从我的账户扣去1000，然后给目标账户增加1000。但是如果中间出了差错，比如我的账户扣钱、对方账户却没有收到钱，这怎么办呢？这种类型的问题可用事务解决，包括在事务内的多条命令要么全部执行，要么全部都不执行，不存在执行一半的状态。在redis-cli中执行事务如下：multi 命令1 命令2 ... execredis先接受各条命令，当接收到exec时redis会将所有命令执行。如果在客户端输入命令时断线，redis就不会执行命令。另外，事务不会被其他命令打断，redis在执行一个事务时会把它执行完，再执行其他命令或事务；redis中的事务也不支持回滚。另外，如果事务中的某条命令有错误，其他命令也会继续执行。事务相关的命令除了 multi 和 exec ，还有：discard 取消事务 watch 监视键 unwatch 放弃监视键watch 是事务的重要功能之一，它用于监视键，保证在事务执行前相关的一些键不被改动，一般格式为：watch multi exec如果watch监视到了键被改动，事务的执行将被阻止（watch的作用不是阻止键被改动，而是阻止事务执行）。5.TTL（time to live）这里介绍一下redis的键过期功能。使用 expire 可以为一个键设置过期时间，单位为秒，到时间后该键会自动被删除。使用 ttl 则可以返回一个键剩余的时间，单位为秒（当键不存在时返回-2，键没有设置过期时间时返回-1）。使用 persist 可以取消键的过期时间，成功取消时返回1，否则返回0。另外，如果用 set 、 getset 等命令为键赋值，也会取消过期时间（incr 、 lpush 、 hset 、 zrem 等修改键值的命令则不会）。pexpire 以毫秒设置ttl。对应的有 pttl 。最后，还有 expireat 和 pexpireat ，它们给出的是UNIX时间，到该时间后删除键。另外，如果键是被expire相关命令删除，watch不会认为键被改动过。使用ttl功能，可以方便地为网页提供cookies等服务。6.排序redis中使用 sort 排序（升序），用 sort key desc 降序排序，用 sort key alpha 对字符串排序。在hash中，使用 sort 键名 by 参考键名*->字段名 可以指定排序的字段，redis会用字段值来替换 * ，by只能出现一次； sort by GET 参考键名*->字段名 可以返回指定的字段，get可以出现多次，用 # 可以代替元素本身的值； sort by get STORE 键名 可以把结果保存到新的键中。7.发布/订阅redis中提供了发布、订阅的功能。类似youtube，发布者向频道（channel）中发布一个消息，订阅者会收到这个消息。发布消息的命令为 publish ，订阅频道的命令为 subscribe 。另外还有以下命令：unsubscribe 可以取订一个或所有频道；psubscribe 按一定的模式订阅频道。支持的匹配符有 ? 、 * 等；punsubscribe 按模式退订频道。在分布式的scrapy中，主机发布待爬url就可以用publish，所有订阅url频道的机器就能收到待爬url。另外，redis 5.0后新增了stream，它主要用于消息队列。publish的一个特点是不会存储之前发布的消息，就是说新订阅的用户不能接受到订阅之前发布的消息。stream就是为了解决这个问题而出现的，它对历史发布的消息进行了持久化8.管道在redis的使用中，上面提到的方法都有返回值。日常使用中，redis-cli发出指令，server收到后执行并返回结果，如果client要执行多条语句，有时只需要最后的值，就可以使用管道。管道应用于多条命令，每条命令的执行不依赖于前面的命令，只需要最后的结果时。管道的具体实现和编程语言有关，比如用python：import redis pool = redis.ConnectionPool(host='127.0.0.1', port=56379, db=0, decode_responses=True) r = redis.Redis(connection_pool=pool) pipe = r.pipeline() pipe.set('test', '123') #pipe的其他操作 pipe.execute()你会发现这和pymysql差不多。使用管道，即使在局域网中，也可以大大降低延迟。9.redis GEO在redis 3.2版本新增了geo功能，主要用于计算实际的地理位置。这里就不过多介绍了。10.其他save 命令可将内存中的数据写入磁盘，默认情况下是dump.rdb文件。恢复数据时将这个文件放到redis安装目录（用 config get dir 查看）即可（docker-redis则是放到容器的/data下）。bgsave 在后台执行save。client 相关操作，包括 client list （查看所有连接的客户端）、 client setname （设置当前连接的名称）、 client getname （获取连接名称）、 client pause （客户端挂起多少毫秒）、 client kill 关闭客户端连接。redis支持分区，就是把一个很大的键分成小份，每个redis实例只保存它的一个子集，这个技术稍微有些复杂，不拓展了。redis命令行使用Lua语言，并且支持编写脚本（如同mysql）。客户端提供一个脚本后，其会永久存储在redis中，便于其他客户端使用。Lua的语法这里也不介绍了。config 系列命令，可获取server的配置。redis支持主从模式，这主要是因为一台机器的资源有限。当需要扩展时，你可以在一台机器上运行主redis，其他机器上运行从redis；主redis负责写入的操作，从redis会把数据拷贝到本地、负责读取的操作，也称为读写分离。最后，要提醒一下：注意redis的安全性；安装redis的时候提到过 make BUILD_TLS=yes 这个选项，但是redis应该尽量追求效率，只在内网运行，尽量不用tls等安全手段，因为会降低redis的速度。redis内部使用Lua语言，有时你使用爬虫入库的东西可能被当成脚本执行，比如说给你把redis清空（ flushall ），这点要注意（反爬手段之一）；如果想更深入学习redis，可以参考 redis的官方文档 或 《redis实战》 等书籍。
本文较长，属于系统性总结，可作为使用手册查阅。步骤/目录： 1.Debugging Spiders 2.Common Practices 3.Broad Crawls 4.使用浏览器的开发者工具 5.下载图片与文件 6.AutoThrottle extension 7.Benchmarking 8.爬虫的暂停与恢复 9.scrapy架构 10.下载中间件 11.爬虫中间件 12.添加extensions 13.其他本文首发于个人博客https://lisper517.top/index.php/archives/52/，转载请注明出处。本文的目的是详细介绍scrapy，主要翻译自 scrapy官方文档 。本文写作日期为2022年9月15日。运行的平台为win10，编辑器为VS Code。在上一篇文章中，介绍了scrapy的基础概念，现在讲解一些进阶的用法和技巧。1.Debugging Spiders介绍了一些调试爬虫的方式，主要有：（1） scrapy parse 命令。（2） scrapy.shell 类，使用例：from scrapy.shell import inspect_response def parse_details(self, response, item=None): if item: # populate more `item` fields return item else: inspect_response(response, self)（3） scrapy.utils.response 类，可以打开浏览器：from scrapy.utils.response import open_in_browser def parse_details(self, response): if "item name" not in response.body: open_in_browser(response)（4）Logging。（5）Spiders Contracts。2.Common Practices如何使用爬虫。内容包括：（1）使用 scrapy crawl 以外的方式运行爬虫，比如运行多个爬虫或全部爬虫：from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings from scrapy.spiderloader import SpiderLoader # 根据项目配置获取 CrawlerProcess 实例 process = CrawlerProcess(get_project_settings()) # 获取 spiderloader 对象，以进一步获取项目下所有爬虫名称 spider_loader = SpiderLoader(get_project_settings()) # 添加全部爬虫 for spidername in spider_loader.list(): process.crawl(spidername) # 添加一个爬虫 #process.crawl('爬虫名') # 执行 process.start()（2）分布式爬取，涉及scrapyd。（3）避免被ban ip，原文的建议有：rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them) disable cookies (see COOKIES_ENABLED) as some sites may use cookies to spot bot behaviour use download delays (2 or higher). See DOWNLOAD_DELAY setting. if possible, use Google cache to fetch pages, instead of hitting the sites directly use a pool of rotating IPs. For example, the free Tor project or paid services like ProxyMesh. An open source alternative is scrapoxy, a super proxy that you can attach your own proxies to. use a highly distributed downloader that circumvents bans internally, so you can just focus on parsing clean pages. One example of such downloaders is Zyte Smart Proxy Manager3.Broad Crawls目前为止介绍的都是focus爬虫，即聚焦于某一个站点。如果是google、baidu等搜索引擎，它们会用Broad爬虫，这部分介绍了如何把scrapy改造成Broad爬虫，有兴趣的可阅读原文。Scrapy defaults are optimized for crawling specific sites. These sites are often handled by a single Scrapy spider, although this is not necessary or required (for example, there are generic spiders that handle any given site thrown at them). In addition to this “focused crawl”, there is another common type of crawling which covers a large (potentially unlimited) number of domains, and is only limited by time or other arbitrary constraint, rather than stopping when the domain was crawled to completion or when there are no more requests to perform. These are called “broad crawls” and is the typical crawlers employed by search engines. These are some common properties often found in broad crawls: they crawl many domains (often, unbounded) instead of a specific set of sites they don’t necessarily crawl domains to completion, because it would be impractical (or impossible) to do so, and instead limit the crawl by time or number of pages crawled they are simpler in logic (as opposed to very complex spiders with many extraction rules) because data is often post-processed in a separate stage they crawl many domains concurrently, which allows them to achieve faster crawl speeds by not being limited by any particular site constraint (each site is crawled slowly to respect politeness, but many sites are crawled in parallel) As said above, Scrapy default settings are optimized for focused crawls, not broad crawls. However, due to its asynchronous architecture, Scrapy is very well suited for performing fast broad crawls. This page summarizes some things you need to keep in mind when using Scrapy for doing broad crawls, along with concrete suggestions of Scrapy settings to tune in order to achieve an efficient broad crawl. Use the right SCHEDULER_PRIORITY_QUEUE¶ Scrapy’s default scheduler priority queue is 'scrapy.pqueues.ScrapyPriorityQueue'. It works best during single-domain crawl. It does not work well with crawling many different domains in parallel To apply the recommended priority queue use: SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue' Increase concurrency¶ Concurrency is the number of requests that are processed in parallel. There is a global limit (CONCURRENT_REQUESTS) and an additional limit that can be set either per domain (CONCURRENT_REQUESTS_PER_DOMAIN) or per IP (CONCURRENT_REQUESTS_PER_IP). Note The scheduler priority queue recommended for broad crawls does not support CONCURRENT_REQUESTS_PER_IP. The default global concurrency limit in Scrapy is not suitable for crawling many different domains in parallel, so you will want to increase it. How much to increase it will depend on how much CPU and memory you crawler will have available. A good starting point is 100: CONCURRENT_REQUESTS = 100 But the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%. Increasing concurrency also increases memory usage. If memory usage is a concern, you might need to lower your global concurrency limit accordingly. Increase Twisted IO thread pool maximum size¶ Currently Scrapy does DNS resolution in a blocking way with usage of thread pool. With higher concurrency levels the crawling could be slow or even fail hitting DNS resolver timeouts. Possible solution to increase the number of threads handling DNS queries. The DNS queue will be processed faster speeding up establishing of connection and crawling overall. To increase maximum thread pool size use: REACTOR_THREADPOOL_MAXSIZE = 20 Setup your own DNS¶ If you have multiple crawling processes and single central DNS, it can act like DoS attack on the DNS server resulting to slow down of entire network or even blocking your machines. To avoid this setup your own DNS server with local cache and upstream to some large DNS like OpenDNS or Verizon. Reduce log level¶ When doing broad crawls you are often only interested in the crawl rates you get and any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when preforming large broad crawls in production. Using DEBUG level when developing your (broad) crawler may be fine though. To set the log level use: LOG_LEVEL = 'INFO' Disable cookies¶ Disable cookies unless you really need. Cookies are often not needed when doing broad crawls (search engine crawlers ignore them), and they improve performance by saving some CPU cycles and reducing the memory footprint of your Scrapy crawler. To disable cookies use: COOKIES_ENABLED = False Disable retries¶ Retrying failed HTTP requests can slow down the crawls substantially, specially when sites causes are very slow (or fail) to respond, thus causing a timeout error which gets retried many times, unnecessarily, preventing crawler capacity to be reused for other domains. To disable retries use: RETRY_ENABLED = False Reduce download timeout¶ Unless you are crawling from a very slow connection (which shouldn’t be the case for broad crawls) reduce the download timeout so that stuck requests are discarded quickly and free up capacity to process the next ones. To reduce the download timeout use: DOWNLOAD_TIMEOUT = 15 Disable redirects¶ Consider disabling redirects, unless you are interested in following them. When doing broad crawls it’s common to save redirects and resolve them when revisiting the site at a later crawl. This also help to keep the number of request constant per crawl batch, otherwise redirect loops may cause the crawler to dedicate too many resources on any specific domain. To disable redirects use: REDIRECT_ENABLED = False Enable crawling of “Ajax Crawlable Pages”¶ Some pages (up to 1%, based on empirical data from year 2013) declare themselves as ajax crawlable. This means they provide plain HTML version of content that is usually available only via AJAX. Pages can indicate it in two ways: by using #! in URL - this is the default way; by using a special meta tag - this way is used on “main”, “index” website pages. Scrapy handles (1) automatically; to handle (2) enable AjaxCrawlMiddleware: AJAXCRAWL_ENABLED = True When doing broad crawls it’s common to crawl a lot of “index” web pages; AjaxCrawlMiddleware helps to crawl them correctly. It is turned OFF by default because it has some performance overhead, and enabling it for focused crawls doesn’t make much sense. Crawl in BFO order¶ Scrapy crawls in DFO order by default. In broad crawls, however, page crawling tends to be faster than page processing. As a result, unprocessed early requests stay in memory until the final depth is reached, which can significantly increase memory usage. Crawl in BFO order instead to save memory. Be mindful of memory leaks¶ If your broad crawl shows a high memory usage, in addition to crawling in BFO order and lowering concurrency you should debug your memory leaks. Install a specific Twisted reactor¶ If the crawl is exceeding the system’s capabilities, you might want to try installing a specific Twisted reactor, via the TWISTED_REACTOR setting.4.使用浏览器的开发者工具永远不要在xpath表达式里带tbody标签。浏览器对页面的源码会做一些修改（排版美化，执行js）、scrapy不会，这将导致浏览器看到的源码和scrapy中不同，其中一个点就是firefox浏览器喜欢在表格中加tbody标签。其他的部分笔者也不介绍了，其他地方有更详细的使用开发者工具教程，最好是找一些图文教程或视频教程看。5.下载图片与文件使用image管道和files管道。它们的特点有：避免重复下载（会scrapy会给文件生成hash身份码用于去重），指定下载地点（本地目录，FTP服务器，google云）。image管道还有另外的特点，包括自动转换图片为jpg格式、RGB模式（这对下载gif图片有些麻烦），生成图片指纹（thumbnail），对下载图片的长、宽进行限制。官方文档关于使用Files管道的流程解说：The typical workflow, when using the FilesPipeline goes like this: In a Spider, you scrape an item and put the URLs of the desired into a file_urls field. The item is returned from the spider and goes to the item pipeline. When the item reaches the FilesPipeline, the URLs in the file_urls field are scheduled for download using the standard Scrapy scheduler and downloader (which means the scheduler and downloader middlewares are reused), but with a higher priority, processing them before other pages are scraped. The item remains “locked” at that particular pipeline stage until the files have finish downloading (or fail for some reason). When the files are downloaded, another field (files) will be populated with the results. This field will contain a list of dicts with information about the downloaded files, such as the downloaded path, the original scraped url (taken from the file_urls field), the file checksum and the file status. The files in the list of the files field will retain the same order of the original file_urls field. If some file failed downloading, an error will be logged and the file won’t be present in the files field.关于Images管道的介绍：Using the ImagesPipeline is a lot like using the FilesPipeline, except the default field names used are different: you use image_urls for the image URLs of an item and it will populate an images field for the information about the downloaded images. The advantage of using the ImagesPipeline for image files is that you can configure some extra functions like generating thumbnails and filtering the images based on their size. The Images Pipeline requires Pillow 4.0.0 or greater. It is used for thumbnailing and normalizing images to JPEG/RGB format.使用Media管道的流程如下：（1）在settings.py中修改默认的ITEM_PIPELINES，加上Media管道。然后设置存储目录（不设置存储目录的话也不会下载），比如：FILES_STORE = '/path/to/valid/dir' IMAGES_STORE = '/path/to/valid/dir'（2）指定文件名。默认情况下，文件名是以其来源网址进行 SHA-1 hash 后得到的字符串为名，如果是image则加上jpg后缀。但是你也可以在Media管道类中重写 file_path() 方法来自己命名文件，比如对 http://www.example.com/product/images/large/front/0000000004166 ，想命名为 00b08510e4_front.jpg （需要hash一下）：import hashlib from os.path import splitext def file_path(self, request, response=None, info=None, *, item=None): image_url_hash = hashlib.shake_256(request.url.encode()).hexdigest(5) image_perspective = request.url.split('/')[-2] image_filename = f'{image_url_hash}_{image_perspective}.jpg' return image_filename自己指定文件名最重要的原则是避免重名，因为这时旧的文件会被覆盖。同时，重写 file_path() 方法也可以用来将图片以原来的格式保存，这一点在保存gif文件时尤其有用，具体见后。（3）存储。支持的存储有本地存储，FTP服务器，Amazon S3存储，google云，这里只介绍本地存储，以image来说，它会被存储到：<IMAGES_STORE>/full/<FILE_NAME>这个目录下。其中full是因为要把thumbnail和图片分开存储而存在的。关于FTP服务器存储，请看原文：New in version 2.0. FILES_STORE and IMAGES_STORE can point to an FTP server. Scrapy will automatically upload the files to the server. FILES_STORE and IMAGES_STORE should be written in one of the following forms: ftp://username:password@address:port/path ftp://address:port/path If username and password are not provided, they are taken from the FTP_USER and FTP_PASSWORD settings respectively. FTP supports two different connection modes: active or passive. Scrapy uses the passive connection mode by default. To use the active connection mode instead, set the FEED_STORAGE_FTP_ACTIVE setting to True.修改文件的流程如下：（1）在settings.py里打开Media管道，并用 FILES_STORE 和 IMAGES_STORE 设置存放目录。（2）在items.py里添加专门爬取Media的field，比如：import scrapy class MyItem(scrapy.Item): # ... other item fields ... image_urls = scrapy.Field() images = scrapy.Field()image_urls 和 images 也可以在settings中更改：IMAGES_URLS_FIELD = 'field_name_for_your_images_urls' IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images' FILES_URLS_FIELD = 'field_name_for_your_files_urls' FILES_RESULT_FIELD = 'field_name_for_your_processed_files'后面两个是Files管道的。如果你的Media管道不是默认名，则在前面加上该名称，比如你的Images管道类叫做 MyPipeline ，那么在settings里需要指定为：MYPIPELINE_IMAGES_URLS_FIELD = MYPIPELINE_IMAGES_RESULT_FIELD = MYPIPELINE_FILES_URLS_FIELD = MYPIPELINE_FILES_RESULT_FIELD = 最后，讲解一些其余特征：（1）对近期（90天）内下载过的文件或图片，scrapy不会重复下载，这段时间的长短可以在settings.py中调整：# 120 days of delay for files expiration FILES_EXPIRES = 120 MYPIPELINE_FILES_EXPIRES = 120 # 30 days of delay for images expiration IMAGES_EXPIRES = 30 MYPIPELINE_IMAGES_EXPIRES = 30注意它们的单位是天。scrapy会将文件最后修改时间与现在时间比较，来判断一个文件是否在这个下载保护期内。（2）设置图片指纹，如：IMAGES_THUMBS = { 'small': (50, 50), 'big': (270, 270), }这里的键是指纹名，后面则是指纹图片长宽。如上设置时，一张图片会存储三个文件：<IMAGES_STORE>/full/图片名.jpg <IMAGES_STORE>/thumbs/small/图片名.jpg <IMAGES_STORE>/thumbs/big/图片名.jpg（3）过滤小图片。格式如下：IMAGES_MIN_HEIGHT = 110 IMAGES_MIN_WIDTH = 110如果对长宽都有限制，则图片要满足这两个限制才会被下载。默认情况下没有图片长宽限制。（4）允许重定向。默认情况下，Media管道发出的请求被重定向时scrapy会认为下载失败了。使用：MEDIA_ALLOW_REDIRECTS = True则允许Media管道下载重定向。重写Media管道类的方法，对于Files管道类，方法有：（1） file_path(self, request, response=None, info=None, *, item=None) ，这个方法每次下载item时会调用一次，用于给出存储的文件名。默认情况下它返回的是 full/<request URL hash>.<extension> ，前面要加上settings.py里设置的目录才是绝对路径。比如对于 https://example.com/a/b/c/foo.png 这张图片，使用例如下：import os from urllib.parse import urlparse from scrapy.pipelines.files import FilesPipeline class MyFilesPipeline(FilesPipeline): def file_path(self, request, response=None, info=None, *, item=None): return 'files/' + os.path.basename(urlparse(request.url).path)（2） get_media_requests(item, info) ，前面提到过，管道类会通过item拿到需要下载的url，然后会使用这个方法生成request对象去下载文件。这些request对象完成下载后会以2元素元组的列表格式发送给下面提到的 item_completed() 方法，每个2元素元组的格式为 (success, file_info_or_error) ，前者是bool，表示是否下载成功，后者是一个字典（如果下载成功），包含下载的一些信息，或者是一个twisted的Failure对象（如果下载失败）。另外注意， item_completed() 方法收到的元组列表的顺序和 get_media_requests(item, info) 发出的请求顺序是对应的。（3） item_completed(results, item, info) ，当一个item的所有下载请求都成功或失败时被调用。它的重写例子：from itemadapter import ItemAdapter from scrapy.exceptions import DropItem def item_completed(self, results, item, info): file_paths = [x['path'] for ok, x in results if ok] if not file_paths: raise DropItem("Item contains no files") adapter = ItemAdapter(item) adapter['file_paths'] = file_paths return item注意这个方法需要返回item，因为它要把下载的文件交给Media管道处理。Images管道类时Files管道类的扩展，修改了field名、添加了一些功能。（1） file_path(self, request, response=None, info=None, *, item=None) 方法，同FilesPipeline。（2） get_media_requests(item, info) ，和FilesPipeline相比只是更改了field名。（3） item_completed(results, item, info) ，和FilesPipeline相比只是更改了field名。6.AutoThrottle extension它的目的是通过scrapy负载和目标网页服务器的状态自动决定爬取速度。官方文档介绍了它的算法与原理，对于使用来说，你只需要知道在settings.py中设置 AUTOTHROTTLE_ENABLED = True 即可。更详细的设置如下：AUTOTHROTTLE_ENABLED¶ Default: False Enables the AutoThrottle extension. AUTOTHROTTLE_START_DELAY¶ Default: 5.0 The initial download delay (in seconds). AUTOTHROTTLE_MAX_DELAY¶ Default: 60.0 The maximum download delay (in seconds) to be set in case of high latencies. AUTOTHROTTLE_TARGET_CONCURRENCY¶ Default: 1.0 Average number of requests Scrapy should be sending in parallel to remote websites. By default, AutoThrottle adjusts the delay to send a single concurrent request to each of the remote websites. Set this option to a higher value (e.g. 2.0) to increase the throughput and the load on remote servers. A lower AUTOTHROTTLE_TARGET_CONCURRENCY value (e.g. 0.5) makes the crawler more conservative and polite. Note that CONCURRENT_REQUESTS_PER_DOMAIN and CONCURRENT_REQUESTS_PER_IP options are still respected when AutoThrottle extension is enabled. This means that if AUTOTHROTTLE_TARGET_CONCURRENCY is set to a value higher than CONCURRENT_REQUESTS_PER_DOMAIN or CONCURRENT_REQUESTS_PER_IP, the crawler won’t reach this number of concurrent requests. At every given time point Scrapy can be sending more or less concurrent requests than AUTOTHROTTLE_TARGET_CONCURRENCY; it is a suggested value the crawler tries to approach, not a hard limit. AUTOTHROTTLE_DEBUG¶ Default: False Enable AutoThrottle debug mode which will display stats on every response received, so you can see how the throttling parameters are being adjusted in real time.7.Benchmarking使用一个scrapy自带的最简爬虫测试你的硬件上限。简单的 scrapy bench 这一行命令即可，输出中会有几行： [scrapy.extensions.logstats] INFO: Crawled 518 pages (at 2880 pages/min), scraped 0 items (at 0 items/min) ，表示每分钟能爬2880个页面左右。但是从爬虫的复杂度、网页服务器的资源占用等角度考虑，实际环境中不会有这么快的速度。8.爬虫的暂停与恢复通过scheduler在硬盘中存储进度，去重器在硬盘上存储去重信息，加上能保存爬虫信息的一些扩展（extensions），可以做到暂停、恢复爬虫（中间可以关机的那种），这对大型网站的爬取尤其重要。通过在settings.py中指定 JOBDIR 就能做到这一点，但是要注意这个目录在不同的爬虫之间，或者同一个爬虫的不同实例之间都不能共享。比如你使用 scrapy crawl somespider -s JOBDIR=crawls/somespider-1 （不太清楚官方文档的JOBDIR为什么最后带一个 -1 ），然后对cmd或终端发送 ctrl+C 或其他一些信号时，就能安全地暂停爬虫；之后使用同样的 scrapy crawl 命令即可恢复爬虫。对于同一项目的多个爬虫，笔者认为可以在爬虫类中使用 custom_settings 这个属性指定 JOBDIR ，但是尚未实践。原文中给出的方法是使用爬虫的state属性：def parse_item(self, response): # parse item here self.state['items_count'] = self.state.get('items_count', 0) + 1最后需要提醒两点：（1）如果你的爬虫依赖Cookies，由于Cookies是会过期的，所以最好在暂停爬虫后不久就继续爬虫。另一个催促你尽快继续爬虫的因素是网页有可能会更新，导致你之前爬的数据有变化、过时或失效（比如爬取网页上提供的股市信息，而这个股市信息是每天更新的、只放出当天的）。（2）request对象也需要保存，这时会使用pickle模块，这就要求request对象是能用pickle序列化的。9.scrapy架构其实这个部分很尴尬，对初学者来说放在第一课感觉太枯燥，对老手来说没有必要讲。这里复制原文如下，图片自己随便找一张scrapy架构图看：This document describes the architecture of Scrapy and how its components interact. Overview¶ The following diagram shows an overview of the Scrapy architecture with its components and an outline of the data flow that takes place inside the system (shown by the red arrows). A brief description of the components is included below with links for more detailed information about them. The data flow is also described below. Data flow¶ Scrapy architecture The data flow in Scrapy is controlled by the execution engine, and goes like this: The Engine gets the initial Requests to crawl from the Spider. The Engine schedules the Requests in the Scheduler and asks for the next Requests to crawl. The Scheduler returns the next Requests to the Engine. The Engine sends the Requests to the Downloader, passing through the Downloader Middlewares (see process_request()). Once the page finishes downloading the Downloader generates a Response (with that page) and sends it to the Engine, passing through the Downloader Middlewares (see process_response()). The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing through the Spider Middleware (see process_spider_input()). The Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine, passing through the Spider Middleware (see process_spider_output()). The Engine sends processed items to Item Pipelines, then send processed Requests to the Scheduler and asks for possible next Requests to crawl. The process repeats (from step 3) until there are no more requests from the Scheduler. Components¶ Scrapy Engine¶ The engine is responsible for controlling the data flow between all components of the system, and triggering events when certain actions occur. See the Data Flow section above for more details. Scheduler¶ The scheduler receives requests from the engine and enqueues them for feeding them later (also to the engine) when the engine requests them. Downloader¶ The Downloader is responsible for fetching web pages and feeding them to the engine which, in turn, feeds them to the spiders. Spiders¶ Spiders are custom classes written by Scrapy users to parse responses and extract items from them or additional requests to follow. For more information see Spiders. Item Pipeline¶ The Item Pipeline is responsible for processing the items once they have been extracted (or scraped) by the spiders. Typical tasks include cleansing, validation and persistence (like storing the item in a database). For more information see Item Pipeline. Downloader middlewares¶ Downloader middlewares are specific hooks that sit between the Engine and the Downloader and process requests when they pass from the Engine to the Downloader, and responses that pass from Downloader to the Engine. Use a Downloader middleware if you need to do one of the following: process a request just before it is sent to the Downloader (i.e. right before Scrapy sends the request to the website); change received response before passing it to a spider; send a new Request instead of passing received response to a spider; pass response to a spider without fetching a web page; silently drop some requests. For more information see Downloader Middleware. Spider middlewares¶ Spider middlewares are specific hooks that sit between the Engine and the Spiders and are able to process spider input (responses) and output (items and requests). Use a Spider middleware if you need to post-process output of spider callbacks - change/add/remove requests or items; post-process start_requests; handle spider exceptions; call errback instead of callback for some of the requests based on response content. For more information see Spider Middleware. Event-driven networking¶ Scrapy is written with Twisted, a popular event-driven networking framework for Python. Thus, it’s implemented using a non-blocking (aka asynchronous) code for concurrency. For more information about asynchronous programming and Twisted see these links: Introduction to Deferreds Twisted - hello, asynchronous programming Twisted Introduction - Krondo10.下载中间件在settings.py中可以打开下载中间件，它的值越小表示离 process_request() 方法越早被唤醒、 process_response() 方法越晚被唤醒，或者理解为值越小离engine越近。这个值是比较有意义的，因为有时候你自己的下载中间件需要依赖之前的一些东西。默认的顺序如下：DOWNLOADER_MIDDLEWARES_BASE = { 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100, 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300, 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350, 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400, 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500, 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550, 'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560, 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600, 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700, 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750, 'scrapy.downloadermiddlewares.stats.DownloaderStats': 850, 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900, }如果值设置为None，表示禁用这个中间件。如果要自己修改下载中间件，可以参考middlewares.py里的注释。另外，还有一些子类，如 scrapy.downloadermiddlewares.cookies.CookiesMiddleware ，用来保持session、存放cookies。你可以在settings.py中通过 COOKIES_ENABLED 和 COOKIES_DEBUG 来对它进行行为设置（默认前者打开，后者关闭）。默认情况下，一个项目中的爬虫使用同一cookies，你可以使用request对象的 meta 里的 cookiejar 键来修改这一行为（一个项目中的不同爬虫使用各自的cookies）：for i, url in enumerate(urls): yield scrapy.Request(url, meta={'cookiejar': i}, callback=self.parse_page)cookiejar这个键不太持久，每次在parse系列方法中生成request对象都要手动指定一次：def parse_page(self, response): # do some processing return scrapy.Request("http://www.example.com/otherpage", meta={'cookiejar': response.meta['cookiejar']}, callback=self.parse_other_page)另外，还有DefaultHeadersMiddleware、DownloadTimeoutMiddleware、HttpAuthMiddleware、HttpCacheMiddleware、HttpCompressionMiddleware、HttpProxyMiddleware、RedirectMiddleware和其他各种 DOWNLOADER_MIDDLEWARES_BASE 里出现的下载中间件，这里不介绍了。11.爬虫中间件具体见middlewares.py文件。12.添加extensions具体见原文：The extensions framework provides a mechanism for inserting your own custom functionality into Scrapy. Extensions are just regular classes. Extension settings¶ Extensions use the Scrapy settings to manage their settings, just like any other Scrapy code. It is customary for extensions to prefix their settings with their own name, to avoid collision with existing (and future) extensions. For example, a hypothetic extension to handle Google Sitemaps would use settings like GOOGLESITEMAP_ENABLED, GOOGLESITEMAP_DEPTH, and so on. Loading & activating extensions¶ Extensions are loaded and activated at startup by instantiating a single instance of the extension class per spider being run. All the extension initialization code must be performed in the class __init__ method. To make an extension available, add it to the EXTENSIONS setting in your Scrapy settings. In EXTENSIONS, each extension is represented by a string: the full Python path to the extension’s class name. For example: EXTENSIONS = { 'scrapy.extensions.corestats.CoreStats': 500, 'scrapy.extensions.telnet.TelnetConsole': 500, } As you can see, the EXTENSIONS setting is a dict where the keys are the extension paths, and their values are the orders, which define the extension loading order. The EXTENSIONS setting is merged with the EXTENSIONS_BASE setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled extensions. As extensions typically do not depend on each other, their loading order is irrelevant in most cases. This is why the EXTENSIONS_BASE setting defines all extensions with the same order (0). However, this feature can be exploited if you need to add an extension which depends on other extensions already loaded. Available, enabled and disabled extensions¶ Not all available extensions will be enabled. Some of them usually depend on a particular setting. For example, the HTTP Cache extension is available by default but disabled unless the HTTPCACHE_ENABLED setting is set. Disabling an extension¶ In order to disable an extension that comes enabled by default (i.e. those included in the EXTENSIONS_BASE setting) you must set its order to None. For example: EXTENSIONS = { 'scrapy.extensions.corestats.CoreStats': None, } Writing your own extension¶ Each extension is a Python class. The main entry point for a Scrapy extension (this also includes middlewares and pipelines) is the from_crawler class method which receives a Crawler instance. Through the Crawler object you can access settings, signals, stats, and also control the crawling behaviour. Typically, extensions connect to signals and perform tasks triggered by them. Finally, if the from_crawler method raises the NotConfigured exception, the extension will be disabled. Otherwise, the extension will be enabled. Sample extension¶ Here we will implement a simple extension to illustrate the concepts described in the previous section. This extension will log a message every time: a spider is opened a spider is closed a specific number of items are scraped The extension will be enabled through the MYEXT_ENABLED setting and the number of items will be specified through the MYEXT_ITEMCOUNT setting. Here is the code of such extension: import logging from scrapy import signals from scrapy.exceptions import NotConfigured logger = logging.getLogger(__name__) class SpiderOpenCloseLogging: def __init__(self, item_count): self.item_count = item_count self.items_scraped = 0 @classmethod def from_crawler(cls, crawler): # first check if the extension should be enabled and raise # NotConfigured otherwise if not crawler.settings.getbool('MYEXT_ENABLED'): raise NotConfigured # get the number of items from settings item_count = crawler.settings.getint('MYEXT_ITEMCOUNT', 1000) # instantiate the extension object ext = cls(item_count) # connect the extension object to signals crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened) crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed) crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped) # return the extension object return ext def spider_opened(self, spider): logger.info("opened spider %s", spider.name) def spider_closed(self, spider): logger.info("closed spider %s", spider.name) def item_scraped(self, item, spider): self.items_scraped += 1 if self.items_scraped % self.item_count == 0: logger.info("scraped %d items", self.items_scraped) Built-in extensions reference¶ General purpose extensions¶ Log Stats extension¶ classscrapy.extensions.logstats.LogStats[source]¶ Log basic stats like crawled pages and scraped items. Core Stats extension¶ classscrapy.extensions.corestats.CoreStats[source]¶ Enable the collection of core statistics, provided the stats collection is enabled (see Stats Collection). Telnet console extension¶ classscrapy.extensions.telnet.TelnetConsole[source]¶ Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging. The telnet console must be enabled by the TELNETCONSOLE_ENABLED setting, and the server will listen in the port specified in TELNETCONSOLE_PORT. Memory usage extension¶ classscrapy.extensions.memusage.MemoryUsage[source]¶ Note This extension does not work in Windows. Monitors the memory used by the Scrapy process that runs the spider and: sends a notification e-mail when it exceeds a certain value closes the spider when it exceeds a certain value The notification e-mails can be triggered when a certain warning value is reached (MEMUSAGE_WARNING_MB) and when the maximum value is reached (MEMUSAGE_LIMIT_MB) which will also cause the spider to be closed and the Scrapy process to be terminated. This extension is enabled by the MEMUSAGE_ENABLED setting and can be configured with the following settings: MEMUSAGE_LIMIT_MB MEMUSAGE_WARNING_MB MEMUSAGE_NOTIFY_MAIL MEMUSAGE_CHECK_INTERVAL_SECONDS Memory debugger extension¶ classscrapy.extensions.memdebug.MemoryDebugger[source]¶ An extension for debugging memory usage. It collects information about: objects uncollected by the Python garbage collector objects left alive that shouldn’t. For more info, see Debugging memory leaks with trackref To enable this extension, turn on the MEMDEBUG_ENABLED setting. The info will be stored in the stats. Close spider extension¶ classscrapy.extensions.closespider.CloseSpider[source]¶ Closes a spider automatically when some conditions are met, using a specific closing reason for each condition. The conditions for closing a spider can be configured through the following settings: CLOSESPIDER_TIMEOUT CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT CLOSESPIDER_ERRORCOUNT Note When a certain closing condition is met, requests which are currently in the downloader queue (up to CONCURRENT_REQUESTS requests) are still processed. CLOSESPIDER_TIMEOUT¶ Default: 0 An integer which specifies a number of seconds. If the spider remains open for more than that number of second, it will be automatically closed with the reason closespider_timeout. If zero (or non set), spiders won’t be closed by timeout. CLOSESPIDER_ITEMCOUNT¶ Default: 0 An integer which specifies a number of items. If the spider scrapes more than that amount and those items are passed by the item pipeline, the spider will be closed with the reason closespider_itemcount. If zero (or non set), spiders won’t be closed by number of passed items. CLOSESPIDER_PAGECOUNT¶ Default: 0 An integer which specifies the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason closespider_pagecount. If zero (or non set), spiders won’t be closed by number of crawled responses. CLOSESPIDER_ERRORCOUNT¶ Default: 0 An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders won’t be closed by number of errors. StatsMailer extension¶ classscrapy.extensions.statsmailer.StatsMailer[source]¶ This simple extension can be used to send a notification e-mail every time a domain has finished scraping, including the Scrapy stats collected. The email will be sent to all recipients specified in the STATSMAILER_RCPTS setting. Emails can be sent using the MailSender class. To see a full list of parameters, including examples on how to instantiate MailSender and use mail settings, see Sending e-mail. Debugging extensions¶ Stack trace dump extension¶ classscrapy.extensions.debug.StackTraceDump[source]¶ Dumps information about the running process when a SIGQUIT or SIGUSR2 signal is received. The information dumped is the following: engine status (using scrapy.utils.engine.get_engine_status()) live references (see Debugging memory leaks with trackref) stack trace of all threads After the stack trace and engine status is dumped, the Scrapy process continues running normally. This extension only works on POSIX-compliant platforms (i.e. not Windows), because the SIGQUIT and SIGUSR2 signals are not available on Windows. There are at least two ways to send Scrapy the SIGQUIT signal: By pressing Ctrl-while a Scrapy process is running (Linux only?) By running this command (assuming <pid> is the process id of the Scrapy process): kill -QUIT <pid> Debugger extension¶ classscrapy.extensions.debug.Debugger[source]¶ Invokes a Python debugger inside a running Scrapy process when a SIGUSR2 signal is received. After the debugger is exited, the Scrapy process continues running normally. For more info see Debugging in Python. This extension only works on POSIX-compliant platforms (i.e. not Windows).13.其他包括处理动态生成的数据（官方文档讲的很浅），处理内存泄漏，分布式部署爬虫（scrapyd），对Coroutines（协程）和asyncio（异步读写）的支持，各组件的API，scheduler，Item Exporters（导出为XML、CSV、JSON及JSON LINE等格式），这里就不介绍了。
本文较长，属于系统性总结，可作为使用手册查阅。步骤/目录： 1.scrapy介绍与安装 2.scrapy使用例一 3.scrapy命令行工具 4.爬虫类 （1）CrawlSpider （2）XMLFeedSpider与CSVFeedSpider （3）SitemapSpider 5.选择器对象 6.Items 7.Item Loaders 8.scrapy shell 9.Item Pipeline 10.Feed exports 11.request与response 12.link extractor 13.settings 14.Exceptions 15.logging 16.Stats Collection 17.Sending e-mail 18.Telnet Console 19.FAQ本文首发于个人博客https://lisper517.top/index.php/archives/51/，转载请注明出处。本文的目的是详细介绍scrapy，主要翻译自 scrapy官方文档 。本文写作日期为2022年9月14日。运行的平台为win10，编辑器为VS Code。1.scrapy介绍与安装scrapy（/ˈskreɪpaɪ/）是一个高效、通用的网页爬取框架，可用于提取页面数据、自动化测试网页、网页归档等。scrapy完全使用python语言（但是它依赖的库有些可能用了其他语言，比如C++），所以安装、使用scrapy之前需要安装、熟悉python。scrapy要求python版本在3.6以上，依赖的库有：lxml, an efficient XML and HTML parser parsel, an HTML/XML data extraction library written on top of lxml, w3lib, a multi-purpose helper for dealing with URLs and web page encodings twisted, an asynchronous networking framework cryptography and pyOpenSSL, to deal with various network-level security needs其中，有些依赖有版本要求：Twisted 14.0 lxml 3.4 pyOpenSSL 0.14使用 pip install Scrapy 即可安装scrapy及其依赖。如果用了Anaconda等包管理器，则使用 conda install -c conda-forge scrapy 命令安装scrapy。另外，官方文档最推荐的是在python虚拟环境中安装scrapy，以避免可能的包冲突，具体见 python官方文档 。2.scrapy使用例一安装好scrapy后，开始使用scrapy爬取一个网站： https://quotes.toscrape.com ，这是一个用来练习爬虫的网站。创建一个放scrapy爬虫的目录，比如 D:\spiders\scrapy 或 ~/spider/scrapy ，然后在cmd或终端中进入该目录，使用 scrapy startproject tutorial 来创建一个scrapy项目，这里的 tutorial 是项目名，你可以自行替换。然后在 tutorial/tutorial/spiders 目录下新建一个爬虫文件，比如 quotes_spider.py ，文件内容如下：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" def start_requests(self): urls = [ 'https://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/2/', ] for url in urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): page = response.url.split("/")[-2] filename = f'quotes-{page}.html' with open(filename, 'wb') as f: f.write(response.body) self.log(f'Saved file {filename}')在这个爬虫类中， name 指定了爬虫名，在一个项目中的爬虫名不能相同； start_requests() 方法用于爬取初始网址，这个方法重写了父类方法，它需要返回可迭代的Requests对象； parse() 方法也重写了父类方法，它是默认的解析页面的方法，用于处理Response对象，Response对象是TextResponse类实例化来的，有一些帮助解析页面的属性和方法。 parse() 方法和其他自己写的 parse_xxx() 方法，它们的主要作用是解析响应对象，提取信息（以字典形式），找到页面上其他需要爬取的url、创建新的Request对象。接下来开启爬虫。在cmd或终端中进入项目文件夹，使用 scrapy crawl quotes 开启爬虫。注意这里跟的是爬虫名，不是爬虫文件名。之后，在cmd进入的目录下会生成两个html文件，即这次爬虫运行的结果。另外， start_requests() 方法也可以不写，而是改成写爬虫类的 start_urls 属性（起始url字符串的列表），这个列表中的url会自动被父类中的 start_requests() 方法调用，而不用显式指定：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/2/', ] def parse(self, response): page = response.url.split("/")[-2] filename = f'quotes-{page}.html' with open(filename, 'wb') as f: f.write(response.body)scrapy命令行还支持对Response进行调试，方法是 scrapy shell "url" 获取响应对象，然后使用 response.css('title') 、 response.css('title::text').re(r'Quotes.*') 、 response.xpath('//title') 等方法来查找html标签，但是这部分功能用浏览器自带的开发者模式（按F12）也能代替、且更方便，所以这里就不介绍了。最后，提一下Selector对象。 response.css() 方法和 response.xpath() 方法都返回选择器对象，而且在scrapy底层css选择器是转换成xpath表达式后查找标签的，而且xpath表达式比css选择器的功能更强，所以scrapy官方文档更推荐使用xpath，但是关于xpath表达式的内容这里就不展开了，有需要的可阅读笔者之前的文章 https://lisper517.top/index.php/archives/41/ 或参考 runoob教程 。Selector对象有 get() 、 getall() 方法来返回内容，不同在于前者只返回第一个、后者返回包含所有内容的列表。两个get方法是新方法，与之对应的旧方法是 extract_first() 、 extract() （现在它们的别名就是get和getall），但是现在新方法用的更多。它们之间只有一些微小的区别，get()只会返回一个结果，getall()总会返回一个列表（如果能取到东西的话），所以新方法的行为更加predictable。在刚才的爬虫中修改一些内容：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', 'https://quotes.toscrape.com/page/2/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('small.author::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), }python中的 yield 关键字功能和return类似，但用在递归程序中非常节约资源，在scrapy的爬虫中经常使用。再次用 scrapy crawl quotes 运行爬虫，会发现打印出了几个dict，这也是scrapy返回数据的格式。光在命令行中输出字典还不够，更常用的方法是把数据持久化存储，比如存成文件。在scrapy运行时就可以指定输出文件，如 scrapy crawl quotes -O quotes.jl ，虽然这种方式比较简陋，但命令行的 -O 参数会覆盖其他所有输出设置。这里的.jl文件比.json的好处是容易在后面附加数据（append）。以上提到的方法是在爬虫文件中就处理爬取到的数据，然而对于复杂一点的存储，更常用的方法是更改pipelines.py文件，即设置item pipeline，这部分内容会在后面讲解。scrapy更重要的功能是全站爬取，也就是对提取出来的url再次解析。在上面的例子中，可以把爬虫文件改写为：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('small.author::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } next_page = response.css('li.next a::attr(href)').get() if next_page is not None: next_page = response.urljoin(next_page) yield scrapy.Request(next_page, callback=self.parse)response.urljoin() 是用于有些时候，从a标签的href属性提取的网址不完整，这个方法会自动检测并补全网址。更重要的是这里用yield又构造了一个request对象， parse() 方法里出现了两次yield，这是yield和return使用的区别之一。在 scrapy.Request(next_page, callback=self.parse) 中，next_page指明下一个网址（scrapy将把这个网址放入请求队列，当下载器有空时就发出请求），callback指定调用的方法就是 parse() 方法自己，也就是说 parse() 方法自己调用了自己，这称为递归；有些时候callback也可以是自己写的其他 parse_xxx() 方法，这种情况一般是因为两个网页的格式不同。如果解析的是相近的页面，还可以使用 response.follow() 方法，这时可以省略 response.urljoin() 方法：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('span small::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } next_page = response.css('li.next a::attr(href)').get() if next_page is not None: yield response.follow(next_page, callback=self.parse)response.follow() 方法会自动调用urljoin。另外， response.follow(next_page, callback=self.parse) 的第一个参数还能换成Selector对象，只要这个标签有href属性；或者用 follow_all 方法，但是传入一个Selector对象的列表甚至是css表达式，格式稍微改一下：import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'https://quotes.toscrape.com/page/1/', ] def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('span small::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } yield from response.follow_all(css='ul.pager a', callback=self.parse)正如前面提到的，callback指定的回调函数可以是其他自己写的 parse_xxx() 方法：import scrapy class AuthorSpider(scrapy.Spider): name = 'author' start_urls = ['https://quotes.toscrape.com/'] def parse(self, response): author_page_links = response.css('.author + a') yield from response.follow_all(author_page_links, self.parse_author) pagination_links = response.css('li.next a') yield from response.follow_all(pagination_links, self.parse) def parse_author(self, response): def extract_with_css(query): return response.css(query).get(default='').strip() yield { 'name': extract_with_css('h3.author-title::text'), 'birthdate': extract_with_css('.author-born-date::text'), 'bio': extract_with_css('.author-description::text'), }以上所有的方法都不需要担心重复访问同一个网址，因为scrapy自带网址去重。要关掉这个功能，可以在follow系列方法中使用 dont_filter=True ，或者设置DUPEFILTER_CLASS类来指定，但不推荐这样做。在 parse 系列方法中还可以传递item数据，称为请求传参，这将在后续介绍。现在稍微提一下使用场景，比如你要爬取一个博客，博客的目录页有博文的标题、上传时间，但是又要进入每篇博文爬取详细内容，这时就可以使用请求传参。3.scrapy命令行工具scrapy在命令行中使用时有一些注意事项。首先，在项目目录里有一个scrapy.cfg文件，它是这个项目的配置文件，优先级最高（相对于scrapy的全局配置、用户配置）。在这个文件中，可以指定多个项目共用一个settings.py文件。在cmd输入 scrapy ，如果cmd在scrapy项目目录之内，第一行会显示当前项目内是否有活动的爬虫，其他行则是使用帮助。下面就详细讲解scrapy命令行， [] 括起来表示可选项。。（1）查看某个命令的详细帮助： scrapy <command> -h 。（2）创建项目： scrapy startproject 项目名 [项目目录名] 。创建项目后进入项目目录。（3）创建爬虫： scrapy genspider [-t template] <爬虫名> <待爬域名> ， -t 指定爬虫模板，待爬域名可以随便写。（4）查看爬虫模板： scrapy genspider -l 。（5）开启爬虫： scrapy crawl <spider> 。（6）检测爬虫： scrapy check [-l] <spider> 。（7）输出项目中所有爬虫： scrapy list 。（8）编辑爬虫： scrapy edit <spider> ，将用默认的编辑器打开并编辑爬虫，这个编辑器可用 EDITOR 环境变量指定。（9）爬取一个网页： scrapy fetch <url> ，爬取网页并输出其内容到standard output。在项目中使用时，它是根据settings.py或爬虫的设置取网页的，比如有时你在settings.py里设置了UA，或者对爬虫有一些个性化设置。可选项有 --spider=SPIDER （使用某个爬虫的设置）、 --headers （打印请求头）、 --no-redirect （不重定向）。（10）查看网页： scrapy view <url> ，可选项有 --spider=SPIDER 、 --no-redirect 。和 fetch 一起，这两个命令可用于检查爬虫和自己看到的网页有无不同。（11）查看设置： scrapy settings [options] ，比如 scrapy settings --get BOT_NAME 。在项目内时这个命令显示项目设置，否则显示默认设置。（12）运行单独爬虫文件： scrapy runspider <spider_file.py> ，运行一个能独立运行的爬虫文件，很少用。（13）显示scrapy版本： scrapy version [-v] 。其他命令用的不多，这里就不介绍了。另外，可以自己定制scrapy命令，需要用到 COMMANDS_MODULE 这项设置，具体参考原文。4.爬虫类在 scrapy使用例一中 介绍最多的就是爬虫文件中的爬虫类。所有scrapy爬虫类都需要继承自scrapy.Spider父类，这个父类的主要功能有 start_requests() 、 start_urls 、 parse() ，这些已经在前文介绍过了。其他还有一些属性及方法：（1）name，爬虫名字，用于区别项目内的各个爬虫。一般来说，一个爬虫类只会实例化一个对象，但也能实例化多个。常用的name编写方法是根据爬取的域名来截取。（2）allowed_domains，允许爬取的域名（包括子域名），这个一般可以注释掉。（3）start_urls，前文提过。（4）custom_settings，用于指定爬虫的个性化设置，会覆盖掉settings.py里的设置。注意它必须写在类属性而不能在__init__里指定，因为实例化之前就会上传设置。（5）logger，用爬虫名创建的logger对象，可以自己写log信息。或者也可用 log(message[, level, component]) 方法写log。（6）closed(reason)，当爬虫结束时调用一次这个方法。（7）start_requests() 和 parse(response) ，前文已介绍。另外，scrapy提供了一些衍生的爬虫类，方便爬取全站数据，或者根据sitemap爬取，或者解析XML feed。下面的例子中，假设 items.py 文件如下：import scrapy class TestItem(scrapy.Item): id = scrapy.Field() name = scrapy.Field() description = scrapy.Field()（1）CrawlSpider即scrapy.spiders.CrawlSpider，常用于爬取全站数据，它的特点是提取规则（rules）。一个CrawlSpider例子如下：import scrapy from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider, Rule class CrawlspiSpider(CrawlSpider): name = 'crawlspi' #allowed_domains = ['xxx.com'] start_urls = ['http://www.xxx.com/'] rules = ( Rule(LinkExtractor(allow=r'page=\d+'), callback='parse_item', follow=True), Rule(LinkExtractor(allow=r'xxxx'), callback='parse_detail', follow=True), Rule(LinkExtractor(restrict_xpaths='//a', allow=r'/wzshow_\d+\.html'), callback='parse_item'), # 还可以通过xpath解析网页 ) # LinkExtractor(allow=r'Items/')根据规则提取链接（正则） # Rule对象会对提取到的网页自动发送请求并通过callback解析（无法使用请求传参） # follow=True即对提取到的链接页面再应用规则提取链接（全站爬取） # 一般第一个Rule提取其他页码链接，第二个Rule提取当前页面详情url # 对请求传参，在items中再写一个DetailItem类，并且有一个共同的主键PRIMARY KEY用于识别 # 在Pipeline中对item.__class__.__name__判断，对不同的item类处理 # 插入Mysql中时通过主键处理 def parse_item(self, response): item = {} #item['domain_id'] = response.xpath('//input[@id="sid"]/@value').get() #item['name'] = response.xpath('//div[@id="name"]').get() #item['description'] = response.xpath('//div[@id="description"]').get() return item def parse_detail(self, response): item = {} return item如果创建了多个Rule对象，而一个链接符合多个规则，则会应用第一个匹配的规则。提取规则对象的完整实例化格式如下：scrapy.spiders.Rule(link_extractor=None, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None, errback=None)link_extractor指定Link Extractor对象，如果省略则匹配所有提取的链接。它是用来生成request对象的，这里的request对象的 meta['link_text'] 可以给出请求的url；callback是一个callable方法或字符串，用于调用指定的方法，一般是parse系列方法；cb_kwargs是一个参数字典，将传递给回调函数（所以也可以用请求传参，上面的注释适用于1.7版本之前的scrapy）；follow则指定，对于使用规则提取到的网页，是否继续应用规则提取链接；process_links，也是一个callable方法或字符串，用于调用指定的方法，但这个方法主要用来对提取到的链接进行筛选；process_request，也是一个callable方法或字符串，用于调用指定的方法（该方法的第一个参数为request对象，第二个参数为该request对象生成的response对象，返回一个request对象或者None），它主要对请求进行筛选（返回None时就是筛选掉了请求对象）；errback是发生exception时的回调函数。官方文档给出的例子是：import scrapy from scrapy.spiders import CrawlSpider, Rule from scrapy.linkextractors import LinkExtractor class MySpider(CrawlSpider): name = 'example.com' allowed_domains = ['example.com'] start_urls = ['http://www.example.com'] rules = ( # Extract links matching 'category.php' (but not matching 'subsection.php') # and follow links from them (since no callback means follow=True by default). Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))), # Extract links matching 'item.php' and parse them with the spider's method parse_item Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'), ) def parse_item(self, response): self.logger.info('Hi, this is an item page! %s', response.url) item = scrapy.Item() item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)') item['name'] = response.xpath('//td[@id="item_name"]/text()').get() item['description'] = response.xpath('//td[@id="item_description"]/text()').get() item['link_text'] = response.meta['link_text'] url = response.xpath('//td[@id="additional_data"]/@href').get() return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item)) def parse_additional_page(self, response, item): item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get() return item注意这里用的都是return而非yield。（2）XMLFeedSpider与CSVFeedSpider分别用于解析xml的feed网页和csv，这里就不详细介绍了。它们也使用return而非yield。（3）SitemapSpider根据sitemap爬网页，这个其实用的也很少。它能手动指定sitemap网页，或者根据robots.txt找到sitemap网页。5.选择器对象在爬虫界，常用的解析网页html或xml的方式有beautifulsoup（缺点是慢）、正则表达式（麻烦，容易解析错误）、lxml；lxml又分为css、xpath等。scrapy的Selector对象提供了一些方法，也能解析html，它在底层使用的是lxml的xpath表达式，解析速度也差不多。正如前文提到的， response.css() 和 response.xpath() 方法返回选择器对象列表，对选择器对象列表可使用 get() 和 getall() 方法，这两个方法如果取不到对象就会返回None，但是如果用 get(default="some-string") 也可以指定取不到时返回什么。除了这两个方法，scrapy还提供了选择器的其他方法来取出属性等，比如attrib、re方法，这里就不介绍了。另外，在xpath函数里使用 .//text() 时最好用 . 代替；xpath还可以传递参数，比如 response.xpath('//div[count(a)=$cnt]/@id', cnt=5) 。scrapy还为xpath提供了自己的has-class方法： response.xpath('//p[has-class("foo", "bar-baz")]') ，这样就能选中像 First 和 Second 这样的标签（属性值变化），但它比css方法稍慢。parsel.xpathfuncs.set_xpathfunc(fname, func) 可以自己注册xpath函数。选择器对象的构造函数为 scrapy.selector.Selector(*args, **kwargs) ，参数有： response ，指定HtmlResponse或XmlResponse对象； text ，指定响应的正文，不能和 response 一起用； type ，指定格式，可选择html、xml或None，为None时自动选择格式；另外，为了方便，scrapy还自带了选择器列表的类，它衍生于基础的list，这样对于元素为选择器的列表也能使用以上方法。6.Itemsitems系列的类就是字典或者可以当成字典使用，用于爬虫处理完信息后的传递。items类在items.py中进行定义，常见的有：（1）字典，就是python字典。（2）item对象，它提供了类似字典的API和一些其他功能，构造函数为 scrapy.item.Item([arg]) 或 scrapy.Item([arg]) ，简单来说你需要在items.py里规定一个field，然后才能在爬虫中使用这个field，否则会产生exception。另外，还有 Item.copy() 和 Item.deepcopy() 方法用于浅拷贝和深拷贝item对象。一个使用item对象的items.py文件如下：import scrapy class IpoolItem(scrapy.Item): one_field = scrapy.Field() another_field = scrapy.Field()（3）dataclass对象，它支持对每个field设置默认值，使用例为：from dataclasses import dataclass @dataclass class CustomItem: one_field: str another_field: int虽然规定了field的类型，但在爬虫中为field赋值时不会检查类型。（4）attr.s对象，使用例为：import attr @attr.s class CustomItem: one_field = attr.ib() another_field = attr.ib()同样支持默认值。items系列的对象可以使用字典的 keys() 、 items() 方法。另外，由于python的机制，在items中存放列表或字典等数据时，如果要拷贝则应当使用深拷贝。最后，items对象还支持一些metadata（超参数）；你可以从自己定义的item类衍生子类，来增加field或修改field的超参数：class IpoolItem(scrapy.Item): one_field = scrapy.Field() another_field = scrapy.Field() class IpoolItem(IpoolItem): third_field = scrapy.Field()7.Item Loadersitem提供了存放数据的容器，而item loader可以帮助你更便捷地把数据放进容器。比方说，你可以使用item loader多次对一个field赋值，item loader会把你赋的值组合起来（字符串会append）。item loader是在爬虫文件中使用的，使用例：from scrapy.loader import ItemLoader from myproject.items import Product def parse(self, response): l = ItemLoader(item=ProductItem(), response=response) l.add_xpath('name', '//div[@class="product_name"]') l.add_xpath('name', '//div[@class="product_title"]') l.add_xpath('price', '//p[@id="price"]') l.add_css('stock', 'p#stock') l.add_value('last_updated', 'today') # you can also use literal values return l.load_item()这里使用了 add_xpath() 、 add_css() 、 add_value() 、 load_item() 方法，注意它们的行为。在使用 load_item() 之前，所有的数据都暂存在item loader里，使用之后才会赋值到item的field中。实际上，对于每个field，每次在item loader中赋值时都是从一个input processor进入，使用 load_item() 时从一个output processor出来，这涉及到一些其他用法。Itemloader的构造函数为 scrapy.loader.ItemLoader(item=None, selector=None, response=None, parent=None, **context) ，其中 selector 和 response 是互斥的，前者优先。另外，如果要更改值而不是append，ItemLoader类提供了replace系列方法，就是把add系列中的add换成replace。最后，Itemloader可以停留在某个节点位置：loader = ItemLoader(item=Item()) # load stuff not in the footer footer_loader = loader.nested_xpath('//footer') footer_loader.add_xpath('social', 'a[@class = "social"]/@href') footer_loader.add_xpath('email', 'a[@class = "email"]/@href') # no need to call footer_loader.load_item() loader.load_item()这里用的 add_xpath 中的xpath表达式都是相对于footer标签的位置，也就是说完整的xpath表达式为 //footer/a[@class = "social"]/@href 。总的来说，itemloader是为了简化item生成而存在的，在某些情况下有用，但一般不会用到。8.scrapy shell它的主要作用是检测xpath、css表达式。由于浏览器的开发者模式能更便捷地做到相同的事情，这里就不介绍了。9.Item Pipelineitem管道，在爬虫生成item类后会把这些item数据传递到管道中进行处理，即scrapy项目中的pipelines.py文件。那你可能就要问为什么我不在爬虫中，刚爬到数据就把数据处理了呢？这一方面是为了代码的简洁性，另一方面管道有一些独到的优势。一般来说，管道可以用于：理清html数据，验证数据，数据去重，数据存入数据库。在pipelines.py中可以写你自己的管道，必须提供的方法只有 process_item(self, item, spider) ，它用于处理item，它必须返回一个item对象或Twisted的Deferred，或者raise一个Dropitem的exception（此时这个item就不会向下面的管道继续传递）。另外，你还可以选择写以下方法：open_spider(self, spider) ，只在爬虫开启时调用一次，你可以在这里使用pymysql创建connection、cursor，或者创建一个文件读写对象；相对的是 close_spider(self, spider) ，只在爬虫关闭时调用一次，可以关闭conn、cur、文件读写对象；另外还有 from_crawler(cls, crawler) 方法，当它存在时，会根据crawler对象实例化一个管道，所以它也必须返回一个管道对象，这个方法的作用是为管道提供访问settings等核心的途径，以及提供函数钩子（hook）。管道文件的范例：from itemadapter import ItemAdapter from scrapy.exceptions import DropItem class PricePipeline: vat_factor = 1.15 def process_item(self, item, spider): adapter = ItemAdapter(item) if adapter.get('price'): if adapter.get('price_excludes_vat'): adapter['price'] = adapter['price'] * self.vat_factor return item else: raise DropItem(f"Missing price in {item}")再来一个（存入mongodb）：import pymongo from itemadapter import ItemAdapter class MongoPipeline: collection_name = 'scrapy_items' def __init__(self, mongo_uri, mongo_db): self.mongo_uri = mongo_uri self.mongo_db = mongo_db @classmethod def from_crawler(cls, crawler): return cls( mongo_uri=crawler.settings.get('MONGO_URI'), mongo_db=crawler.settings.get('MONGO_DATABASE', 'items') ) def open_spider(self, spider): self.client = pymongo.MongoClient(self.mongo_uri) self.db = self.client[self.mongo_db] def close_spider(self, spider): self.client.close() def process_item(self, item, spider): self.db[self.collection_name].insert_one(ItemAdapter(item).asdict()) return item最后，如果自己在pipelines.py里写了其他管道，或者修改了默认管道名，需要在settings.py里打开管道：ITEM_PIPELINES = { 'myproject.pipelines.管道名1': 300, 'myproject.pipelines.管道名2': 800, }后面的300、800，它们的取值范围为0-1000，item将从低值的管道流经高值的管道。10.Feed exports你可以用Feed exports来把数据导出成json、json lines（jl）、csv或xml等格式，或者python的pickle，这样数据就能用于不同的系统了。这个功能主要需要在settings.py中开启，相关的设置有：FEEDS FEED_EXPORT_ENCODING FEED_STORE_EMPTY FEED_EXPORT_FIELDS FEED_EXPORT_INDENT FEED_STORAGES FEED_STORAGE_FTP_ACTIVE FEED_STORAGE_S3_ACL FEED_EXPORTERS FEED_EXPORT_BATCH_ITEM_COUNT第一项是必须的。范例：FEEDS={ 'items.json': { 'format': 'json', 'encoding': 'utf8', 'store_empty': False, 'item_classes': [MyItemClass1, 'myproject.items.MyItemClass2'], 'fields': None, 'indent': 4, 'item_export_kwargs': { 'export_empty_fields': True, }, }, '/home/user/documents/items.xml': { 'format': 'xml', 'fields': ['name', 'price'], 'item_filter': MyCustomFilter1, 'encoding': 'latin1', 'indent': 8, }, pathlib.Path('items.csv.gz'): { 'format': 'csv', 'fields': ['price', 'name'], 'item_filter': 'myproject.filters.MyCustomFilter2', 'postprocessing': [MyPlugin1, 'scrapy.extensions.postprocessing.GzipPlugin'], 'gzip_compresslevel': 5, }, }各个参数的解释：The following is a list of the accepted keys and the setting that is used as a fallback value if that key is not provided for a specific feed definition: format: the serialization format. This setting is mandatory, there is no fallback value. batch_item_count: falls back to FEED_EXPORT_BATCH_ITEM_COUNT. New in version 2.3.0. encoding: falls back to FEED_EXPORT_ENCODING. fields: falls back to FEED_EXPORT_FIELDS. item_classes: list of item classes to export. If undefined or empty, all items are exported. New in version 2.6.0. item_filter: a filter class to filter items to export. ItemFilter is used be default. New in version 2.6.0. indent: falls back to FEED_EXPORT_INDENT. item_export_kwargs: dict with keyword arguments for the corresponding item exporter class. New in version 2.4.0. overwrite: whether to overwrite the file if it already exists (True) or append to its content (False). The default value depends on the storage backend: Local filesystem: False FTP: True Note Some FTP servers may not support appending to files (the APPE FTP command). S3: True (appending is not supported) Standard output: False (overwriting is not supported) New in version 2.4.0. store_empty: falls back to FEED_STORE_EMPTY. uri_params: falls back to FEED_URI_PARAMS. postprocessing: list of plugins to use for post-processing. The plugins will be used in the order of the list passed. New in version 2.6.0.其他设置的解析：FEED_EXPORT_ENCODING¶ Default: None The encoding to be used for the feed. If unset or set to None (default) it uses UTF-8 for everything except JSON output, which uses safe numeric encoding (\uXXXX sequences) for historic reasons. Use utf-8 if you want UTF-8 for JSON too. FEED_EXPORT_FIELDS¶ Default: None A list of fields to export, optional. Example: FEED_EXPORT_FIELDS = ["foo", "bar", "baz"]. Use FEED_EXPORT_FIELDS option to define fields to export and their order. When FEED_EXPORT_FIELDS is empty or None (default), Scrapy uses the fields defined in item objects yielded by your spider. If an exporter requires a fixed set of fields (this is the case for CSV export format) and FEED_EXPORT_FIELDS is empty or None, then Scrapy tries to infer field names from the exported data - currently it uses field names from the first item. FEED_EXPORT_INDENT¶ Default: 0 Amount of spaces used to indent the output on each level. If FEED_EXPORT_INDENT is a non-negative integer, then array elements and object members will be pretty-printed with that indent level. An indent level of 0 (the default), or negative, will put each item on a new line. None selects the most compact representation. Currently implemented only by JsonItemExporter and XmlItemExporter, i.e. when you are exporting to .json or .xml. FEED_STORE_EMPTY¶ Default: False Whether to export empty feeds (i.e. feeds with no items). FEED_STORAGES¶ Default: {} A dict containing additional feed storage backends supported by your project. The keys are URI schemes and the values are paths to storage classes. FEED_STORAGE_FTP_ACTIVE¶ Default: False Whether to use the active connection mode when exporting feeds to an FTP server (True) or use the passive connection mode instead (False, default). For information about FTP connection modes, see What is the difference between active and passive FTP?. FEED_STORAGE_S3_ACL¶ Default: '' (empty string) A string containing a custom ACL for feeds exported to Amazon S3 by your project. For a complete list of available values, access the Canned ACL section on Amazon S3 docs. FEED_STORAGES_BASE¶ Default: { '': 'scrapy.extensions.feedexport.FileFeedStorage', 'file': 'scrapy.extensions.feedexport.FileFeedStorage', 'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage', 's3': 'scrapy.extensions.feedexport.S3FeedStorage', 'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage', } A dict containing the built-in feed storage backends supported by Scrapy. You can disable any of these backends by assigning None to their URI scheme in FEED_STORAGES. E.g., to disable the built-in FTP storage backend (without replacement), place this in your settings.py: FEED_STORAGES = { 'ftp': None, } FEED_EXPORTERS¶ Default: {} A dict containing additional exporters supported by your project. The keys are serialization formats and the values are paths to Item exporter classes. FEED_EXPORTERS_BASE¶ Default: { 'json': 'scrapy.exporters.JsonItemExporter', 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter', 'jl': 'scrapy.exporters.JsonLinesItemExporter', 'csv': 'scrapy.exporters.CsvItemExporter', 'xml': 'scrapy.exporters.XmlItemExporter', 'marshal': 'scrapy.exporters.MarshalItemExporter', 'pickle': 'scrapy.exporters.PickleItemExporter', } A dict containing the built-in feed exporters supported by Scrapy. You can disable any of these exporters by assigning None to their serialization format in FEED_EXPORTERS. E.g., to disable the built-in CSV exporter (without replacement), place this in your settings.py: FEED_EXPORTERS = { 'csv': None, } FEED_EXPORT_BATCH_ITEM_COUNT¶ New in version 2.3.0. Default: 0 If assigned an integer number higher than 0, Scrapy generates multiple output files storing up to the specified number of items in each output file. When generating multiple output files, you must use at least one of the following placeholders in the feed URI to indicate how the different output file names are generated: %(batch_time)s - gets replaced by a timestamp when the feed is being created (e.g. 2020-03-28T14-45-08.237134) %(batch_id)d - gets replaced by the 1-based sequence number of the batch. Use printf-style string formatting to alter the number format. For example, to make the batch ID a 5-digit number by introducing leading zeroes as needed, use %(batch_id)05d (e.g. 3 becomes 00003, 123 becomes 00123). For instance, if your settings include: FEED_EXPORT_BATCH_ITEM_COUNT = 100 And your crawl command line is: scrapy crawl spidername -o "dirname/%(batch_id)d-filename%(batch_time)s.json" The command line above can generate a directory tree like: ->projectname -->dirname --->1-filename2020-03-28T14-45-08.237134.json --->2-filename2020-03-28T14-45-09.148903.json --->3-filename2020-03-28T14-45-10.046092.json Where the first and second files contain exactly 100 items. The last one contains 100 items or fewer. FEED_URI_PARAMS¶ Default: None A string with the import path of a function to set the parameters to apply with printf-style string formatting to the feed URI. The function signature should be as follows: scrapy.extensions.feedexport.uri_params(params, spider)¶ Return a dict of key-value pairs to apply to the feed URI using printf-style string formatting. Parameters params (dict) – default key-value pairs Specifically: batch_id: ID of the file batch. See FEED_EXPORT_BATCH_ITEM_COUNT. If FEED_EXPORT_BATCH_ITEM_COUNT is 0, batch_id is always 1. New in version 2.3.0. batch_time: UTC date and time, in ISO format with : replaced with -. See FEED_EXPORT_BATCH_ITEM_COUNT. New in version 2.3.0. time: batch_time, with microseconds set to 0. spider (scrapy.Spider) – source spider of the feed items Caution The function should return a new dictionary, modifying the received params in-place is deprecated. For example, to include the name of the source spider in the feed URI: Define the following function somewhere in your project: # myproject/utils.py def uri_params(params, spider): return {**params, 'spider_name': spider.name} Point FEED_URI_PARAMS to that function in your settings: # myproject/settings.py FEED_URI_PARAMS = 'myproject.utils.uri_params' Use %(spider_name)s in your feed URI: scrapy crawl <spider_name> -o "%(spider_name)s.jl"因为笔者用这个功能不多，这里就不翻译了。11.request与responserequest对象由爬虫生成，最后传递到下载器；response则是由下载器根据request生成，返回给发出request的爬虫。二者都有一些衍生类，下面先介绍一下基类。classscrapy.http.Request(*args, **kwargs) ，它的参数有： （1）url ，一个网址字符串；.（2）callback ，当request返回response后会用这个参数指定的回调函数接受response对象，所以这个回调函数一般是爬虫中 parse 系列的方法，默认是用 parse() ，而解析出错时会调用 errback 指定的回调函数。（3） method， 字符串，发起请求的方法，默认是 GET .（4） meta ，字典，它会赋给request.meta数据成员（浅拷贝），就是之前提到的请求传参（但是现在更推荐使用 cb_kwargs 来请求传参）。它还会传递给对应的response对象。（5） body ，可能是bytes或str，表示请求体，实际的请求体会以二进制形式存储。（6） headers ，字典，就是请求头。（7） cookies ，字典，用于设置请求头里的cookie，但目前版本的scrapy（2.6）这个参数还不能使用，需要设置cookie的话用 request.cookies 在下载中间件或CookiesMiddlewares里设置，另外需要注意的是cookies会根据网站返回的cookies更改，这点更像真的浏览器，要关闭这个功能可以在 request.meta 里把 dont_merge_cookies 赋值为True。（8） encoding ，字符串，指定request的字符集，默认为 uft-8 。这个编码格式会用来对url进行百分号编码（如果平时观察过url，会发现空格转换成了%20，汉字也用%转码），并且将请求体转换成二进制。（9） priority ，这个请求的优先级，默认为0。scheduler将用优先级来决定请求发送的顺序，越大的优先级越高。（10） dont_filter ，scheduler会对网址进行去重过滤，这个参数为True时就不会去重过滤。比如有些网址每次访问的结果都不同，就可以把去重关闭，但要小心使用、避免loop。（11） errback ，处理request出现exception时会调用的函数，包括页面的404错误。（12） flags ，list，用来logging。（13） cb_kwargs ，字典，就是callback指定的回调函数的参数字典。下面介绍request对象的成员：（1） url ，str，请求网址。注意这个网址是被百分号编码过的。它是read-only的，要替换它请使用request对象的 replace() 方法，见后。（2） method ，str，表示请求方法，比如 "GET" ， "POST" ， "PUT" 。（3） headers ，类似字典的对象，表示请求头。（4） body ，二进制的请求体。同样是只读的，可以用 replace() 方法更改。（5） meta ，字典，用来存储一些metadata，可以被其他extensions、中间件访问。另外，request的meta字典会传递给对应的response。（6） cb_kwargs ，字典，传递给回调函数的参数字典。（7） attributes: Tuple[str, ...]= ('url', 'callback', 'method', 'headers', 'body', 'cookies', 'meta', 'encoding', 'priority', 'dont_filter', 'errback', 'flags', 'cb_kwargs') ，一个包含请求的基本信息的字符串元组。被 Request.replace() ， Request.to_dict() 和 request_from_dict() 调用。（8） copy()，返回一个请求对象的浅拷贝。（9） replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs]) ，它会返回一个请求对象，所有提到的请求对象成员会被替换为新值、未提到的保留原值，用于更改请求对象的一些成员。这里也是浅拷贝。（10） to_dict(*, spider: Optional[Spider] = None)→ dict ，和 attributes 方法功能类似，但是返回一个字典。使用这个字典，搭配 scrapy.utils.request.request_from_dict(d: dict, *, spider: Optional[Spider] = None)→ Request 方法，可以根据字典生成request对象。 spider: Optional[Spider] = None 可选项用于根据爬虫名找到对应的爬虫，也就是说这个request生成的response对象必须用该爬虫的 parse 方法。下面介绍一下errback回调函数的书写格式。它接受的第一个参数必须是failure，使用例：import scrapy from scrapy.spidermiddlewares.httperror import HttpError from twisted.internet.error import DNSLookupError from twisted.internet.error import TimeoutError, TCPTimedOutError class ErrbackSpider(scrapy.Spider): name = "errback_example" start_urls = [ "http://www.httpbin.org/", # HTTP 200 expected "http://www.httpbin.org/status/404", # Not found error "http://www.httpbin.org/status/500", # server issue "http://www.httpbin.org:12345/", # non-responding host, timeout expected "https://example.invalid/", # DNS error expected ] def start_requests(self): for u in self.start_urls: yield scrapy.Request(u, callback=self.parse_httpbin, errback=self.errback_httpbin, dont_filter=True) def parse_httpbin(self, response): self.logger.info('Got successful response from {}'.format(response.url)) # do something useful here... def errback_httpbin(self, failure): # log all failures self.logger.error(repr(failure)) # in case you want to do something special for some errors, # you may need the failure's type: if failure.check(HttpError): # these exceptions come from HttpError spider middleware # you can get the non-200 response response = failure.value.response self.logger.error('HttpError on %s', response.url) elif failure.check(DNSLookupError): # this is the original request request = failure.request self.logger.error('DNSLookupError on %s', request.url) elif failure.check(TimeoutError, TCPTimedOutError): request = failure.request self.logger.error('TimeoutError on %s', request.url)以及用failure.request.cb_kwargs的例子：def parse(self, response): request = scrapy.Request('http://www.example.com/index.html', callback=self.parse_page2, errback=self.errback_page2, cb_kwargs=dict(main_url=response.url)) yield request def parse_page2(self, response, main_url): pass def errback_page2(self, failure): yield dict( main_url=failure.request.cb_kwargs['main_url'], )下面介绍一下meta这个超数据。2.6版本的scrapy，一般不用它来请求传参，而是用于在各组件之间传递信息。它自带的键有：bindaddress cookiejar dont_cache dont_merge_cookies dont_obey_robotstxt dont_redirect dont_retry download_fail_on_dataloss download_latency download_maxsize download_timeout ftp_password (See FTP_PASSWORD for more info) ftp_user (See FTP_USER for more info) handle_httpstatus_all handle_httpstatus_list max_retry_times proxy redirect_reasons redirect_urls referrer_policy比较值得注意的可能是proxy，可以在下载中间件中为请求对象设置代理。另外scrapy还提供了一些request对象的子类，包括：（1）FormRequest，主要用于处理html的表单，比如用于模拟登录（简单的输入用户名密码），但是这个功能使用request对象也不复杂。（2）JsonRequest，用于处理会返回json数据的请求，也没什么用。最后介绍一下response对象。构造函数为 classscrapy.http.Response(*args, **kwargs) ，参数有：（1） url ，字符串，响应来源网址（原文未提到是否被百分号转义过，似乎没有；原文提到Response.request.url doesn’t always equal Response.url）。（2） status ，int，响应的状态码。这和http协议有关，正常情况下一般是200，如果是404等就表示错误。（3） headers ，字典，响应头。（4） body ，bytes，二进制的响应体。如果希望拿到响应源码，使用 response.text （但是注意只有TextResponse及其子类才有text数据成员）。（5） flags ，list，一个包含 Response.flags 属性的列表。传递给response对象时使用的是浅拷贝。（6） request ，scrapy.Request类或其子类的实例，就是用于生成response的request对象。（7） certificate ，twisted.internet.ssl.Certificate，SSL证书。（8） ip_address ，ipaddress.IPv4Address或者ipaddress.IPv6Address，表示响应来源网址的ip地址。（9） protocol ，str，下载响应的协议，比如 "HTTP/1.0" ， "HTTP/1.1" 或者 "h2" 。response对象有以下成员：（1） url ，只读，可用 replace() 方法更改。（2） status 。（3） headers ，可用 response.headers.getlist('键名') 或 get() 方法来取出响应头中的某项值。（4） body ，只读，可用 replace() 方法更改。（5） request ，发生http重定向时，原来的request会和重定向后的response关联；这个属性只应该在爬虫文件、爬虫中间件中使用，不要在下载中间件里调用。（6） meta ，从request对象传递过来的超数据，即使发生重定向或网页retry也不会改变关联。（7） cb_kwargs ，scrapy 2.0版本后新增，从request对象那里拿到的字典参数表，会传递给回调函数。它也不受重定向和retry影响。（8） flags ，表示response状态，比如 "cached" 、 "redirected" ，会用于logging。（9） certificate 。（10） ip_address 。（11） protocol 。（12） attributes: Tuple[str, ...]= ('url', 'status', 'headers', 'body', 'flags', 'request', 'certificate', 'ip_address', 'protocol') ，一个字符串元组，包含response对象的信息，被 replace() 方法调用。（13） copy() ，返回当前response对象的浅拷贝。（14） replace([url, status, headers, body, request, flags, cls]) ，把提到的值更改，其余值不变，返回新的response对象，同样是浅拷贝。（15） urljoin(url) ，对于互相关联的url，使用这个方法会把相对url改写成绝对url。它实际上是使用了 urllib.parse.urljoin(response.url, url) 这个方法。（16） follow(url, callback=None, method='GET', headers=None, body=None, cookies=None, meta=None, encoding='utf-8', priority=0, dont_filter=False, errback=None, cb_kwargs=None, flags=None)→ Request ，返回一个request对象，参数和request对象的构造方法一样，不同在于url参数可以使用相对url，或者scrapy.link.Link对象。（17） follow_all(urls, callback=None, method='GET', headers=None, body=None, cookies=None, meta=None, encoding='utf-8', priority=0, dont_filter=False, errback=None, cb_kwargs=None, flags=None)→ Generator[Request, None, None] ，类似 follow() 方法。下面介绍一些response的子类：（1）scrapy.http.TextResponse，添加了编码方式，可用于图像、声音等媒体文件爬取。最主要的区别是在构造函数中有一个参数 encoding ，str，指定响应的解析格式，为None时会自动匹配。另外它还多出了一些属性、方法，包括 text 、 encoding 、 selector 、 xpath() 、 css() 、 json() （将json文件转换为python对象）。（2）scrapy.http.HtmlResponse和scrapy.http.XmlResponse，它们都是TextResponse的子类。12.link extractorlink extractor对象可以从response对象中提取链接，可用于全站爬取。这在前面介绍CrawlSpider类时提到过。除了用于CrawlSpider爬虫类，也可以用于其他爬虫，比如用 LxmlLinkExtractor.extract_links 方法得到响应对象里所有符合条件的link对象列表。它的构造函数为 scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), restrict_css=(), tags=('a', 'area'), attrs=('href',), canonicalize=False, unique=True, process_value=None, strip=True) ，参数有：（1） allow ，str或list，一个re字符串或re字符串列表，提取的链接必须满足这些re规则。不给出或为空字符串时，表示对链接不做re限制。（2） deny ，str或list，用于给出排除链接的re规则，它的优先级高于 allow （先排除后提取）。也可以不给出或为空str。（3） allow_domains ，str或list，str或str列表，只有response.url在这里面时才会用于提取链接。（4） deny_domains ，和 allow_domains 相反。（5） deny_extensions ，str列表，用于给出提取链接时忽略的extension，默认由 scrapy.linkextractors.IGNORED_EXTENSIONS 指定。在scrapy 2.0版本后， IGNORED_EXTENSIONS 包括 7z, 7zip, apk, bz2, cdr, dmg, ico, iso, tar, tar.gz, webm, 和 xz。（6） restrict_xpaths ，str或list，只在这些xpath表达式选中的节点里提取链接。（7） restrict_css ，str或list，功能同上。（8） restrict_text ，str或list，和 allow 类似，不同之处在于各个re规则之间是 或 的关系，也就是链接只要满足其中一条re规则就可以被提取。（9） tags ，str或list，用于提取链接的标签名必须在这里给出的标签名中。常用的标签名有 a ， area 。（10） attrs ，list，用于提取链接的属性，默认是 'href' 。（11） canonicalize ，bool，是否对被提取的链接使用 w3lib.url.canonicalize_url 来规范化，默认为False，不推荐改动这个值。（12） unique ，bool，是否在提取链接这个环节就去重。（13） process_value ，collections.abc.Callable，它是一个函数，用于对提取的链接进行处理，默认是 lambda x: x 。使用例：要提取链接 Link text ，使用：def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1)（14） strip ，bool，是否对提取到的链接str运行 strip() 方法，这个不需要改动，因为有些标准要求url前后不得有空白字符。LxmlLinkExtractor还有 extract_links(response) 方法，用于将提取规则应用于response对象、返回符合规则的link对象列表。link对象，构造函数为 scrapy.link.Link(url, text='', fragment='', nofollow=False) ，表示用linkextractor提取出的链接对象。原文给出的参数例子：Dont follow this one Parameters url – the absolute url being linked to in the anchor tag. From the sample, this is https://example.com/nofollow.html. text – the text in the anchor tag. From the sample, this is Dont follow this one. fragment – the part of the url after the hash symbol. From the sample, this is foo. nofollow – an indication of the presence or absence of a nofollow value in the rel attribute of the anchor tag.13.settings这部分内容参考 Python，爬虫与深度学习（11）——scrapy中settings的可选项 或原文。下面是一些没有提到的可选项：AJAXCRAWL_ENABLED AUTOTHROTTLE_DEBUG AUTOTHROTTLE_ENABLED AUTOTHROTTLE_MAX_DELAY AUTOTHROTTLE_START_DELAY AUTOTHROTTLE_TARGET_CONCURRENCY CLOSESPIDER_ERRORCOUNT CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT CLOSESPIDER_TIMEOUT COMMANDS_MODULE COMPRESSION_ENABLED COOKIES_DEBUG COOKIES_ENABLED FEEDS FEED_EXPORTERS FEED_EXPORTERS_BASE FEED_EXPORT_BATCH_ITEM_COUNT FEED_EXPORT_ENCODING FEED_EXPORT_FIELDS FEED_EXPORT_INDENT FEED_STORAGES FEED_STORAGES_BASE FEED_STORAGE_FTP_ACTIVE FEED_STORAGE_S3_ACL FEED_STORE_EMPTY FEED_URI_PARAMS FILES_EXPIRES FILES_RESULT_FIELD FILES_STORE FILES_STORE_GCS_ACL FILES_STORE_S3_ACL FILES_URLS_FIELD HTTPCACHE_ALWAYS_STORE HTTPCACHE_DBM_MODULE HTTPCACHE_DIR HTTPCACHE_ENABLED HTTPCACHE_EXPIRATION_SECS HTTPCACHE_GZIP HTTPCACHE_IGNORE_HTTP_CODES HTTPCACHE_IGNORE_MISSING HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS HTTPCACHE_IGNORE_SCHEMES HTTPCACHE_POLICY HTTPCACHE_STORAGE HTTPERROR_ALLOWED_CODES HTTPERROR_ALLOW_ALL HTTPPROXY_AUTH_ENCODING HTTPPROXY_ENABLED IMAGES_EXPIRES IMAGES_MIN_HEIGHT IMAGES_MIN_WIDTH IMAGES_RESULT_FIELD IMAGES_STORE IMAGES_STORE_GCS_ACL IMAGES_STORE_S3_ACL IMAGES_THUMBS IMAGES_URLS_FIELD MAIL_FROM MAIL_HOST MAIL_PASS MAIL_PORT MAIL_SSL MAIL_TLS MAIL_USER MEDIA_ALLOW_REDIRECTS METAREFRESH_ENABLED METAREFRESH_IGNORE_TAGS METAREFRESH_MAXDELAY REDIRECT_ENABLED REDIRECT_MAX_TIMES REFERER_ENABLED REFERRER_POLICY RETRY_ENABLED RETRY_HTTP_CODES RETRY_PRIORITY_ADJUST RETRY_TIMES TELNETCONSOLE_HOST TELNETCONSOLE_PASSWORD TELNETCONSOLE_PORT TELNETCONSOLE_USERNAME14.Exceptionsscrapy提供了一些exception类。有些不太能翻译，就照搬原文了。（1） scrapy.exceptions.CloseSpider(reason='cancelled') ，爬虫的回调函数要求爬虫关闭、停止时会引发这个错误。 reason 参数用于描述关闭原因。使用例：def parse_page(self, response): if 'Bandwidth exceeded' in response.body: raise CloseSpider('bandwidth_exceeded')（2） scrapy.exceptions.DontCloseSpider，This exception can be raised in a spider_idle signal handler to prevent the spider from being closed.（3） scrapy.exceptions.DropItem ，这个错误是在管道处理数据时，如果item不符合要求，可以手动raise这个错误来放弃item。（4） scrapy.exceptions.IgnoreRequest ，This exception can be raised by the Scheduler or any downloader middleware to indicate that the request should be ignored.（5） scrapy.exceptions.NotConfigured ，This exception can be raised by some components to indicate that they will remain disabled. Those components include:Extensions Item pipelines Downloader middlewares Spider middlewaresThe exception must be raised in the component’s __init__ method.（6） scrapy.exceptions.NotSupported ，This exception is raised to indicate an unsupported feature.（7） scrapy.exceptions.StopDownload(fail=True) ，Raised from a bytes_received or headers_received signal handler to indicate that no further bytes should be downloaded for a response.The fail boolean parameter controls which method will handle the resulting response:If fail=True (default), the request errback is called. The response object is available as the response attribute of the StopDownload exception, which is in turn stored as the value attribute of the received Failure object. This means that in an errback defined as def errback(self, failure), the response can be accessed though failure.value.response.If fail=False, the request callback is called instead.In both cases, the response could have its body truncated: the body contains all bytes received up until the exception is raised, including the bytes received in the signal handler that raises the exception. Also, the response object is marked with "download_stopped" in its Response.flags attribute.Note: fail is a keyword-only parameter, i.e. raising StopDownload(False) or StopDownload(True) will raise a TypeError.See the documentation for the bytes_received and headers_received signals and the Stopping the download of a Response topic for additional information and examples.15.loggingscrapy.log是从python自带的logging衍生而来，但是舍弃了很多部分。logging模块里有5种log等级，从高到低依次为： logging.CRITICAL ， logging.ERROR ， logging.WARNING ， logging.INFO ， logging.DEBUG 。原生的logging模块使用例如下：import logging logging.warning("This is a warning") logging.log(logging.WARNING, "This is a warning")使用logging模块时，常用的做法是为每个模块单独开一个log文件（单独的log handler），如下：import logging logger = logging.getLogger(__name__) logger.warning("This is a warning")在scrapy中，爬虫对象有自己的logger，使用例如下：import scrapy class MySpider(scrapy.Spider): name = 'myspider' start_urls = ['https://scrapy.org'] def parse(self, response): self.logger.info('Parse function called on %s', response.url)这个logger handler使用的是爬虫的name。然后，在settings.py文件中对log进行一些设置，可选项有：LOG_FILE LOG_FILE_APPEND LOG_ENABLED LOG_ENCODING LOG_LEVEL LOG_FORMAT LOG_DATEFORMAT LOG_STDOUT LOG_SHORT_NAMES详细的解释见 Python，爬虫与深度学习（11）——scrapy中settings的可选项 ，其余的就不多介绍了。16.Stats Collection收集爬虫的状态信息，这个部分在分布式爬虫中用的更多，但是有现成的分布式scrapy（scrapy-redis、scrapyd），这里就不介绍了。17.Sending e-mail简单来说就是scrapy提供了发送邮件的功能。但是其实python自己的smtplib用来发邮件绰绰有余，另外邮件更多用于分布式，最后即使是分布式scrapy更多用的也是网页UI，方便快捷，笔者感觉发送邮件的功能有些鸡肋（除非你需要在上班时摸鱼查看爬虫发送来的邮件），但还是介绍一下。使用例如下：from scrapy.mail import MailSender mailer = MailSender() #使用构造函数生成 mailer = MailSender.from_settings(settings) #使用from_settings方法，从settings生成 mailer.send(to=["someone@example.com"], subject="Some subject", body="Some body", cc=["another@example.com"]) #发送邮件的方法MailSender的构造函数为 scrapy.mail.MailSender(smtphost=None, mailfrom=None, smtpuser=None, smtppass=None, smtpport=None) ，参数有：（1） smtphost (str or bytes) – the SMTP host to use for sending the emails. If omitted, the MAIL_HOST setting will be used.（2） mailfrom (str) – the address used to send emails (in the From: header). If omitted, the MAIL_FROM setting will be used.（3） smtpuser – the SMTP user. If omitted, the MAIL_USER setting will be used. If not given, no SMTP authentication will be performed.（4） smtppass (str or bytes) – the SMTP pass for authentication.（5） smtpport (int) – the SMTP port to connect to（6） smtptls (bool) – enforce using SMTP STARTTLS（7） smtpssl (bool) – enforce using a secure SSL connection如果对smtp邮件服务感到陌生，可参考 Docker树莓派实践——Typecho及其备份 这篇文章中typecho的CommentNotifier插件这部分入门。18.Telnet Consolescrapy提供了telent控制台来连接、操作scrapy项目，这在分布式中用的更多，这里也不介绍了。19.FAQ官方文档提供的一些Frequently Asked Questions，复制如下：How does Scrapy compare to BeautifulSoup or lxml?¶ BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them. Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any Python code. In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django. Can I use Scrapy with BeautifulSoup?¶ Yes, you can. As mentioned above, BeautifulSoup can be used for parsing HTML responses in Scrapy callbacks. You just have to feed the response’s body into a BeautifulSoup object and extract whatever data you need from it. Here’s an example spider using BeautifulSoup API, with lxml as the HTML parser: from bs4 import BeautifulSoup import scrapy class ExampleSpider(scrapy.Spider): name = "example" allowed_domains = ["example.com"] start_urls = ( 'http://www.example.com/', ) def parse(self, response): # use lxml to get decent HTML parsing speed soup = BeautifulSoup(response.text, 'lxml') yield { "url": response.url, "title": soup.h1.string } Note BeautifulSoup supports several HTML/XML parsers. See BeautifulSoup’s official documentation on which ones are available. Did Scrapy “steal” X from Django?¶ Probably, but we don’t like that word. We think Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy. We believe that, if something is already done well, there’s no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve. We’d be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us! Does Scrapy work with HTTP proxies?¶ Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See HttpProxyMiddleware. How can I scrape an item with attributes in different pages?¶ See Passing additional data to callback functions. Scrapy crashes with: ImportError: No module named win32api¶ You need to install pywin32 because of this Twisted bug. How can I simulate a user login in my spider?¶ See Using FormRequest.from_response() to simulate a user login. Does Scrapy crawl in breadth-first or depth-first order?¶ By default, Scrapy uses a LIFO queue for storing pending requests, which basically means that it crawls in DFO order. This order is more convenient in most cases. If you do want to crawl in true BFO order, you can do it by setting the following settings: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue' While pending requests are below the configured values of CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN or CONCURRENT_REQUESTS_PER_IP, those requests are sent concurrently. As a result, the first few requests of a crawl rarely follow the desired order. Lowering those settings to 1 enforces the desired order, but it significantly slows down the crawl as a whole. My Scrapy crawler has memory leaks. What can I do?¶ See Debugging memory leaks. Also, Python has a builtin memory leak issue which is described in Leaks without leaks. How can I make Scrapy consume less memory?¶ See previous question. How can I prevent memory errors due to many allowed domains?¶ If you have a spider with a long list of allowed_domains (e.g. 50,000+), consider replacing the default OffsiteMiddleware spider middleware with a custom spider middleware that requires less memory. For example: If your domain names are similar enough, use your own regular expression instead joining the strings in allowed_domains into a complex regular expression. If you can meet the installation requirements, use pyre2 instead of Python’s re to compile your URL-filtering regular expression. See issue 1908. See also other suggestions at StackOverflow. Note Remember to disable scrapy.spidermiddlewares.offsite.OffsiteMiddleware when you enable your custom implementation: SPIDER_MIDDLEWARES = { 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None, 'myproject.middlewares.CustomOffsiteMiddleware': 500, } Can I use Basic HTTP Authentication in my spiders?¶ Yes, see HttpAuthMiddleware. Why does Scrapy download pages in English instead of my native language?¶ Try changing the default Accept-Language request header by overriding the DEFAULT_REQUEST_HEADERS setting. Where can I find some example Scrapy projects?¶ See Examples. Can I run a spider without creating a project?¶ Yes. You can use the runspider command. For example, if you have a spider written in a my_spider.py file you can run it with: scrapy runspider my_spider.py See runspider command for more info. I get “Filtered offsite request” messages. How can I fix them?¶ Those messages (logged with DEBUG level) don’t necessarily mean there is a problem, so you may not need to fix them. Those messages are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider. For more info see: OffsiteMiddleware. What is the recommended way to deploy a Scrapy crawler in production?¶ See Deploying Spiders. Can I use JSON for large exports?¶ It’ll depend on how large your output is. See this warning in JsonItemExporter documentation. Can I return (Twisted) deferreds from signal handlers?¶ Some signals support returning deferreds from their handlers, others don’t. See the Built-in signals reference to know which ones. What does the response status code 999 means?¶ 999 is a custom response status code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a download delay of 2 (or higher) in your spider: class MySpider(CrawlSpider): name = 'myspider' download_delay = 2 # [ ... rest of the spider code ... ] Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting. Can I call pdb.set_trace() from my spiders to debug them?¶ Yes, but you can also use the Scrapy shell which allows you to quickly analyze (and even modify) the response being processed by your spider, which is, quite often, more useful than plain old pdb.set_trace(). For more info see Invoking the shell from spiders to inspect responses. Simplest way to dump all my scraped items into a JSON/CSV/XML file?¶ To dump into a JSON file: scrapy crawl myspider -O items.json To dump into a CSV file: scrapy crawl myspider -O items.csv To dump into a XML file: scrapy crawl myspider -O items.xml For more information see Feed exports What’s this huge cryptic __VIEWSTATE parameter used in some forms?¶ The __VIEWSTATE parameter is used in sites built with ASP.NET/VB.NET. For more info on how it works see this page. Also, here’s an example spider which scrapes one of these sites. What’s the best way to parse big XML/CSV data feeds?¶ Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory. In order to avoid parsing all the entire feed at once in memory, you can use the functions xmliter and csviter from scrapy.utils.iterators module. In fact, this is what the feed spiders (see Spiders) use under the cover. Does Scrapy manage cookies automatically?¶ Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them back on subsequent requests, like any regular web browser does. For more info see Requests and Responses and CookiesMiddleware. How can I see the cookies being sent and received from Scrapy?¶ Enable the COOKIES_DEBUG setting. How can I instruct a spider to stop itself?¶ Raise the CloseSpider exception from a callback. For more info see: CloseSpider. How can I prevent my Scrapy bot from getting banned?¶ See Avoiding getting banned. Should I use spider arguments or settings to configure my spider?¶ Both spider arguments and settings can be used to configure your spider. There is no strict rule that mandates to use one or the other, but settings are more suited for parameters that, once set, don’t change much, while spider arguments are meant to change more often, even on each spider run and sometimes are required for the spider to run at all (for example, to set the start url of a spider). To illustrate with an example, assuming you have a spider that needs to log into a site to scrape data, and you only want to scrape data from a certain section of the site (which varies each time). In that case, the credentials to log in would be settings, while the url of the section to scrape would be a spider argument. I’m scraping a XML document and my XPath selector doesn’t return any items¶ You may need to remove namespaces. See Removing namespaces. How to split an item into multiple items in an item pipeline?¶ Item pipelines cannot yield multiple items per input item. Create a spider middleware instead, and use its process_spider_output() method for this purpose. For example: from copy import deepcopy from itemadapter import is_item, ItemAdapter class MultiplyItemsMiddleware: def process_spider_output(self, response, result, spider): for item in result: if is_item(item): adapter = ItemAdapter(item) for _ in range(adapter['multiply_by']): yield deepcopy(item) Does Scrapy support IPv6 addresses?¶ Yes, by setting DNS_RESOLVER to scrapy.resolver.CachingHostnameResolver. Note that by doing so, you lose the ability to set a specific timeout for DNS requests (the value of the DNS_TIMEOUT setting is ignored). How to deal with <class 'ValueError'>: filedescriptor out of range in select() exceptions?¶ This issue has been reported to appear when running broad crawls in macOS, where the default Twisted reactor is twisted.internet.selectreactor.SelectReactor. Switching to a different reactor is possible by using the TWISTED_REACTOR setting. How can I cancel the download of a given response?¶ In some situations, it might be useful to stop the download of a certain response. For instance, sometimes you can determine whether or not you need the full contents of a response by inspecting its headers or the first bytes of its body. In that case, you could save resources by attaching a handler to the bytes_received or headers_received signals and raising a StopDownload exception. Please refer to the Stopping the download of a Response topic for additional information and examples. Running runspider I get error: No spider found in file: <filename>¶ This may happen if your Scrapy project has a spider module with a name that conflicts with the name of one of the Python standard library modules, such as csv.py or os.py, or any Python package that you have installed. See issue 2680.
步骤/目录： 1.介绍 （1）sitemap （2）robots.txt 2.使用scrapy生成sitemap本文首发于个人博客https://lisper517.top/index.php/archives/50/，转载请注明出处。本文的目的是介绍sitemap与robots.txt。本文写作日期为2022年9月14日。运行的平台为win10，编辑器为VS Code。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.介绍如果感觉这些概念有些抽象，可以访问一些网址看看范例，比如 https://www.baidu.com/robots.txt 、 https://img.kuaidaili.com/sitemap.xml 等。（1）sitemapsitemap是一个站点的网页地图，它可以帮助爬虫，给出哪些网页可供抓取（sitemap应该只包含允许爬取的网页）、和各网址的基础数据（更新时间、更新频率、重要程度等）。因为搜索引擎都需要爬虫来获取网页的详情，用sitemap可以做SEO（search engine optimization，搜索引擎优化），这样baidu、google等各大搜索引擎的爬虫能爬到自己网页的页面，则在搜索关键字时自己网页出现的频率也会升高。sitemap的格式等信息可以到 sitemap官网 查看，下面是一份范例：<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.example.com/</loc> <lastmod>2005-01-01</lastmod> <changefreq>monthly</changefreq> <priority>0.8</priority> </url> <url> <loc>http://www.example.com/.../.../</loc> <lastmod>2005-01-01</lastmod> <changefreq>monthly</changefreq> <priority>0.8</priority> </url> ... </urlset> 一个编写好的sitemap，一般会放在网站的根目录下，命名为sitemap.xml。另外，除了被动等待自己的网页被搜索引擎爬取，还可以自己向搜索引擎提交sitemap；wordpress等模板在建站后会自动生成sitemap。（2）robots.txtrobots.txt也是一个建议爬虫如何爬取自己网站的文件，同样也放在网站根目录下。它的格式为：User-agent: * Allow: / Disallow: / Sitemap: http://www.***.com/sitemap.xmlUser-agent后面用于指定搜索引擎爬虫，比如Googlebot、Baiduspider、， * 则表示对所有爬虫适用；Allow、Disallow后面跟允许或禁止爬取的目录，只写目录名不用写全网址；Sitemap则是指明自己网站的sitemap在哪里，这个只用写一次（User-agent、Allow和Disallow则可能出现多次）。另外，如果写成：User-agent: Googlebot Allow: / User-agent: * Disallow: /就是允许google爬虫访问自己的全部页面，但禁止其他爬虫访问自己的全部页面。这个例子的重点在于前面指明单个爬虫，后面用通配符*表示其他所有爬虫。2.使用scrapy生成sitemap首先要提醒，这种方式其实和爬虫软件去爬取你的页面一样。如果你的网站，网页之间内链不多，建议还是从网站的根目录或者数据库里生成sitemap。安装好python、scrapy环境后，在win10机器上的cmd中使用如下命令：cd 放scrapy爬虫的目录 scrapy startproject sitemap cd sitemap scrapy genspider sitemap_mysite example.com然后开始编辑scrapy项目。items.py如下：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class SitemapItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() url = scrapy.Field() middlewares.py如下：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html import random from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter class SitemapSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) class SitemapDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. UA_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.33', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0' ] @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called request.headers['User-Agent'] = random.choice(self.UA_list) return None def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) pipelines.py如下：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter import os class SitemapPipeline: def open_spider(self, spider): file_name = 'urls.txt' abs_file_path = os.path.join(os.getcwd(), file_name) self.fp = open(abs_file_path, 'a', encoding='utf-8') def process_item(self, item, spider): url = item['url'] self.fp.write(url + '\n') return item def close_spider(self, spider): self.fp.close()settings.py如下：# Scrapy settings for sitemap project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html LOG_LEVEL = 'ERROR' BOT_NAME = 'sitemap' SPIDER_MODULES = ['sitemap.spiders'] NEWSPIDER_MODULE = 'sitemap.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'sitemap (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'sitemap.middlewares.SitemapSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html DOWNLOADER_MIDDLEWARES = { 'sitemap.middlewares.SitemapDownloaderMiddleware': 543, } # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { 'sitemap.pipelines.SitemapPipeline': 300, } # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 最后是sitemap_mysite.py：import scrapy, re from sitemap.items import SitemapItem class SitemapMysiteSpider(scrapy.Spider): name = 'sitemap_mysite' #allowed_domains = ['example.com'] start_urls = ['https://lisper517.top/'] def parse(self, response): print(response.url) item = SitemapItem() item['url'] = response.url yield item other_urls = re.findall(r'"https://lisper517.top/.+?"', response.text) other_urls = [other_urls[i][1:-1] for i in range(len(other_urls))] for other_url in other_urls: if not other_url.startswith('https://lisper517.top/resources'): yield scrapy.Request(url=other_url, callback=self.parse)可自行把笔者的网址更改为自己的站点。在cmd中，使用 scrapy crawl sitemap_mysite 运行这个爬虫，在当前目录下会生成 urls.txt 文件，可以在这里查看自己的网站上有哪些站点。根据这份文件，生成自己的sitemap.xml和robots.txt，放到网站的根目录下即可。
步骤/目录： 1.settings的指定方法 2.setting中的常用设置与默认值 （1）爬虫效率相关 （2）字典类设置 （3）log相关 （4）其他的重要设置 （5）其他的不重要设置本文首发于个人博客https://lisper517.top/index.php/archives/49/，转载请注明出处。本文的目的是介绍scrapy的settings的可选项及使用方法。本文写作日期为2022年9月14日，主要参考了 scrapy官方文档 。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.settings的指定方法一共有5种指定settings的方法：（1）爬虫单独指定，就是在爬虫类下面用 custom_settings 数据成员指定，如：class MySpider(scrapy.Spider): name = 'myspider' custom_settings = { 'CONCURRENT_REQUESTS': 1, 'DOWNLOAD_DELAY': 10, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 }这种方法适合有多个爬虫、多个网站，但有些网站需要调整设置的时候，比较常用。（2）命令行指定，即 scrapy crawl myspider -s LOG_FILE=scrapy.log 这样的格式，缺点是能指定的项比较少，只能用来覆盖部分settings.py里的设置。另外的三种方法比较进阶，这里就不介绍了。最后，在爬虫类里，可以通过 self.settings 调出爬虫的设置。2.setting中的常用设置与默认值这里提到的settings可选项只是很少一部分，其他的要不就是更加进阶，要不就是一般不会用到。（1）爬虫效率相关CONCURRENT_ITEMS=100 ，设置每个响应拿到后有多少个item pipeline同时运行。由于爬虫的瓶颈一般在网络，这个设置没有太大用处。CONCURRENT_REQUESTS=16 ，下载器最多能同时发送多少个请求。对于一些封ip的网站，需要用这个设置。CONCURRENT_REQUESTS_PER_DOMAIN=8 ，对一个域名最多发送多少请求。对于一些封ip的网站，可以用这个设置。CONCURRENT_REQUESTS_PER_IP=0 ，对一个ip最多发送多少请求。如果设置了这个，则 CONCURRENT_REQUESTS_PER_DOMAIN 会失效。DOWNLOAD_DELAY=0 ，用于设置从同一网站爬取连续页面时的等待时间，单位为秒（可以用浮点数）。对于一些封ip的网站，需要用这个设置。这个等待时间不是定死的，实际的时间会在0.5倍~1.5倍 DOWNLOAD_DELAY 之间浮动（若不想浮动，用 RANDOMIZE_DOWNLOAD_DELAY=False 来指定，但一般不要这样做，因为有些网站通过分析请求间隔来判别爬虫）。除了在爬虫类中用 custom_settings 指定、在 settings.py 里指定，还可以通过爬虫的 download_delay 属性来指定。总结一下，如果有些网站会封访问频繁的ip，可以在爬虫类里使用： custom_settings = { 'CONCURRENT_REQUESTS': 1, 'DOWNLOAD_DELAY': 10, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 }这样的设置来限制访问频率。（2）字典类设置这些设置一般都是指定中间件等类，和它们的优先级（或者顺序），一般越小优先级越高。DOWNLOADER_MIDDLEWARES ，用于设置下载中间件及其优先级。EXTENSIONS ，类似 DOWNLOADER_MIDDLEWARES ，用于指定扩展及其优先级。ITEM_PIPELINES ，用于指定item pipelines及其优先级。（3）log相关scrapy使用的是python自带的logging模块，所以有些设置需要参考logging。LOG_ENABLED=True ，是否允许记录log。LOG_ENCODING='utf-8' ，log的字符集。LOG_FILE=None ，logging输出的文件名。为None时，使用standard error。LOG_FILE_APPEND=True ，如果为False，每次运行时会覆盖 LOG_FILE 指定的log文件。LOG_FORMAT='%(asctime)s [%(name)s] %(levelname)s: %(message)s' ，指定每条log的格式。想自己设置格式请参考logging模块的格式说明。LOG_DATEFORMAT='%Y-%m-%d %H:%M:%S' ，用于指定 LOG_FORMAT 设置中 %(asctime)s 的输出格式。LOG_LEVEL='DEBUG' ，设置log显示的最低级别。从高到低为CRITICAL, ERROR, WARNING, INFO, DEBUG。LOG_STDOUT=False ，如果为True，会将所有输出重定向到log，比如python的print函数。（4）其他的重要设置DEFAULT_REQUEST_HEADERS 设置请求头，默认为：{ 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en', }这里可以设置一些固定的项。如果想设置user-agent，一般在middlrewares.py的下载中间件里设置；这里不能设置cookies，而应该在下载中间件的子类CookiesMiddleware中对 request.cookies 赋值来进行设置，例子可参考 这篇文章 。ROBOTSTXT_OBEY=True ，如果为False，则scrapy不会遵守网站的robots.txt文件（这个文件是网站发出的声明，用于指导爬虫的行为，但只是声明、没有实际约束作用）。TEMPLATES_DIR ，用于指定startproject时的项目模板、genspider时的爬虫模板。这个文件夹默认是scrapy库路径下的templates目录。USER_AGENT="Scrapy/VERSION (+https://scrapy.org)" ，settings.py里设置UA的地方。但一般都是在下载中间件里设置，那里的设置会覆盖掉settings.py里的UA设置。（5）其他的不重要设置DEPTH_LIMIT=0 ，爬取网页的深度，0表示无限制。DEPTH_PRIORITY=0 ，用于调整不同深度的请求的优先级。优先级的算法为 request.priority = request.priority - ( depth * DEPTH_PRIORITY ) ，也就是 DEPTH_PRIORITY 为正时越深的请求优先级越低（广度优先遍历，BFO），负时相反（深度优先遍历，DFO）。scrapy默认是DFO，这在大多数情况下效率更高，如果你实在需要改成BFO，可以在settings.py中如下设置：DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'DOWNLOAD_TIMEOUT=180 ，下载器的超时时间，单位为秒。还可以通过爬虫的 download_timeout 属性，和请求的 download_timeout 属性来指定。DOWNLOAD_MAXSIZE=1073741824 ，下载器最多下载的响应大小，单位为Byte，默认的大小为1024MB。同样，还能通过爬虫的 download_maxsize 或请求的 download_maxsize 属性指定。DOWNLOAD_WARNSIZE=33554432 ，当响应大小达到这个值后下载器会发出警告，默认值为32MB。爬虫、请求的 download_warnsize 属性也可以设置。JOBDIR='' ，指定一个路径，该路径用于保存爬虫状态。与爬虫的暂停、继续有关。REDIRECT_PRIORITY_ADJUST=+2 ，如果请求被重定向，这个选项用来设置重定向后的请求与原来请求的优先级关系，正表示重定向请求的优先级更高。URLLENGTH_LIMIT=2083 ，爬取的URL最大长度。这个是在发出请求的时候过滤url用的，因为有时网页服务器或者你自己的scrapy项目代码错误，会出现很长的url。如果想去掉这个限制，把它设置为0。
步骤/目录： 1.运行mysql数据库与准备工作 2.自建库 3.创建、修改、运行scrapy项目 （1）其他配置 （2）爬虫配置 a.a2u.py b.clarketm.py c.cool_proxy.py d.free_proxy_list.py e.ip3366.py f.ipaddress.py g.jiangxianli.py h.kuaidaili.py i.pubproxy.py j.rmccurdy.py k.sunny9577.py l.TheSpeedX.py （3）运行爬虫 4.验证ip 5.日常使用本文首发于个人博客https://lisper517.top/index.php/archives/48/，转载请注明出处。本文的目的是使用 scrapy+mysql 搭建一个自己的代理ip池，即ipool。由于本文是对前文进行阶段性总结+拓展，所以内容较多，笔者希望本文单独拿出来也能成功跟着一步步操作。本文写作日期为2022年9月11日。mysql数据库运行在树莓派上，其他部分运行在win10机器；使用的编辑器为VS Code。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.运行mysql数据库与准备工作笔者在树莓派上使用docker安装mysql，需要预先安装docker、docker-compose，在树莓派上安装docker、docker-compose可查看 Docker入门（一） 。也不一定非要在树莓派上运行mysql，或者也不一定要用docker，或者也可以用redis等其他数据库代替mysql（使用其他数据库时后续的py文件可能需要自行调整），反正最终的目的是运行一个数据库，可以参考 runoob教程 使用其他方法安装运行mysql数据库。在树莓派上进行如下操作：mkdir -p /docker/mysql/data/mysql mkdir -p /docker/mysql/conf/mysql mkdir -p /docker/mysql/backup/mysql docker pull mysql:8.0.29 nano /docker/mysql/docker-compose.yml在docker-compose.yml文件中写入如下内容：version: "3.9" services: mysql: image: mysql:8.0.29 environment: MYSQL_ROOT_PASSWORD: mysqlpasswd ports: - "53306:3306" command: - mysqld - --character-set-server=utf8mb4 volumes: - /docker/mysql/data/mysql:/var/lib/mysql - /docker/mysql/conf/mysql:/etc/mysql/conf.d - /docker/mysql/backup/mysql:/backup logging: driver: syslog restart: always记得把 MYSQL_ROOT_PASSWORD: mysqlpasswd 这里改成自己想设置的密码，把 53306 也可以改成自己想要的端口（不建议用默认的3306端口）。这里的 /docker/mysql/backup/mysql 目录是为了方便docker-mysql导出、备份数据。然后运行：cd /docker/mysql docker-compose config docker-compose up -d接下来进容器服务里看看mysql是否正常运行：docker exec -it mysql-mysql-1 bash #如果是老版本的docker-compose，把这一条命令中的所有 - 换成 _ mysql -uroot -p #输入密码 SHOW DATABASES;确认mysql正常运行即可。为了保证创建database和table的命令能记录下来，这些命令将写在python文件中。然后在win10机器上准备一下环境。需要安装python3，安装scrapy和其他库，创建自建库文件夹，创建scrapy项目文件夹。win10机器安装python3的过程这里就不讲了。安装好后，在cmd中运行 pip install Scrapy ，其他可能需要的库还有requests（处理网络请求）、bs4（对响应内容进行解析）、pymysql（连接mysql数据库）、lxml（网页解析器）、cryptography（连接mysql时密码加密）、selenium（操控浏览器），在cmd中使用 pip install requests bs4 pymysql lxml cryptography selenium 下载（这几个库在本文中不一定都会用到，但学习爬虫迟早会用），注意若pip下载失败就一直使用上述命令直到成功，或者bing搜索一下pip换源。最后创建文件夹 D:\PythonLib\MyPythonLib 和 D:\spiders\scrapy ，或者你也可以自己选择其他路径。2.自建库在 D:\PythonLib\MyPythonLib 中新建一个空白的 __init__.py 文件，然后到python的安装目录里存放第三方库的文件夹（win10上一般是site-packages目录），比如 C:\Program Files\Python310\Lib\site-packages 中，新建一个 PythonLib.pth 文件，写入一行内容： D:\PythonLib 。接下来创建自建库文件，本次需要用到的自建库主要与log、mysql相关。新建 D:\PythonLib\MyPythonLib\log.py ，写入：#!/usr/bin/python # -*- coding: UTF-8 -*- import logging, time, os def get_logger(abs_path: str, log_file_name: str = '', getLogger_name: str = ''): '''传入绝对路径和log文件名，返回log对象。 :param abs_path: log文件存储的绝对路径 :param log_file_name: log文件名，若log文件名为空，则使用默认的文件名 :param getLogger_name: logger对象名，如果在同一个程序中需要同时写多个log文件，建议这个参数不要相同 :return log对象''' try: formatter = logging.Formatter( '%(lineno)d : %(asctime)s : %(levelname)s : %(funcName)s : %(message)s' ) if log_file_name == '': log_file_name = 'log#{}.txt'.format(time.strftime('-%Y-%m-%d')) fileHandler = logging.FileHandler( (os.path.join(abs_path, log_file_name)), mode='w', encoding='utf-8') fileHandler.setFormatter(formatter) if getLogger_name == '': getLogger_name = 'logger' log = logging.getLogger(getLogger_name) log.setLevel(logging.DEBUG) log.addHandler(fileHandler) return log except: raise Exception('error: MyPythonLib.log.get_logger failed ')新建 D:\PythonLib\MyPythonLib\mysql.py ，写入（注意把get_conn函数中对应的host、port、密码改成自己的mysql配置）：#!/usr/bin/python # -*- coding: UTF-8 -*- import pymysql, os, logging, time from MyPythonLib import log as mylog def get_conn(database_name: str = 'mysql'): '''指定数据库名，返回mysql连接对象。 :param database_name: 数据库名，不指定时默认用mysql库 :return Connection''' try: conn = pymysql.connect(host='127.0.0.1', port=53306, user='root', passwd='mysqlpasswd', db=database_name, charset='utf8') return conn except: raise Exception('MyPythonLib.mysql.get_conn failed ') def str_into_mysql(string: str) -> str: '''输入一个字符串，将其转换为可以在mysql中储存的形式。 这个字符串的使用位置是列值处，比如 "SELECT * FROM 表名 WHERE 列名='字符串';" 中的字符串，若包含 \\ 、 ' 、 " 这三个字符，就需要用本函数处理。 :param string: 目标字符串 :return 可以存入mysql的字符串 ''' try: string = string.replace('\\', '\\\\') string = string.replace("'", "\\'") string = string.replace('"', '\\"') return string except: raise Exception('MyPythonLib.mysql.str_into_mysql failed ') def table_exists(cur, table_name: str) -> bool: '''判断表在当前库中是否存在。 :param cur: 游标对象 :param table_name: 表名 :return 存在则返回True，否则返回False ''' try: cur.execute("SHOW TABLES;") results = cur.fetchall() for table in results: table = table[0] if table == table_name: return True return False except: raise Exception('MyPythonLib.mysql.table_exists failed ') class mysql_api_ipool(): """通过这个接口实现提交爬取的ip、取出ip的操作。""" conn = get_conn() cur = conn.cursor() #这个cursor是类公用的，用于不需要实例化对象的mysql操作 def __init__(self, log: logging.Logger = None): """:param log: 可选的log对象，若不提供，将在当前文件夹下新建logs目录、创建log文件""" if log == None: log_dir = os.path.join(os.getcwd(), 'logs') if not os.path.isdir(log_dir): os.mkdir(log_dir) log_file_name = 'mysql_api_ipool_log#{}.txt'.format( time.strftime('-%Y-%m-%d_%H_%M_%S')) log = mylog.get_logger(log_dir, log_file_name, 'mysql_api_ipool') self.log = log message = 'mysql_api_ipool object initialized' self.log.info(message) self.cur = mysql_api_ipool.conn.cursor() #对象自己的cur mysql_api_ipool.check_table() message = 'database ipool and table ipool checked' self.log.info(message) def __del__(self): self.cur.close() message = 'mysql_api_ipool object deleted' self.log.info(message) @staticmethod def check_table(): """检查一下mysql中是否已经建好了表，若表不存在则新建表。 :这里并没有检查：若表存在，表的格式是否正确。 数据库的各列含义与默认值： :no，自增主键； :ip，代理ip（"0.0.0.0"）； :port，代理ip端口（0）； :anonymous，是否为高匿，0为否，1为是，3为其他（3）； :type，0为HTTP，1为HTTPS，2为都可，3为其他（3）； :physical_address，代理ip的地理位置（"unknown"）； :response_time，响应时间，单位为ms（-1）； :last_verify，上次验证该代理ip的时间（"1000-01-01 00:00:00"）； :grade，该代理ip的分数。每验证成功一次会加分，失败则扣分，扣到0会被删除，在爬取ip时不会使用到（100）； :created，该行创建时间。""" cur, conn = mysql_api_ipool.cur, mysql_api_ipool.conn command = ''' CREATE TABLE IF NOT EXISTS ipool ( no BIGINT UNSIGNED AUTO_INCREMENT, ip VARCHAR(50) NOT NULL DEFAULT "0.0.0.0" COMMENT "IP address", port SMALLINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "port", anonymous BIT(2) NOT NULL DEFAULT 3 COMMENT "whether ip is anonymous", type BIT(2) NOT NULL DEFAULT 3 COMMENT "HTTP or HTTPS or both or others", physical_address VARCHAR(100) NOT NULL DEFAULT "unknown" COMMENT "where is the server", response_time MEDIUMINT NOT NULL DEFAULT -1 COMMENT "response_time in microseconds", last_verify DATETIME NOT NULL DEFAULT '1000-01-01 00:00:00' COMMENT "last verify time", grade TINYINT UNSIGNED NOT NULL DEFAULT 100 COMMENT "grade of ip, used for validation", created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (no));''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute("CREATE DATABASE IF NOT EXISTS ipool;") cur.execute("USE ipool;") cur.execute(command) conn.commit() except: conn.rollback() raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.check_table failed') def save_ips(self, ips_dict_list: list): """存入多个ip（以字典的列表形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 :param ips_dict_list: ip字典的列表，每个ip是一个字典。""" cur, conn, log = self.cur, mysql_api_ipool.conn, self.log if len(ips_dict_list) == 0: message = 'no ip submitted' log.info(message) return command = '''INSERT INTO ipool (ip,port,anonymous,type,physical_address,response_time,last_verify) VALUES ''' for ip_dict in ips_dict_list: ip, port, anonymous, type_, physical_address, response_time, last_verify = "0.0.0.0", 0, 3, 3, 'unknown', -1, "1000-01-01 00:00:00" if 'ip' in ip_dict.keys(): ip = ip_dict['ip'] if 'port' in ip_dict.keys(): port = ip_dict['port'] if 'anonymous' in ip_dict.keys(): anonymous = ip_dict['anonymous'] if 'type' in ip_dict.keys(): type_ = ip_dict['type'] if 'physical_address' in ip_dict.keys(): physical_address = ip_dict['physical_address'] physical_address = str_into_mysql(physical_address) if 'response_time' in ip_dict.keys(): response_time = ip_dict['response_time'] if 'last_verify' in ip_dict.keys(): last_verify = ip_dict['last_verify'] command += '("{}",{},{},{},"{}",{},"{}"),'.format( ip, port, anonymous, type_, physical_address, response_time, last_verify) message = 'trying to save ip:port={}:{}'.format(ip, port) log.info(message) command = command[:-1] + ';' try: cur.execute(command) conn.commit() message = 'saving ips successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.save_ips failed') def check_repeat(self): """检查爬到的数据里有没有重复的，只检查ip和port都相同的。 如果有重复的，保留no最小的一条记录。""" cur, conn, log = mysql_api_ipool.cur, mysql_api_ipool.conn, self.log cur.execute('USE ipool;') cur.execute( 'SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1;') repeated_items = cur.fetchall() if len(repeated_items) == 0: message = 'no repeated ip:port in ipool' log.info(message) return else: message = 'found repeated ip:port {} kinds'.format( len(repeated_items)) log.info(message) command = ''' DELETE FROM ipool WHERE no IN (SELECT no FROM (SELECT no FROM ipool WHERE (ip,port) IN (SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1) AND no NOT IN (SELECT MIN(no) FROM ipool GROUP BY ip,port HAVING COUNT(*)>1)) AS a);''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) conn.commit() message = 'repeated ip:port deleted successfully. ' log.info(message) except: conn.rollback() raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.check_repeat failed') @staticmethod def get_ips(random: bool = False, anonymous: list = [0, 1, 2, 3], type_: list = [0, 1, 2, 3], response_time: int = 30000, last_verify_interval: str = '48:00:00', grade: int = 100, limit: int = 1000) -> tuple: """根据要求返回多个ip，只返回ip和port。 :param random: 是否随机返回ip :param anonymous: 以列表形式指定匿名性 :param type_: 对HTTP还是HTTPS或其他代理 :param response_time: 响应时间在多少ms之内 :param last_verify_interval: 用于指定上次验证的时间距现在不超过多久，若为None，则是不限定验证时间 :param grade: ip的分数必须 >= grade :param limit: 最多返回几条代理ip :return tuple('ip:port','ip:port', ...)""" cur = mysql_api_ipool.cur cur.execute('USE ipool;') anonymous, type_ = tuple(anonymous), tuple(type_) if last_verify_interval == None: last_verify_interval = '99999999:00:00' command = ''' SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" AND grade >= {} ORDER BY response_time,last_verify DESC LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, grade, limit) if random: command = ''' SELECT ip,port FROM (SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" AND grade >= {} ) AS a ORDER BY RAND() LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, grade, limit) command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) except: raise Exception('MyPythonLib.mysql.mysql_api_ipool.get_ips failed') ips = cur.fetchall() ips = tuple( [ips[i][0] + ':' + str(ips[i][1]) for i in range(len(ips))]) return ips def get_to_be_validated_ips(self, last_verify_interval: str = '48:00:00', limit: int = -1): """返回需要验证的ip，只返回no、ip和port。 :param last_verify_interval: 用于指定上次验证的时间距现在大于多久 :param limit: 最多返回几条代理ip，为-1时全部取出 :return tuple((no,ip,port,garde), (no,ip,port,garde), ...)""" cur, log = self.cur, self.log cur.execute('USE ipool;') command = ''' SELECT no,ip,port,grade FROM ipool WHERE (NOW()-last_verify)>"{}" ORDER BY last_verify,no LIMIT {}; '''.format(last_verify_interval, limit) command = command.replace('\n', '') command = command.replace(' ', '') if limit == -1: command = command.replace(' LIMIT -1', '') try: cur.execute(command) except: raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.get_to_be_validated_ips failed' ) message = 'get {} to_be_validated ips'.format(limit) log.info(message) return cur.fetchall() def update_ip(self, ip_dict: dict): '''根据ip_dict更新数据，ip_dict中必须包含no、ip、port，另外还应该有至少一个需要更改的列，未提到的列不更改。''' cur, conn, log = self.cur, mysql_api_ipool.conn, self.log try: no, ip, port = ip_dict['no'], ip_dict['ip'], ip_dict['port'] except: raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.update_ip failed: ip_dict must include no,ip,port' ) if len(ip_dict) <= 3: raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.update_ip failed: at least 1 column must be updated' ) cur.execute('USE ipool;') command = 'SELECT no,ip,port FROM ipool WHERE no={};'.format(no) cur.execute(command) results = cur.fetchall() if len(results) == 0: message = 'ip no.{} not exists'.format(no) log.warning(message) return if ip != results[0][1] or port != results[0][2]: message = 'ip no.{} ip:port not right: {}:{}, recieved {}:{}, update given up'.format( no, results[0][1], results[0][2], ip, port) log.error(message) return command = '''UPDATE ipool SET ''' if 'anonymous' in ip_dict.keys(): command += 'anonymous={},'.format(ip_dict['anonymous']) if 'type' in ip_dict.keys(): command += 'type={},'.format(ip_dict['type']) if 'physical_address' in ip_dict.keys(): command += 'physical_address="{}",'.format( str_into_mysql(ip_dict['physical_address'])) if 'response_time' in ip_dict.keys(): command += 'response_time={},'.format(ip_dict['response_time']) if 'last_verify' in ip_dict.keys(): command += 'last_verify="{}",'.format(ip_dict['last_verify']) if 'grade' in ip_dict.keys(): command += 'grade={},'.format(ip_dict['grade']) command = command[:-1] + ' WHERE no={};'.format(no) try: cur.execute(command) conn.commit() message = 'update ip no.{} successfully'.format(no) log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'MyPythonLib.mysql.mysql_api_ipool.update_ip failed') 3.创建并修改scrapy项目本次使用到的提供代理ip的网站有12个，如下：http://www.ip3366.net/free/ https://www.kuaidaili.com https://ip.jiangxianli.com/ https://www.cool-proxy.net/ https://free-proxy-list.net https://www.ipaddress.com/proxy-list/ https://www.proxynova.com/proxy-server-list/ http://pubproxy.com/api/proxy?limit=5&format=txt&type=http&level=anonymous&last_check=60&no_country=CN https://www.rmccurdy.com/.scripts/proxy/good.txt https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.json https://raw.githubusercontent.com/a2u/free-proxy-list/master/free-proxy-list.txt https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list.txt https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt其中 https://www.kuaidaili.com 页数很多（最早的数据是2014年的，有4000多页），但用scrapy会封ip，所以只爬前2页。另外，还有3个可提供代理ip的网站：http://proxy-list.org/english/ https://www.proxynova.com/proxy-server-list/ http://spys.one/en/anonymous-proxy-list/这三个网站使用js生成代理ip表格，爬取它们最简单的方法是使用selenium，但它们提供的代理ip不多，且proxynova可用 https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.json 替代，所以这里就不爬了。（1）其他配置使用cmd，进行如下操作：D: cd D:\spiders\scrapy scrapy startproject ipool在资源管理器中打开 D:\spiders\scrapy\ipool\ipool ，修改一下4个文件，修改settings.py为：# Scrapy settings for ipool project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html LOG_LEVEL = 'ERROR' # 只打印错误信息 BOT_NAME = 'ipool' SPIDER_MODULES = ['ipool.spiders'] NEWSPIDER_MODULE = 'ipool.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'ipool (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False #不遵守robots.txt # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'ipool.middlewares.IpoolSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html DOWNLOADER_MIDDLEWARES = { 'ipool.middlewares.IpoolDownloaderMiddleware': 543, } # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = {'ipool.pipelines.MysqlPipeline': 1} # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' pipelines.py为：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter from MyPythonLib.mysql import mysql_api_ipool import os class MysqlPipeline: def open_spider(self, spider): self.mysql_api_ipool = mysql_api_ipool() def process_item(self, item, spider): self.mysql_api_ipool.save_ips(item['ips_dict_list']) return item def close_spider(self, spider): self.mysql_api_ipool.check_repeat()middlewares.py为：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter # 爬虫中间件 class IpoolSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) import random # 下载中间件 class IpoolDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. UA_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.33', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0' ] @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s # 拦截无异常的请求，可进行UA伪装 def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called request.headers['User-Agent'] = random.choice(self.UA_list) return None # 拦截所有响应，可用于爬动态加载的数据 def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response # 拦截发生异常的请求，可进行代理ip设置 def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) items.py为：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class IpoolItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() ips_dict_list = scrapy.Field() （2）爬虫配置要爬12个网站，所以需要写12个爬虫。其实有几个网站的数据格式差不多，这些爬虫还能再精简合并一下。这些爬虫都放在 D:\spiders\scrapy\ipool\ipool\spiders 目录下。a.a2u.pyimport scrapy, re from ipool.items import IpoolItem class A2uSpider(scrapy.Spider): name = 'a2u' start_urls = [ 'https://raw.githubusercontent.com/a2u/free-proxy-list/master/free-proxy-list.txt' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') port = int(port) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield itemb.clarketm.pyimport scrapy, re from ipool.items import IpoolItem class ClarketmSpider(scrapy.Spider): name = 'clarketm' start_urls = [ 'https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list.txt' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') ip = ip.strip() port = int(port.strip()) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item c.cool_proxy.pyimport scrapy, json, time from ipool.items import IpoolItem class CoolProxySpider(scrapy.Spider): name = 'cool_proxy' start_urls = ['https://www.cool-proxy.net/proxies.json'] def parse(self, response): ip_nodes = json.loads(response.text.replace('\'', '\"')) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node['ip'].strip() port = int(ip_node['port']) anonymous = (0, 1)[1 == ip_node['anonymous']] type_ = 3 physical_address = ip_node['country_name'].strip() response_time = int(1000 * float(ip_node['response_time_average'])) last_verify = float(ip_node['update_time']) last_verify = time.localtime(last_verify) last_verify = time.strftime('%Y-%m-%d %H:%M:%S', last_verify) ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item d.free_proxy_list.pyimport scrapy, time from ipool.items import IpoolItem class FreeProxyListSpider(scrapy.Spider): name = 'free_proxy_list' start_urls = ['https://free-proxy-list.net/'] def parse(self, response): ip_nodes = response.xpath( r'//*[@id="list"]/div/div[2]/div/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath('./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath('./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['anonymous' in ip_node.xpath('./td[5]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = (0, 1)['yes' in ip_node.xpath('./td[7]/text()').extract()[0]] physical_address = ip_node.xpath( './td[4]/text()').extract()[0].strip() response_time = 1000 last_verify = time.strftime('%Y-%m-%d %H:%M:%S') ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield iteme.ip3366.pyimport scrapy from ipool.items import IpoolItem class Ip3366Spider(scrapy.Spider): name = 'ip3366' #allowed_domains = ['www.ip3366.net'] # 允许的域名。如果域名在此之外，则不会爬取 start_urls = [ 'http://www.ip3366.net/free/', 'http://www.ip3366.net/free/?stype=2' ] # 起始url的列表 base_url = 'http://www.ip3366.net/free/' def parse(self, response): ip_nodes = response.xpath(r'//*[@id="list"]/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath(r'./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath(r'./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['高匿' in ip_node.xpath(r'./td[3]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = ( 0, 1)['HTTPS' in ip_node.xpath(r'./td[4]/text()').extract()[0]] physical_address = ip_node.xpath( r'./td[5]/text()').extract()[0].strip() response_time = 1000 * int( ip_node.xpath(r'./td[6]/text()').extract()[0].strip().replace( '秒', '')) response_time = (response_time, 250)[response_time == 0] #这个网站中有很多代理ip的延迟为0，不太准确，一律改成250即0.25s last_verify = ip_node.xpath( r'./td[7]/text()').extract()[0].strip().replace('/', '-') ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item next_url = response.xpath( r'//*[@id="listnav"]/ul/a[text()="下一页"]/@href').extract() if next_url != None and len(next_url) != 0: next_url = Ip3366Spider.base_url + next_url[0] yield scrapy.Request(url=next_url, callback=self.parse) # 回调函数是自己，即递归解析网页 # 本方法在爬虫结束时被调用 def closed(self, spider): passf.ipaddress.pyimport scrapy from ipool.items import IpoolItem class IpaddressSpider(scrapy.Spider): name = 'ipaddress' start_urls = ['https://www.ipaddress.com/proxy-list/'] def parse(self, response): ip_nodes = response.xpath(r'/html/body/div[1]/main/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath('./td[1]/a[1]/text()').extract()[0].strip() port = int( ip_node.xpath('./td[1]/text()').extract()[0].strip().replace( '"', '').replace(':', '')) anonymous = ( 0, 1)['anonymous' in ip_node.xpath('./td[2]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = 3 physical_address = ip_node.xpath( './td[3]/text()').extract()[0].strip()[3:] response_time = 1000 last_verify = ip_node.xpath( './td[4]/text()').extract()[0].strip() + ':00' ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield itemg.jiangxianli.pyimport scrapy, json from ipool.items import IpoolItem class JiangxianliSpider(scrapy.Spider): name = 'jiangxianli' start_urls = ['https://ip.jiangxianli.com/api/proxy_ips?page=1'] def parse(self, response): ip_nodes = json.loads(response.text)['data']['data'] ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for i in range(len(ip_nodes)): ip_dict = {} ip = ip_nodes[i]['ip'].strip() port = int(ip_nodes[i]['port'].strip()) anonymous = (0, 1)[2 == ip_nodes[i]['anonymity']] type_ = (0, 1)['https' == ip_nodes[i]['protocol']] physical_address = ip_nodes[i]['ip_address'].strip() response_time = ip_nodes[i]['speed'] last_verify = ip_nodes[i]['validated_at'].strip() ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item next_url = json.loads(response.text)['data']['next_page_url'] if next_url != None: yield scrapy.Request(url=next_url, callback=self.parse) h.kuaidaili.pyimport scrapy from ipool.items import IpoolItem class KuaidailiSpider(scrapy.Spider): name = 'kuaidaili' #allowed_domains = ['www.kuaidaili.com'] start_urls = [ 'https://free.kuaidaili.com/free/inha/1/', 'https://free.kuaidaili.com/free/intr/1/' ] base_url = 'https://free.kuaidaili.com' custom_settings = { 'CONCURRENT_REQUESTS': 1, 'DOWNLOAD_DELAY': 10, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1 } def parse(self, response): ip_nodes = response.xpath(r'//*[@id="list"]/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath(r'./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath(r'./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['高匿' in ip_node.xpath(r'./td[3]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = ( 0, 1)['HTTPS' in ip_node.xpath(r'./td[4]/text()').extract()[0]] physical_address = ip_node.xpath( r'./td[5]/text()').extract()[0].strip() response_time = 1000 * float( ip_node.xpath(r'./td[6]/text()').extract()[0].strip().replace( '秒', '')) last_verify = ip_node.xpath(r'./td[7]/text()').extract()[0].strip() ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item page_num = response.url.split('/')[-2] next_url = response.url.replace(page_num, str(int(page_num) + 1)) other_urls_list = response.xpath( r'//*[@id="listnav"]/ul/li/a/@href').extract() other_urls_list = [ KuaidailiSpider.base_url + other_urls_list[i] for i in range(len(other_urls_list)) ] if int(page_num) < 5 and next_url in other_urls_list: yield scrapy.Request(url=next_url, callback=self.parse) custom_settings 用于指定这个爬虫的个性化设置，本例中限制了爬取的频率；并且只爬前5页。i.pubproxy.pyimport scrapy, re from ipool.items import IpoolItem class PubproxySpider(scrapy.Spider): name = 'pubproxy' start_urls = [ 'http://pubproxy.com/api/proxy?limit=5&format=txt&type=http&level=anonymous&last_check=60&no_country=CN' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') port = int(port) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield itemj.rmccurdy.pyimport scrapy, re from ipool.items import IpoolItem class RmccurdySpider(scrapy.Spider): name = 'rmccurdy' start_urls = ['https://www.rmccurdy.com/.scripts/proxy/good.txt'] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') port = int(port) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item k.sunny9577.pyimport scrapy, json from ipool.items import IpoolItem class Sunny9577Spider(scrapy.Spider): name = 'sunny9577' start_urls = [ 'https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.json' ] def parse(self, response): ip_nodes = json.loads(response.text)['proxynova'] ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node['ip'].strip() port = int(ip_node['port'].strip()) anonymous = (0, 1)['Anonymous' in ip_node['anonymity']] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = 3 physical_address = ip_node['country'] ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item l.TheSpeedX.pyimport scrapy, re from ipool.items import IpoolItem class ThespeedxSpider(scrapy.Spider): name = 'TheSpeedX' start_urls = [ 'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt', 'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/socks5.txt', 'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/socks4.txt' ] def parse(self, response): ip_nodes = re.findall( r'[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}\.[1-2]{0,1}\d{0,2}:\d{1,5}', response.text) ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip, port = ip_node.split(':') ip = ip.strip() port = int(port.strip()) ip_dict['ip'] = ip ip_dict['port'] = port ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item （3）运行爬虫这里涉及到了多个爬虫，运行的时候一个个使用 scrapy crawl 爬虫名 比较麻烦。scrapy提供了其他方法，可同时运行一个项目中的多个爬虫或全部爬虫。以下内容参考了 这篇文章 。在 D:\spiders\scrapy\ipool\ipool 下新建一个py文件，比如 crawl_all.py ，内容为：from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings from scrapy.spiderloader import SpiderLoader # 根据项目配置获取 CrawlerProcess 实例 process = CrawlerProcess(get_project_settings()) # 获取 spiderloader 对象，以进一步获取项目下所有爬虫名称 spider_loader = SpiderLoader(get_project_settings()) # 添加全部爬虫 for spidername in spider_loader.list(): process.crawl(spidername) # 添加一个爬虫 #process.crawl('爬虫名') # 执行 process.start()如上所述，添加一个爬虫就用 process.crawl('爬虫名') ，而 spider_loader.list() 方法可获取项目中的所有爬虫名，两者搭配就可运行全部爬虫。另外需要提醒，笔者爬取的几个网站中有几个不能直接访问，比如 raw.githubusercontent.com 那几个，你可以把它们去掉，或者使用一些设置。如果需要使用V2rayN并使用requests库或scrapy，可查看 Docker实践——如何查阅wiki百科及V2Ray ，其中提到了如何使用V2rayN并运行爬虫。最后，在cmd中进入 D:\spiders\scrapy\ipool\ipool ，运行 python crawl_all.py 即可。4.验证ip原本想使用树莓派多进程验证ip，但占用树莓派资源太高，改在win10电脑上验证。在win10电脑的自建库目录中，除了 log.py 、 mysql.py ，还需要添加一个spider.py，文件内容如下：#!/usr/bin/python # -*- coding: UTF-8 -*- import requests, random headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36' } def get_session(max_retries: int = 3) -> requests.Session: '''接受最大重连数，返回一个requests会话对象 :param max_retries: 最大重连次数 :return 会话对象''' # 设置重连次数。包括开头，若出现异常，最多尝试连接4次 try: session = requests.Session() session.mount('http://', requests.adapters.HTTPAdapter(max_retries=max_retries)) session.mount('https://', requests.adapters.HTTPAdapter(max_retries=max_retries)) return session except: raise Exception('error: MyPythonLib.spider.get_session failed ') def multiprocess_job_spliter(process_num: int, to_do_list, random_shuffle: bool = True) -> dict: '''给出进程数、总任务序列，返回分好的任务列表。 :param process_num: 进程数（需要将总任务序列分为多少份） :param to_do_list: 总任务序列（元组或列表），比如待爬取的所有url字符串 :param random_shuffle: 是否打乱总任务序列的顺序 :return {0: todo_list1, 1: todo_list2, ... }''' try: to_do_list = list(to_do_list) if random_shuffle: random.shuffle(to_do_list) total_num = len(to_do_list) one_share = total_num // process_num share_list = {} for i in range(0, process_num): if i != process_num - 1: share_list[i] = to_do_list[(i * one_share):((i + 1) * one_share)] elif i == process_num - 1: share_list[i] = to_do_list[(i * one_share):] return share_list except: raise Exception( 'error: MyPythonLib.spider.multiprocess_job_spliter failed ')这个文件后面还有很多内容，但实际上只会用到 multiprocess_job_spliter 这个函数。然后在一个合适的地方放python脚本，比如就放在scrapy的ipool项目中： D:\spiders\scrapy\ipool\ipool\update_ips.py ，文件内容如下：#!/usr/bin/python # -*- coding: UTF-8 -*- from MyPythonLib import mysql, spider from MyPythonLib import log as mylog import logging, os, time, multiprocessing headers = spider.headers class update_ips(): test_http_url = 'http://example.org/' test_https_url = 'https://example.org/' def __init__(self, log: logging.Logger = None): """:param log: 可选的log对象，若不提供，将在当前文件夹下新建logs目录、创建log文件""" if log == None: log_dir = os.path.join(os.getcwd(), 'logs') if not os.path.isdir(log_dir): os.mkdir(log_dir) log_file_name = 'mysql_api_ipool_log#{}.txt'.format( time.strftime('-%Y-%m-%d_%H_%M_%S')) log = mylog.get_logger(log_dir, log_file_name, 'mysql_api_ipool') self.log = log message = 'ready to update ips' self.log.info(message) self.mysql_api_ipool = mysql.mysql_api_ipool(self.log) self.conn = self.mysql_api_ipool.conn self.cur = self.conn.cursor() self.mysql_api_ipool.check_table() message = 'database ipool and table ipool checked' self.log.info(message) self.cur.execute('USE ipool;') self.conn.commit() def __del__(self): self.cur.close() message = 'update ips complete' self.log.info(message) @staticmethod def validate_ips_single_process(ips_list: list, process_no: int = 0, grade_minus: int = 50): '''验证ip，只涉及type、last_verify、grade的改变。若grade<=0，会将这个ip删除。 :这个方法是单进程的。 :param ips_list: 待验证ip的列表，格式为 [(no,ip,port,grade), (no,ip,port,grade), ...] :param process_no: 进程编号，主要用于log文件取名 :param grade_minus: 若代理ip无法使用，则扣除该分数''' global headers log = mylog.get_logger( abs_path=os.path.join(os.getcwd(), 'logs'), log_file_name='validate_ips-{}'.format(process_no), getLogger_name=str(process_no)) message = 'received {} ips, validating...'.format(len(ips_list)) log.info(message) mysql_api_ipool = mysql.mysql_api_ipool(log) session = spider.get_session(max_retries=1) for no, ip, port, grade in ips_list: ip_port = '{}:{}'.format(ip, port) proxies = {'http': ip_port, 'https': ip_port} http_success, https_success = True, True try: session.get(url=update_ips.test_http_url, timeout=10, headers=headers, proxies=proxies) except: http_success = False try: session.get(url=update_ips.test_https_url, timeout=10, headers=headers, proxies=proxies) except: https_success = False if http_success and not https_success: type_ = 0 elif https_success and not http_success: type_ = 1 elif http_success and https_success: type_ = 2 elif not http_success and not https_success: type_ = 3 ip_dict = {} ip_dict['no'] = no ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['type'] = type_ ip_dict['last_verify'] = time.strftime('%Y-%m-%d %H:%M:%S') if type_ == 3: grade -= grade_minus else: grade = 100 ip_dict['grade'] = grade mysql_api_ipool.update_ip(ip_dict=ip_dict) def validate_ips_multiprocess(self, process_num: int = 5, limit: int = -1, grade_minus: int = 50): '''验证ip，只涉及type、last_verify、grade的改变。若grade<=0，会将这个ip删除。 :这个方法是多进程的，将会调用 validate_ips_single_process 。 :param process_num: 验证ip的进程数 :param limit: 限制待验证ip的条数，为-1时表示全部取出 :param grade_minus: 若代理ip无法使用，则扣除该分数''' cur, conn, log = self.cur, self.conn, self.log command = 'DELETE FROM ipool WHERE grade<=0;' try: cur.execute(command) conn.commit() message = 'delete ips where grade<=0 successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'update_ips.validate_ips_multiprocess failed: delete error') ips_list = self.mysql_api_ipool.get_to_be_validated_ips(limit=limit) ips_list_dict = spider.multiprocess_job_spliter(process_num, ips_list) process_pool = multiprocessing.Pool(processes=process_num) for i in range(process_num): process_pool.apply_async(update_ips.validate_ips_single_process, args=(ips_list_dict[i], i, grade_minus)) process_pool.close() process_pool.join() try: cur.execute(command) conn.commit() message = 'delete ips where grade<=0 successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception( 'update_ips.validate_ips_multiprocess failed: delete error') if __name__ == '__main__': update_ips().validate_ips_multiprocess(process_num=60, limit=-1, grade_minus=50) update_ips().validate_ips_multiprocess(process_num=60, limit=-1, grade_minus=50) 中， process_num 和 limit 可以自行修改，主要与win10机器的配置有关，验证ip的过程mysql读写频率不算高。另外，这里的验证ip只涉及了type（http还是https）、last_verify（最后验证时间）、grade（ip分数）的改变，如果想一同更新代理ip的匿名性、物理地址、响应时间，可以在这里的基础上自行更改。最后，在cmd中，使用：cd D:\spiders\scrapy\ipool\ipool python update_ips.py即可运行验证ip的py脚本。5.日常使用爬取ip和验证ip的脚本可以每天运行一次，顺序如下：（1）若需要使用V2rayN，先开启V2rayN。（2）在cmd中使用如下命令：cd D:\spiders\scrapy\ipool\ipool python crawl_all.py python update_ips.py即可。如果想把命令写成脚本、开机自动运行，可以参考 不同系统下的开机自动运行 。另外，mysql记得时常备份。使用代理ip时，用 MyPythonLib.mysql.mysql_api_ipool.get_ips 方法可取出ip。
步骤/目录： 1.应用场景与scrapy架构 2.安装scrapy与创建项目 3.使用scrapy爬取http://www.ip3366.net/free/ （1）创建项目与修改框架 （2）自建库文件的内容 （3）运行爬虫本文首发于个人博客https://lisper517.top/index.php/archives/47/，转载请注明出处。本文的目的是讲解scrapy的使用方法。本文写作日期为2022年9月10日。爬虫写的好，牢房进的早。使用爬虫时不要占用服务器太多流量，否则可能惹祸上身。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.应用场景与scrapy架构在之前 Python，爬虫与深度学习（6）——爬虫详解（二）构建自己的ip池（简易版） 这篇文章中，提到过 https://free.kuaidaili.com/free/ 这个网站，它列出了4700多页免费代理ip。爬取这些ip是比较耗时的，但还算比较简单，因为需要爬取的每个网页的网址，区别只在最后的页码数字不同。如果我遇到一个网站，我想爬取它的全站数据，也就是说从它的根页面开始，把根页面需要的数据爬取下来，然后提取出根页面上的所有符合条件的超链接（仅提取属于这个网站的，比如下面有备案信息的网页会提供一个到工信部备案的链接，这种链接就不要），依次访问这些链接，对这些链接也都重复 爬取数据、提取链接 这两步，这样一直重复下去，就能爬取该站所有在页面上出现过的链接（有些链接可能在所有页面上都没出现过，这种隐藏起来的链接就没办法了），也就是一般意义上的爬取全站数据。但是要注意，有些时候网页会故意放一些正常人无法看到的链接（在浏览器页面上无法显示的链接，比如超出显示范围，或者没有写内容的a标签等），爬虫通过阅读源码能轻易找到，但是如果访问到这个页面，网页服务器就会识别到你是爬虫，不再提供服务，这也是一种反爬方法。作为写爬虫的一方，我们能做到的是把提取链接的规则写小一点，少用正则提取页面上的所有链接。scrapy就是一种用于全站爬取的爬虫框架，关于它组件的介绍见 runoob教程 ，这里简单介绍一下。用scrapy爬取一个网站的流程是，从一个初始url开始，下载器（Downloader）通过url获得响应，拿给爬虫（Spider）解析、提取出信息和其他要访问的url，把信息交给管道（Item Pipeline）进行存储，比如存成txt文件或存入数据库；其他url则在爬虫中以请求的形式送入调度器（Scheduler），等待合适的时机交给下载器继续下载。另外还有下载中间件和Spider中间件，用于自定义配置，比如有时你想用selenium获得请求，那么就在中间件里把获得响应的部分进行修改。另外，4个组件的中心是scrapy引擎，这个核心组件负责调度、传递组件间的信息。最后，scrapy爬取网页是采用多进程的方法，所以爬起来非常快，但是也要注意可能被封ip。所以在使用scrapy之前，最好手上已经有一定的代理ip。2.安装scrapy与创建项目使用 pip install Scrapy 安装scrapy，然后在cmd或终端中，进入放scrapy爬虫项目的文件夹（这个文件夹以后可以专门用来放爬虫），创建项目：scrapy startproject 爬虫项目名然后进入项目文件夹，创建爬虫：scrapy genspider 爬虫名 "爬虫要爬的网址"网址可以随便写，文件里可以修改。另外，一个项目里可以有多个爬虫。对项目里的各个文件进行修改，最后使用 scrapy crawl 爬虫名 运行爬虫。下面就展示一下如何修改这些文件。3.使用scrapy爬取http://www.ip3366.net/free/这里以 http://www.ip3366.net/free/ 为例，展示一下scrapy的使用，需要爬取该网站的ip。以下的命令在win10机器上运行。（1）创建项目与修改框架新建一个文件夹，比如 D:\spiders\scrapy ，然后进行如下操作：scrapy startproject ipool cd ipool scrapy genspider ip3366 http://www.ip3366.net/free/在创建过程中，可以看到scrapy有框架模板、爬虫模板，目前使用的是默认模板。以后对爬虫若有一定要求，可编写自己的模板。然后在资源管理器中进入 D:\spiders\scrapy\ipool\ipool ，可以看到items、middlewares、pipelines、settings，和ip3366这五个py文件，接下来修改这几个文件。items.py原内容如下：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class IpoolItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() pass这个文件的修改很简单，它用于定义爬取字段。将items.py修改如下：# Define here the models for your scraped items # # See documentation in: # https://docs.scrapy.org/en/latest/topics/items.html import scrapy class IpoolItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() ips_dict_list = scrapy.Field() 然后是middlewares.py，就是两个中间件的配置文件，可以进行一些自定义设置，本例中只是添加一下UA。原文件如下：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter class IpoolSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) class IpoolDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called return None def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) 在下载中间件里对UA进行设置，另外笔者添加了一些注释：# Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals # useful for handling different item types with a single interface from itemadapter import is_item, ItemAdapter # 爬虫中间件 class IpoolSpiderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, or item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request or item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) import random # 下载中间件 class IpoolDownloaderMiddleware: # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. UA_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.33', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0' ] @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s # 拦截无异常的请求，可进行UA伪装 def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called request.headers['User-Agent'] = random.choice(self.UA_list) return None # 拦截所有响应，可用于爬动态加载的数据 def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response # 拦截发生异常的请求，可进行代理ip设置 def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) 接着是pipeline.py，在这里可以处理爬到的数据。管道类可以有多个，最后在setting.py里确定优先级。另外，一定要确定每个pipeline最后都return item，这样一个pipeline把spider传过来的数据（就是item）处理完后才能传给下一个优先级的pipeline。其原文件为：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter class IpoolPipeline: def process_item(self, item, spider): return item 改写为：# Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html # useful for handling different item types with a single interface from itemadapter import ItemAdapter from MyPythonLib.ipool_mysql_api import mysql_api import os class IpoolPipeline: #这个方法在父类中定义过，其只在爬虫开始时调用一次 def open_spider(self, spider): self.fp = open(os.path.join(os.getcwd(), 'ips.txt'), 'w', encoding='utf-8') #这个方法用于处理数据 def process_item(self, item, spider): for ip_dict in item['ips_dict_list']: ip = ip_dict['ip'] port = ip_dict['port'] self.fp.write(ip + ':' + str(port) + '\n') return item #这个方法在父类中定义过，其只在爬虫结束时调用一次 def close_spider(self, spider): self.fp.close() class MysqlPipeline: def open_spider(self, spider): self.mysql_api = mysql_api() def process_item(self, item, spider): self.mysql_api.save_ips(item['ips_dict_list']) return item def close_spider(self, spider): self.mysql_api.check_repeat()在这里定义了两个管道类，一个把爬到的ip、port存成txt文件，另一个把数据存入mysql。这里的MyPythonLib是笔者的自建库，其文件内容将在下一节展示。接下来是setting.py，其原文件内容如下：# Scrapy settings for ipool project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = 'ipool' SPIDER_MODULES = ['ipool.spiders'] NEWSPIDER_MODULE = 'ipool.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'ipool (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = True # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'ipool.middlewares.IpoolSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = { # 'ipool.middlewares.IpoolDownloaderMiddleware': 543, #} # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html #ITEM_PIPELINES = { # 'ipool.pipelines.IpoolPipeline': 300, #} # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 在这里添加、修改几行：# Scrapy settings for ipool project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html LOG_LEVEL = 'ERROR' # 只打印错误信息 BOT_NAME = 'ipool' SPIDER_MODULES = ['ipool.spiders'] NEWSPIDER_MODULE = 'ipool.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'ipool (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False #不遵守robots.txt # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'ipool.middlewares.IpoolSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html DOWNLOADER_MIDDLEWARES = { 'ipool.middlewares.IpoolDownloaderMiddleware': 543, } # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { 'ipool.pipelines.IpoolPipeline': 2, #item数据按对应数字从小到大的顺序流经管道 'ipool.pipelines.MysqlPipeline': 1 } # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 最后是ip3366.py，这个文件主要用于解析爬到的内容。原文件如下：import scrapy class Ip3366Spider(scrapy.Spider): name = 'ip3366' allowed_domains = ['www.ip3366.net'] start_urls = ['http://www.ip3366.net/'] def parse(self, response): pass 把它修改为：import scrapy from ipool.items import IpoolItem class Ip3366Spider(scrapy.Spider): name = 'ip3366' #allowed_domains = ['www.ip3366.net'] # 允许的域名。如果域名在此之外，则不会爬取 start_urls = [ 'http://www.ip3366.net/free/', 'http://www.ip3366.net/free/?stype=2' ] # 起始url的列表 base_url = 'http://www.ip3366.net/free/' def parse(self, response): ip_nodes = response.xpath(r'//*[@id="list"]/table/tbody/tr') ips_dict_list = [] #这是一个字典的列表，和mysql_api.save_ips里的格式匹配 for ip_node in ip_nodes: ip_dict = {} ip = ip_node.xpath(r'./td[1]/text()').extract()[0].strip() port = int(ip_node.xpath(r'./td[2]/text()').extract()[0].strip()) anonymous = ( 0, 1)['高匿' in ip_node.xpath(r'./td[3]/text()').extract()[0]] # (a,b)[条件] 也是python三元表达式的一种写法。注意条件为真时，表达式整体值为b type_ = ( 0, 1)['HTTPS' in ip_node.xpath(r'./td[4]/text()').extract()[0]] physical_address = ip_node.xpath( r'./td[5]/text()').extract()[0].strip() response_time = 1000 * int( ip_node.xpath(r'./td[6]/text()').extract()[0].strip().replace( '秒', '')) response_time = (response_time, 250)[response_time == 0] #这个网站中有很多代理ip的延迟为0，不太准确，一律改成250即0.25s last_verify = ip_node.xpath( r'./td[7]/text()').extract()[0].strip().replace('/', '-') ip_dict['ip'] = ip ip_dict['port'] = port ip_dict['anonymous'] = anonymous ip_dict['type'] = type_ ip_dict['physical_address'] = physical_address ip_dict['response_time'] = response_time ip_dict['last_verify'] = last_verify ips_dict_list.append(ip_dict) item = IpoolItem() item['ips_dict_list'] = ips_dict_list yield item next_url = response.xpath( r'//*[@id="listnav"]/ul/a[text()="下一页"]/@href').extract() if next_url != None and len(next_url) != 0: next_url = Ip3366Spider.base_url + next_url[0] yield scrapy.Request(url=next_url, callback=self.parse) # 回调函数是自己，即递归解析网页 # 本方法在爬虫结束时被调用 def closed(self, spider): pass（2）自建库文件的内容创建自建库的方法见 Python，爬虫与深度学习（7）——建立自己的库 。笔者使用的ipool_mysql_api.py文件内容为：#!/usr/bin/python # -*- coding: UTF-8 -*- import os, logging, time from MyPythonLib import mysql from MyPythonLib import log as mylog if not os.path.isdir(os.path.join(os.getcwd(), 'logs')): os.mkdir(os.path.join(os.getcwd(), 'logs')) class mysql_api(): """通过这个接口实现提交爬取的ip、取出ip的操作。""" conn = mysql.get_conn() cur = conn.cursor() #这个cur是类公用的，用于干一些不需要实例化对象的工作 def __init__(self, log: logging.Logger = None): if log == None: log = mylog.get_logger( os.path.join(os.getcwd(), 'logs'), 'mysql_api_log#{}.txt'.format( time.strftime('-%Y-%m-%d_%H_%M_%S')), 'mysql_api') self.log = log message = 'one of mysql_api object initialized' self.log.info(message) self.cur = mysql_api.conn.cursor() mysql_api.check_table() message = 'database ipool and table ipool checked' self.log.info(message) def __del__(self): self.cur.close() message = 'one of mysql_api object deleted' self.log.info(message) @staticmethod def check_table(): """检查一下mysql中是否已经建好了表，若表不存在则新建表。 这里并没有检查：若表存在，表的格式是否正确。 :数据库的各列含义与默认值： :no，自增主键； :ip，代理ip（"0.0.0.0"）； :port，代理ip端口（0）； :anonymous，是否为高匿，0为否，1为是，3为其他（3）； :type，0为HTTP，1为HTTPS，3为其他（3）； :physical_address，代理ip的地理位置（"unknown"）； :response_time，响应时间，单位为ms（-1）； :last_verify，上次验证该代理ip的时间（"1000-01-01 00:00:00"）； :created，该项目创建时间。""" cur, conn = mysql_api.cur, mysql_api.conn command = ''' CREATE TABLE IF NOT EXISTS ipool ( no BIGINT UNSIGNED AUTO_INCREMENT, ip VARCHAR(50) NOT NULL DEFAULT "0.0.0.0" COMMENT "IP address", port SMALLINT UNSIGNED NOT NULL DEFAULT 0 COMMENT "port", anonymous BIT(2) NOT NULL DEFAULT 3 COMMENT "whether ip is anonymous", type BIT(2) NOT NULL DEFAULT 3 COMMENT "HTTP or HTTPS or both or others", physical_address VARCHAR(100) NOT NULL DEFAULT "unknown" COMMENT "where is the server", response_time MEDIUMINT NOT NULL DEFAULT -1 COMMENT "response_time in microseconds", last_verify DATETIME NOT NULL DEFAULT '1000-01-01 00:00:00' COMMENT "last verify time", created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (no));''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute("CREATE DATABASE IF NOT EXISTS ipool;") cur.execute("USE ipool;") cur.execute(command) conn.commit() except: conn.rollback() raise Exception('mysql_api.check_table failed') def save_ips(self, ips_list: list): """存入多个ip（以字典的列表形式提交）。注意这里只是对不存在的数据添加默认值，没有验证数据类型等。 :数据库的各列含义与默认值： :no，自增主键； :ip，代理ip（"0.0.0.0"）； :port，代理ip端口（0）； :anonymous，是否为高匿，0为否，1为是，3为其他（3）； :type，0为HTTP，1为HTTPS，3为其他（3）； :physical_address，代理ip的地理位置（"unknown"）； :response_time，响应时间，单位为ms（-1）； :last_verify，上次验证该代理ip的时间（"1000-01-01 00:00:00"）； :created，该项目创建时间。""" cur, conn, log = self.cur, mysql_api.conn, self.log command = '''INSERT INTO ipool (ip,port,anonymous,type,physical_address,response_time,last_verify) VALUES ''' if len(ips_list) == 0: message = 'no ip from this website' log.info(message) return for ip_dict in ips_list: ip, port, anonymous, type_, physical_address, response_time, last_verify = "0.0.0.0", 0, 3, 3, 'unknown', -1, "1000-01-01 00:00:00" if 'ip' in ip_dict.keys(): ip = ip_dict['ip'] if 'port' in ip_dict.keys(): port = ip_dict['port'] if 'anonymous' in ip_dict.keys(): anonymous = ip_dict['anonymous'] if 'type' in ip_dict.keys(): type_ = ip_dict['type'] if 'physical_address' in ip_dict.keys(): physical_address = ip_dict['physical_address'] if 'response_time' in ip_dict.keys(): response_time = ip_dict['response_time'] if 'last_verify' in ip_dict.keys(): last_verify = ip_dict['last_verify'] command += '("{}",{},{},{},"{}",{},"{}"),'.format( ip, port, anonymous, type_, physical_address, response_time, last_verify) message = 'trying to save ip:port={}:{}'.format(ip, port) log.info(message) command = command[:-1] + ';' try: cur.execute(command) conn.commit() message = 'saving ips successfully' log.info(message) except: conn.rollback() log.info('错误命令为：' + command) raise Exception('mysql_api.save_ips failed') def check_repeat(self): """检查爬到的数据里有没有重复的，只检查ip和port都相同的。如果有重复的，保留最新的一条记录，也就是no最大的。""" cur, conn, log = mysql_api.cur, mysql_api.conn, self.log cur.execute('USE ipool;') cur.execute( 'SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1;') repeated_items = cur.fetchall() if len(repeated_items) == 0: message = 'no repeated ip:port in ipool' log.info(message) return else: message = 'found repeated ip:port {} kinds'.format( len(repeated_items)) log.info(message) command = ''' DELETE FROM ipool WHERE no IN (SELECT no FROM (SELECT no FROM ipool WHERE (ip,port) IN (SELECT ip,port FROM ipool GROUP BY ip,port HAVING COUNT(*)>1) AND no NOT IN (SELECT MAX(no) FROM ipool GROUP BY ip,port HAVING COUNT(*)>1)) AS a);''' command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) conn.commit() message = 'repeated ip:port deleted successfully. ' log.info(message) except: conn.rollback() raise Exception('mysql_api.check_repeat failed') @staticmethod def get_ips(random: bool = False, anonymous: list = [0, 1, 2, 3], type_: list = [0, 1, 2, 3], response_time: int = 30000, last_verify_interval: str = '48:00:00', limit: int = 1000) -> tuple: """根据要求返回多个ip，只返回ip和port。 :param random: 是否随机返回ip :param anonymous: 以列表形式指定匿名性 :param type_: 对HTTP还是HTTPS或其他代理 :param response_time: 响应时间在多少ms之内 :param last_verify_interval: 用于指定上次验证的时间距现在不超过多久 :param limit: 最多返回几条代理ip :return tuple('ip:port','ip:port', ...)""" cur = mysql_api.cur cur.execute('USE ipool;') anonymous, type_ = tuple(anonymous), tuple(type_) command = ''' SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" ORDER BY response_time,last_verify DESC LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, limit) if random: command = ''' SELECT ip,port FROM (SELECT ip,port FROM ipool WHERE anonymous IN {} AND type IN {} AND response_time BETWEEN 0 AND {} AND (NOW()-last_verify)<"{}" ) ORDER BY RAND() LIMIT {}; '''.format(anonymous, type_, response_time, last_verify_interval, limit) command = command.replace('\n', '') command = command.replace(' ', '') try: cur.execute(command) except: raise Exception('mysql_api.get_ips failed') ips = cur.fetchall() ips = tuple( [ips[i][0] + ':' + str(ips[i][1]) for i in range(len(ips))]) return ipsmysql和log这两个自建库的内容见 Python，爬虫与深度学习（7）——建立自己的库 。（3）运行爬虫使用cmd进入 D:\spiders\scrapy\ipool\ipool ，运行 scrapy crawl ip3366 即可开启爬虫。
步骤/目录： 1.应用场景 2.准备工作 3.在个人库中添加selenium相关代码 4.selenium个人库使用范例本文首发于个人博客https://lisper517.top/index.php/archives/46/，转载请注明出处。本文的目的是讲解selenium，并在个人库中添加相应代码。本文写作日期为2022年9月10日。笔者的能力有限，也并非CS从业人员，很多地方难免有纰漏或者不符合代码原则的地方，请在评论中指出。1.应用场景在爬虫泛滥的今天，各网站采用了越发先进的方法识别爬虫。如果不是经验丰富的爬虫老手，总会在爬某个网站时翻车（老手也是从翻车一步步过来的）。如果对爬取速度要求不是太高，就可以使用selenium与实际的浏览器搭配爬取网页。目前很多网站使用了JavaScript在客户端生成页面，selenium的方法就十分适合爬取这些网页；如果无论怎么办都会被识别，笔者能提供的最终解决方案是手动开启浏览器并交给selenium控制。selenium由于要操作浏览器而速度稍慢，这是其唯一的缺点。2.准备工作下载selenium：pip install selenium另外，edge刚推出时，老版本的selenium还需要下msedge-selenium-tools来帮助扩展edge功能，现在已经不用了。如果新版selenium用不了edge的话，使用 pip install msedge-selenium-tools 来安装这个扩展包，并且selenium也会回到老版本。下载浏览器：去官网下载chrome、edge、firefox浏览器，注意不要下到别人暗改的版本了。老的爬虫教程可能会推荐下载phantomjs浏览器，这是一个没有界面的浏览器，现在很多网站都对phantomjs浏览器有一些识别与限制，所以已经不建议再使用了，何况chrome、edge、firefox也支持无界面。最后是下载webdriver，即浏览器驱动，注意的是和浏览器版本匹配。有些时候浏览器升级了，也要及时更换驱动。chrome、edge、firefox的驱动下载网址各自为：https://chromedriver.chromium.org/downloads https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/ https://github.com/mozilla/geckodriver/releases以chrome为例，打开浏览器，在网址栏输入 chrome://version/ ，可以看到版本（还有 可执行文件路径、个人资料路径 ，下面可能会用到），根据版本选择对应的chrome驱动即可。最后，记下 可执行文件路径 ，把三个浏览器的驱动都放在一个文件夹中，比如 D:\webdrivers ，然后在个人库中添加selenium相关的代码。笔者下面将给出在win10机器上selenium的一种使用方法，可根据自身情况对代码进行修改。除了自己对selenium二次开发，还可以看看别人的成品，比如helium。3.在个人库中添加selenium相关代码添加一个driver类，需要实现的功能有：根据网页返回网页源码，和返回某个标签对象。from logging import Logger import selenium, os from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver import Chrome from selenium.webdriver.chrome.options import Options as ChromeOptions # chrome from selenium.webdriver import Edge from selenium.webdriver.edge.options import Options as EdgeOptions #from msedge.selenium_tools import Edge, EdgeOptions # edge(支持新版edge需要安扩展包pip install msedge-selenium-tools) from selenium.webdriver import Firefox from selenium.webdriver.firefox.options import Options as FirefoxOptions # firefox class driver(): '''用selenium提供的浏览器类。 driver().help() 可打印帮助。''' binary_location_chrome = r'C:\Program Files\Google\Chrome\Application\chrome.exe' binary_location_edge = r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe' binary_location_firefox = r'C:\Program Files\Mozilla Firefox\firefox.exe' #浏览器可执行文件的路径 driver_path = r'D:\webdrivers' #浏览器驱动的位置 driver_download_url = '''webdriver下载地址： https://chromedriver.chromium.org/downloads https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/ https://github.com/mozilla/geckodriver/releases ''' userdata_dir_chrome = os.path.join(driver_path, 'userdata_chrome') if not os.path.isdir(userdata_dir_chrome): os.mkdir(userdata_dir_chrome) userdata_dir_edge = os.path.join(driver_path, 'userdata_edge') if not os.path.isdir(userdata_dir_edge): os.mkdir(userdata_dir_edge) userdata_dir_firefox = os.path.join(driver_path, 'userdata_firefox') if not os.path.isdir(userdata_dir_firefox): os.mkdir(userdata_dir_firefox) def __init__(self, kind: str = 'chrome', options=None, headless=True, disguise=True, independent_userdata=True, log: Logger = False) -> None: ''':param kind: 指定浏览器的种类，支持 edge/chrome/firefox :param options: 是 EdgeOptions 或 ChromeOptions 或 FirefoxOptions ，与kind对应 :param headless: 浏览器是否无头（不显示界面） :param disguise: 是否伪装成正常浏览器（仅chrome和edge） :param independent_userdata: 是否使用独立的userdata。如果开启，该文件夹会在driver_path路径下 :param log: 可选的log对象，用于输出日志''' self.kind = kind self.headless = headless self.disguise = disguise self.independent_userdata = independent_userdata self.options = (options, self.get_default_options())[options == None] #这里没有检查options和kind是否对应 self.browser = self.get_browser() self.log = log def __del__(self): self.browser.quit() @staticmethod def get_options_obj(kind: str, headless: bool = True, disguise: bool = True, independent_userdata: bool = False, userdata_dir: str = ''): '''根据浏览器种类，返回一个初始的options对象。 :注意这个方法可静态调用，你可以用这个方法得到初始options、自己加其他参数。 :param kind: 指定浏览器的种类，支持 edge/chrome/firefox :param headless: 浏览器是否无头（不显示界面） :param disguise: 是否伪装成正常浏览器（仅chrome和edge） :param independent_userdata: 是否使用独立的userdata。如果开启，需要指定userdata的路径 :param userdata_dir: userdata的存放路径''' options = None try: if kind == 'chrome': options = ChromeOptions() options.binary_location = driver.binary_location_chrome options.add_argument('--disable-gpu') #据说谷歌文档提到需要加上这个属性来规避bug elif kind == 'edge': options = EdgeOptions() options.use_chromium = True options.binary_location = driver.binary_location_edge elif kind == 'firefox': options = FirefoxOptions() options.binary_location = driver.binary_location_firefox else: raise Exception( 'error: only support chrome/firefox/edge, please check') if headless: options.add_argument('--headless') if disguise: options.add_experimental_option('excludeSwitches', ['enable-automation']) #firefox的options没有add_experimental_option if independent_userdata: if userdata_dir == '' or not os.path.isdir(userdata_dir): print( 'userdata_dir wrong, please input correct userdata_dir' ) else: userdata_dir = userdata_dir.replace( '\\', '\\\\') #此处仅为windows系统设置路径 options.add_argument( '--user-data-dir={}'.format(userdata_dir)) return options except: raise Exception('driver.get_option_obj failed') def get_default_options(self): '''生成浏览器的默认配置。''' userdata_dir = '' if self.kind == 'chrome': userdata_dir = driver.userdata_dir_chrome elif self.kind == 'edge': userdata_dir = driver.userdata_dir_edge elif self.kind == 'firefox': userdata_dir = driver.userdata_dir_firefox else: raise Exception( 'error: only support chrome/firefox/edge, please check') return driver.get_options_obj(self.kind, self.headless, self.disguise, self.independent_userdata, userdata_dir) def get_browser(self): '''根据配置返回浏览器。''' kind, driver_path = self.kind, driver.driver_path driver_browser, options = None, self.options try: if kind == 'chrome': driver_browser = Chrome(executable_path=os.path.join( driver_path, 'chromedriver.exe'), options=options) elif kind == 'edge': driver_browser = Edge(executable_path=os.path.join( driver_path, 'msedgedriver.exe'), options=options) elif kind == 'firefox': driver_browser = Firefox(executable_path=os.path.join( driver_path, 'geckodriver.exe'), options=options) return driver_browser except selenium.common.exceptions.TimeoutException: raise Exception('driver.get_browser failed') def get_page_source(self, url: str, XPath_expr: str = '', timeout: int = 60) -> str: '''通过浏览器对象获取目标网页源码。 :param url: 目标网页 :param XPath_expr: XPath表达式，用于让页面显式等待 :param timeout: 等待时间 :return 网页源码''' browser, log = self.browser, self.log """script = '''Object.defineProperty(navigator, 'webdriver', {get: ()= > undefined})''' browser.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {"source": script}) #上面这段是运行js代码，将navigator的属性webdriver改为undefined，可隐藏爬虫身份 #如果设置了 disguise=True ，就不需要这个了""" try: browser.get(url) if XPath_expr != '': WebDriverWait(browser, timeout).until( EC.presence_of_element_located((By.XPATH, XPath_expr))) except selenium.common.exceptions.TimeoutException or selenium.common.exceptions.NoSuchElementException: browser.implicitly_wait(10) message = '浏览器等待超时，需要的元素仍未出现。隐式等待10s并返回页面源码，请检查是否正确' if log: log.info(message) else: print(message) return browser.page_source def get_tag_obj(self, url: str, XPath_expr: str, timeout: int = 60): '''通过浏览器对象获取目标标签。 :param url: 目标网页 :param XPath_expr: XPath表达式，一般来说就是目标标签，页面将为该标签显式等待 :param timeout: 等待时间 :return 标签对象''' browser, log = self.browser, self.log try: browser.get(url) element = WebDriverWait(browser, timeout).until( EC.presence_of_element_located((By.XPATH, XPath_expr))) return element except selenium.common.exceptions.TimeoutException or selenium.common.exceptions.NoSuchElementException: message = '浏览器等待超时，需要的元素仍未出现；或许XPath表达式书写错误，请检查' if log: log.info(message) else: print(message) return @staticmethod def help(): '''打印帮助信息。''' help_info = r"""注意，以下内容很多尚未验证。 目录：一、如何用 options 对象及其他选项设置浏览器的行为 二、browser对象的一些操作 三、对标签对象的鼠标操作 四、对标签对象的其他操作 五、显式等待与隐式等待 六、人工打开浏览器并交给程序控制 一、如何用 options 对象及其他选项设置浏览器的行为 1.options基本的设置方法（参数中用字符串指定设置）： （1）添加参数：options.add_argument() （2）添加扩展应用：options.add_extension() 和 options.add_encoded_extension() （3）添加实验性质的设置参数：options.add_experimental_option() 。firefox没有这个选项，所以仅用于chrome和edge 2.options一些常用的设置 （1）设置以开发者模式启动，并且webdriver相关的一些属性改回正常值。这个模式可以骗过爬虫识别 options.add_experimental_option('excludeSwitches', ['enable-automation']) （2）允许浏览器弹窗。如果需要用浏览器打开不只一个页面，必须把这个设置加上 options.add_argument("--disable-popup-blocking") （3）浏览器使用代理 options.add_argument("--proxy-server=http或https://代理ip:端口") （4）不加载图片。如果不爬取图片，这个选项可提升速度 options.add_argument('blink-settings=imagesEnabled=false') （5）禁用GPU显卡辅助渲染。谷歌文档提到需要加上这个属性来规避bug chrome_options.add_argument('--disable-gpu') （6）无头浏览器，即浏览器不提供可视化页面。无桌面的linux必须加这条 options.add_argument('--headless') （7）隐藏滚动条。可用于一些特殊页面 options.add_argument('--hide-scrollbars') （8）改变UA options.add_argument('user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"') （9）指定浏览器分辨率 options.add_argument('window-size=1920x3000') （10）在沙箱之外运行（以最高权限运行） options.add_argument('--no-sandbox') （11）手动指定使用的浏览器位置 options.binary_location = r"浏览器可执行文件的路径" （12）添加插件（crx文件是chrome的插件文件） options.add_extension('crx文件路径') （13）禁用JavaScript options.add_argument("--disable-javascript") 二、browser对象的一些操作 在其他地方，browser对象也常被称为driver对象。 1.browser对象的元素定位 可以使用id、name、tag_name等定位。有如下方法可使用： browser.find_elements_by_id() browser.find_elements_by_name() browser.find_elements_by_class_name() browser.find_elements_by_tag_name() browser.find_elements_by_link_text() browser.find_elements_by_partial_link_text() browser.find_elements_by_xpath() browser.find_elements_by_css_selector() 其中最推荐的是最后两个，通过xpath和css_selector查找元素。 上述方法返回的都是匹配的标签序列；把 elements 改成 element 则只返回第一个匹配的标签。 另外，4.0之后的selenium据说不支持 find_elements_by_id 这些方法了，需要改成： from selenium.webdriver.common.by import By browser.find_element(By.ID, "ID名") browser.find_element(By.NAME, "标签名") 下文也有关于By的应用。 2.browser对象的其他方法和属性 （1）到前一页或后一页：browser.forward() 和 browser.back() （2）刷新：browser.refresh() （3）获取信息：browser有一些属性，比如 browser.page_source、browser.title、browser.current_url 分别为当前页面的源码、标题、网址 （4）切换窗口或框架： switch_to_window("windowName") 和 switch_to_frame("frameName") 。框架，比如弹出一个输入用户密码的小框框就叫框架。 （5）返回到父frame：switch_to_default_content() （6）切换到警告框：a = browser.switch_to_alert() 警告框可以像标签对象一样操作。方法和属性有：a.text、a.accept()、a.dismiss()、a.send_keys() ，分别是警告内容、接受警告、关闭警告、发送按键。send_keys方法详见后。 （7）操作cookie： 获得所有cookie信息（字典形式）：get_cookies() 返回cookie中键对应的值：get_cookie("键名") 添加键值对到cookie（对已存在的键可能会替换其值）：add_cookie(cookie_dict) 删除cookie信息：delete_cookie("键名", optionsString) 。其中 optionsString 是该cookie的选项，目前支持的选项包括 路径、域 删除所有cookie：delete_all_cookies() （8）执行javascript：execute_script(js_code) ，js_code是字符串形式的js代码 （9）屏幕截图：get_screenshot_as_file(r"图片保存路径") ，这个方法可用于无头时查找问题 （10）关闭与退出：close() 关闭单个窗口，quit() 关闭所有窗口并退出 （11）页面最多加载几秒：browser.set_page_load_timeout(timeout) 页面的js最多加载几秒：browser.set_script_timeout(timeout) 三、对标签对象的鼠标操作 鼠标操作称为动作链，在设置完所有动作后用perform()方法执行。 1.一般格式： from selenium.webdriver.common.action_chains import ActionChains #a为一个标签对象 a = browser.find_elements_by_xpath(r'XPath表达式') AC = ActionChains(browser).move_to_element(a) #其他动作 AC.perform() #设置完所有动作后用perform()方法执行 2.常用的鼠标操作 左键单击：click() 右键单击：context_click() 左键双击：double_click() 拖动并松开：drag_and_drop() 在元素处悬停：move_to_element() 四、对标签对象的其他操作 常见的有输入文本，提交，键盘按键。 （1）对一个标签输入文本。常用于填写用户名密码 browser.find_element_by_xpath(r'XPath表达式').send_keys("待输入的文本") （2）清除标签中的文本。输入框里如果有默认值，可以用这个方法清除 标签对象.clear() （3）提交表单：submit() （4）元素的尺寸：size （5）元素的内容：text （6）获得属性值：get_attribute("属性名") （7）元素是否可见：is_displayed() 键盘输入按键： BackSpace：标签对象.send_keys(Keys.BACK_SPACE) Space：send_keys(Keys.SPACE) Tab：send_keys(Keys.TAB) Esc：send_keys(Keys.ESCAPE) Enter：send_keys(Keys.ENTER) Ctrl+A：send_keys(Keys.CONTROL,'a') Ctrl+C：send_keys(Keys.CONTROL,'c') Ctrl+X：send_keys(Keys.CONTROL,'x') Ctrl+V：send_keys(Keys.CONTROL,'v') F1：send_keys(Keys.F1) 下拉框选择值： 定位到下拉框标签，然后用：Select(下拉框标签对象).select_by_value('50') 类似的方法选择值 五、显式等待与隐式等待 selenium一般是用来爬取一些有js的网页，而js加载需要一定时间 可以用显式等待和隐式等待来让浏览器对象度过这段时间，前者更优 隐式等待：等待固定的秒数，使用 browser.implicitly_wait(秒数) 即可 显式等待：等待某个元素被js加载出来，一般格式如下： from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC WebDriverWait(browser, 5, 0.5).until(EC.presence_of_element_located((By.XPATH, "XPath表达式"))) WebDriverWait(浏览器对象，最多等待几秒，每过多少秒检查一次) ，超时后会抛出NoSuchElementException异常 WebDriverWait类除了until方法，还有until_not方法。 EC.presence_of_element_located() 方法需要传入 locator 作为参数，这里用的是 (By.XPATH, "XPath表达式") 六、人工打开浏览器并交给程序控制 有时想尽办法也会被网站识别出爬虫，这时可以试试人工打开浏览器并交给程序控制。以win10系统为例，流程如下： 1.在cmd或终端中输入命令、打开浏览器，比如： chrome.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_chrome" msedge.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_edge" firefox.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_firefox" --user-data-dir 参数用于设置临时的userdata文件夹，这个参数必须也加上。 如果无法找到.exe文件，可以在cmd中进入浏览器的根目录，或把根目录加到系统路径中。 2.在options对象中添加： options.add_experimental_option('debuggerAddress', '127.0.0.1:9222') 即可。 """ print(help_info)selenium的使用过程大概就是指定浏览器，指定浏览器设置（options），打开浏览器即可。详细的使用注意事项都在 driver().help() 里了，下面来试试自建库的功能。4.selenium个人库使用范例打开一个chrome浏览器，打开bing搜索：from MyPythonLib import spider #这一行是笔者的自建库 driver = spider.driver('chrome', headless=False) browser = driver.browser browser.get('http://bing.com') input()手动打开浏览器，交给selenium控制：以windows、chrome为例，在cmd中输入：cd C:\Program Files\Google\Chrome\Application chrome.exe --remote-debugging-port=9222 --user-data-dir="D:\webdrivers\userdata_chrome"注意 D:\webdrivers\userdata_chrome 需要提前创建好。最后在py文件中写入：from MyPythonLib import spider options = spider.driver.get_options_obj('chrome', headless=False, disguise=False) options.add_experimental_option('debuggerAddress', '127.0.0.1:9222') driver = spider.driver('chrome', options=options) browser = driver.browser browser.get('http://bing.com')就能看到自己刚才打开的浏览器从初始页面变成bing搜索。如果需要自己定制options，也按上面的方法，在构造driver时传入需要的options对象即可。笔者为了方便，添加了 spider.driver.get_options_obj() 方法，返回一个默认的options。
步骤/目录： 1.创建文件夹及添加路径 2.自建库的编写 （1）log.py （2）mysql.py （3）spider.py （4）interaction.py本文首发于个人博客https://lisper517.top/index.php/archives/45/，转载请注明出处。本文的目的是讲解自建库的编写，并初步添加一些函数。本文写作日期为2022年9月6日。使用的平台为win10，编辑器为VS code。在python的日常使用中，我们常常需要复用一些代码，有时可以把这些代码写成函数，存在自建库中备用。本文将演示如何自建python库，并添加一些基础的函数到自建库中。1.创建文件夹及添加路径在一个合适的地方新建一个文件夹，用于存储自己写的库。本例中，该文件夹为 D:\PythonLib\MyPythonLib （以后所有自建库都放在这个文件夹里），注意这个文件夹要创建两层。然后新建一个文件： D:\PythonLib\MyPythonLib\__init__.py 。linux下也是同样的操作。然后把这个库添加到python的库路径中。找到python的根目录里存放库的文件夹，比如 C:\Program Files\Python310\Lib\site-packages （linux系统中一般在 /usr/lib/python3/dist-packages ），在这里新建一个.pth文件，比如 PythonLibs.pth ，写上自建库文件夹的位置，即 D:\PythonLib ，linux系统可以写上 /root/PythonLib/MyPythonLib 。2.自建库的编写然后就是编写自建库，在编写自建库时应该慎重，有些地方宁可写复杂一点，也要使代码简单易懂、便于维护；在处理异常时也要更加谨慎，避免有些时候是这些自建函数出现问题，却半天排查不到；注释要详尽，确保第一次用的人也能看懂。有如下的几个函数可以写成自建库：（1）log.pyimport logging import time import os def get_logger(abs_path: str, log_file_name: str = '', getLogger_name: str = ''): '''传入绝对路径和log文件名，返回log对象。 :param abs_path: log文件存储的绝对路径 :param log_file_name: log文件名，若log文件名为空，则使用默认的文件名 :param getLogger_name: logger对象名，如果在同一个程序中需要同时写多个log文件，建议这个参数不要相同 :return log对象''' try: formatter = logging.Formatter( '%(lineno)d : %(asctime)s : %(levelname)s : %(funcName)s : %(message)s' ) if log_file_name == '': log_file_name = 'log#{}.txt'.format(time.strftime('-%Y-%m-%d')) fileHandler = logging.FileHandler( (os.path.join(abs_path, log_file_name)), mode='w', encoding='utf-8') fileHandler.setFormatter(formatter) if getLogger_name == '': getLogger_name = 'logger' log = logging.getLogger(getLogger_name) log.setLevel(logging.DEBUG) log.addHandler(fileHandler) return log except: raise Exception('error: MyPythonLib.log.get_logger failed ')（2）mysql.py注意把mysql连接的服务器ip，端口，用户密码改成自己的。import pymysql def get_conn(database_name: str = 'mysql'): '''指定数据库名，返回mysql连接对象。 :param database_name: 数据库名，不指定时默认用mysql库 :return conn''' try: conn = pymysql.connect(host='127.0.0.1', port=53306, user='root', passwd='mysqlpasswd', db=database_name, charset='utf8') return conn except: raise Exception('error: MyPythonLib.mysql.get_conn failed ') def str_into_mysql(string: str) -> str: '''输入一个字符串，将其转换为可以在mysql中储存的形式。 这个字符串的使用位置是列值处，比如 "SELECT * FROM 表名 WHERE 列名='字符串';" 中的字符串，若包含 \\ 、 ' 、 " 这三个字符，就需要用本函数处理。 :param string: 目标字符串 :return mysql可以识别的字符串 ''' try: string = string.replace('\\', '\\\\') string = string.replace("'", "\\'") string = string.replace('"', '\\"') return string except: raise Exception('error: MyPythonLib.mysql.str_into_mysql failed ') def table_exists(cur, table_name: str) -> bool: '''判断表在当前库中是否存在。 :param cur: 游标对象 :param table_name: 表名 :return 存在则返回True，否则返回False ''' try: cur.execute("SHOW TABLES;") tables = cur.fetchall() for table in tables: table = table[0] if table == table_name: return True return False except: raise Exception('error: MyPythonLib.mysql.table_exists failed ')（3）spider.pyimport requests import random def get_session(max_retries: int = 3) -> requests.Session: '''接受最大重连数，返回一个requests会话对象 :param max_retries: 最大重连次数 :return 会话对象''' # 设置重连次数。包括开头，若出现异常，最多尝试连接4次 try: session = requests.Session() session.mount('http://', requests.adapters.HTTPAdapter(max_retries=max_retries)) session.mount('https://', requests.adapters.HTTPAdapter(max_retries=max_retries)) return session except: raise Exception('error: MyPythonLib.spider.get_session failed ') def multiprocess_job_spliter(process_num: int, to_do_list, random_shuffle: bool = True) -> dict: '''给出进程数、总任务序列，返回分好的任务列表。 :param process_num: 进程数（需要将总任务序列分为多少份） :param to_do_list: 总任务序列（元组或列表），比如待爬取的所有url字符串 :param random_shuffle: 是否打乱总任务序列的顺序 :return {0: todo_list1, 1: todo_list2, ... }''' try: to_do_list = list(to_do_list) if random_shuffle: random.shuffle(to_do_list) total_num = len(to_do_list) one_share = total_num // process_num share_list = {} for i in range(0, process_num): if i != process_num - 1: share_list[i] = to_do_list[(i * one_share):((i + 1) * one_share)] elif i == process_num - 1: share_list[i] = to_do_list[(i * one_share):] return share_list except: raise Exception( 'error: MyPythonLib.spider.multiprocess_job_spliter failed ')（4）interaction.py一些命令行下交互、打印输出的小函数。import time import os def print_progress(start_time: float, progress: int, total: int, msg: str, cls=False): '''图形化打印进度及附加信息（清空屏幕）。 :param start_time: 开始时间，由time.time()得到的float :param progress: 整数，表示目前进度 :param total: 整数，表示总量 :param msg: 附加信息 :param cls: 是否清空屏幕（windows下）''' try: percent = 100. * progress / total x = int(2 * percent / 5) duration = time.time() - start_time h, m, s = duration // 3600, duration // 60 % 60, duration % 60 time_format = '{:0>2.0f}:{:0>2.0f}:{:0>2.0f}'.format(h, m, s) expect_time = duration * total / progress h, m, s = expect_time // 3600, expect_time // 60 % 60, expect_time % 60 expect_time_format = '{:0>2.0f}:{:0>2.0f}:{:0>2.0f}'.format(h, m, s) print('\r已运行{:.2f}% '.format(percent), str(progress) + '/' + str(total), time_format + '/' + expect_time_format + ' [' + '#' * x + '.' * (40 - x) + ']', msg, end='') if cls: os.system('cls') except: raise Exception( 'error: MyPythonLib.interaction.print_progress failed ')