Requests¶

Requests是用Python语言编写的，基于urllib，但是它比urllib更加方便，可以节约我们大量的工作，完全满足HTTP测试需要。

GET请求¶

基本请求¶

import requests

response = requests.get('http://httpbin.org/get')
print(response.text)

就可以打印基本请求信息

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.20.0"
  }, 
  "origin": "222.197.179.9", 
  "url": "http://httpbin.org/get"
}

带参数GET¶

import requests
response = requests.get("http://httpbin.org/get?name=germey&age=22")
print(response.text)

和下面的写法是完全一样的，下面写法的好处是不用写各种符号了。

import requests

data = {
    'name': 'germey',
    'age': 22
}
response = requests.get("http://httpbin.org/get", params=data)
print(response.text)

解析json¶

这个操作可以直接将获取的结果转化为json格式。

import requests
import json

response = requests.get("http://httpbin.org/get")
print(type(response.text))
print(response.json())  # response.json实际上就是执行了一个json.loads方法，和下面语句效果是一样的
print(json.loads(response.text))
print(type(response.json()))

获取二进制数据¶

获取二进制数据在下载图片和下载视频的时候作为常用方法。

import requests

response = requests.get("https://github.com/favicon.ico")
print(type(response.text), type(response.content))
print(response.text)
print(response.content) #使用response.content来获取图片的二进制内容

保存的话，可以使用这种方式来保存：

import requests

response = requests.get("https://github.com/favicon.ico")
with open('favicon.ico', 'wb') as f:
    f.write(response.content)
    f.close()

添加headers¶

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
response = requests.get("https://www.zhihu.com/explore", headers=headers)
print(response.text)

POST请求¶

使用POST请求一定要发一个Form Data，我们可以用data字典来传入表单。

import requests

data = {'name': 'germey', 'age': '22'}
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
response = requests.post("http://httpbin.org/post", data=data, headers=headers)
print(response.json())

响应¶

response属性¶

下面列出常用的response属性。

import requests

response = requests.get('http://www.jianshu.com')
print(type(response.status_code), response.status_code)
print(type(response.headers), response.headers)
print(type(response.cookies), response.cookies)
print(type(response.url), response.url)
print(type(response.history), response.history)

状态码判断¶

下面给出状态码的各个编号对应的名字，我们在实际使用过程中不需要记住状态码的具体数字，使用名字是一样的效果。

('continue',),
('switching_protocols',),
('processing',),
('checkpoint',),
('uri_too_long', 'request_uri_too_long'),
('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\\o/', '✓'),
('created',),
('accepted',),
('non_authoritative_info', 'non_authoritative_information'),
('no_content',),
('reset_content', 'reset'),
('partial_content', 'partial'),
('multi_status', 'multiple_status', 'multi_stati', 'multiple_stati'),
('already_reported',),
('im_used',),

# Redirection.
('multiple_choices',),
('moved_permanently', 'moved', '\\o-'),
('found',),
('see_other', 'other'),
('not_modified',),
('use_proxy',),
('switch_proxy',),
('temporary_redirect', 'temporary_moved', 'temporary'),
('permanent_redirect',
      'resume_incomplete', 'resume',), # These 2 to be removed in 3.0

# Client Error.
('bad_request', 'bad'),
('unauthorized',),
('payment_required', 'payment'),
('forbidden',),
('not_found', '-o-'),
('method_not_allowed', 'not_allowed'),
('not_acceptable',),
('proxy_authentication_required', 'proxy_auth', 'proxy_authentication'),
('request_timeout', 'timeout'),
('conflict',),
('gone',),
('length_required',),
('precondition_failed', 'precondition'),
('request_entity_too_large',),
('request_uri_too_large',),
('unsupported_media_type', 'unsupported_media', 'media_type'),
('requested_range_not_satisfiable', 'requested_range', 'range_not_satisfiable'),
('expectation_failed',),
('im_a_teapot', 'teapot', 'i_am_a_teapot'),
('misdirected_request',),
('unprocessable_entity', 'unprocessable'),
('locked',),
('failed_dependency', 'dependency'),
('unordered_collection', 'unordered'),
('upgrade_required', 'upgrade'),
('precondition_required', 'precondition'),
('too_many_requests', 'too_many'),
('header_fields_too_large', 'fields_too_large'),
('no_response', 'none'),
('retry_with', 'retry'),
('blocked_by_windows_parental_controls', 'parental_controls'),
('unavailable_for_legal_reasons', 'legal_reasons'),
('client_closed_request',),

# Server Error.
('internal_server_error', 'server_error', '/o\\', '✗'),
('not_implemented',),
('bad_gateway',),
('service_unavailable', 'unavailable'),
('gateway_timeout',),
('http_version_not_supported', 'http_version'),
('variant_also_negotiates',),
('insufficient_storage',),
('bandwidth_limit_exceeded', 'bandwidth'),
('not_extended',),
('network_authentication_required', 'network_auth', 'network_authentication'),

下面两种写法实现功能是一样的：

import requests

response = requests.get('http://www.jianshu.com/hello.html')
exit() if not response.status_code == requests.codes.not_found else print('404 Not Found')

import requests

response = requests.get('http://www.jianshu.com')
exit() if not response.status_code == 200 else print('Request Successfully')

高级操作¶

文件上传¶

文件上传需要使用POST操作。我们使用open把文件读取出来，然后赋值给files参数就可以实现文件上传操作了。

import requests

files = {'file': open('favicon.ico', 'rb')}
response = requests.post("http://httpbin.org/post", files=files)
print(response.text)

会话维持¶

有了cookies就可以模拟维持一个登录状态了。可以用来做模拟登陆。

需要注意的是，我们如果想要设置一个cookies，然后读出这个cookies需要设置一个Session对象。如果不设置Session对象，那么相当于两个浏览器在进行get。第二个浏览器是拿不到第一个浏览器的请求的。通过Session这样来维持一个登录会话。

import requests

s = requests.Session()
s.get('http://httpbin.org/cookies/set/number/123456789')
response = s.get('http://httpbin.org/cookies')
print(response.text)

证书验证¶

我们在浏览网页的时候，网页证书可能是错误的，直接使用requests就会报出一个SSL认证的错误。我们爬网页的时候如果这个网页是https协议的话，他会首先检查一下证书的合法性。如果证书不合法，就会报出SSL Error，那么程序就会中断掉。如果想要避免这个问题，关掉证书验证功能就可以了。

import requests

response = requests.get('https://www.12306.cn', verify=False)
print(response.status_code)

但是，依旧会有警告信息，如果想要看起来好看一些，可以添加下面代码把警告信息消除掉。

import requests
from requests.packages import urllib3
urllib3.disable_warnings()
response = requests.get('https://www.12306.cn', verify=False)
print(response.status_code)

代理设置¶

import requests

proxies = {
  "http": "http://127.0.0.1:9743",
  "https": "https://127.0.0.1:9743",
}

response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)

如果代理需要用户名和密码的话，可以这样写，其中user的位置写用户名，password的位置写密码。@作为分隔符。

import requests

proxies = {
    "http": "http://user:password@127.0.0.1:9743/",
}
response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)

如果你的代理是ShadowSocks的话，那么可以这样：

pip3 install 'requests[socks]'

import requests

proxies = {
    'http': 'socks5://127.0.0.1:9742',
    'https': 'socks5://127.0.0.1:9742'
}
response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)

超时设置¶

import requests
from requests.exceptions import ReadTimeout #引入Timeout类
try:
    response = requests.get("http://httpbin.org/get", timeout = 0.5)
    print(response.status_code)
except ReadTimeout:
    print('Timeout')

认证设置¶

有些网站需要输入用户名和密码，登录的时候需要进行认证。可以通过给requests传入一个auth参数来做到登录验证。

import requests
from requests.auth import HTTPBasicAuth

r = requests.get('http://120.27.34.24:9001', auth=HTTPBasicAuth('user', '123'))
print(r.status_code)

或者也可以写成这样：

import requests

r = requests.get('http://120.27.34.24:9001', auth=('user', '123'))
print(r.status_code)

异常处理¶

这些具体异常Error的定义和其他有什么Error可以从Requests官方文档中找到。

import requests
from requests.exceptions import ReadTimeout, ConnectionError, RequestException
try:
    response = requests.get("http://httpbin.org/get", timeout = 0.5)
    print(response.status_code)
except ReadTimeout:
    print('Timeout')
except ConnectionError:
    print('Connection error')
except RequestException:
    print('Error')