Getting started with requests HTTP library

requests is one of the most advanced and simple to use HTTP libraries. The presentations is going to show how to get started with it, how to consume REST APIs and last but not least how to properly mock and test our code.

Transcript

1. REQUESTS Getting started with @andreagrandi Andrea Grandi HTTP library
2. WHO AM I • Andrea Grandi • I live in London (UK) • Software Developer at Government Digital Service • Python/Django developer • passionate about micro controllers, IoT, Arduino, Golang and mobile development
3. WHAT IS AN HTTP REQUEST
4. WHAT IS REQUESTS • HTTP library for Python • Supports Python 2.6, 2.7 and 3.3 - 3.5 • Used by large companies in the world: Her Majesty's Government, Amazon, Google, Twilio, Runscope, Mozilla, Heroku, PayPal, etc… • Actively maintained • Easy to use
5. PHILOSOPHY • Beautiful is better than ugly • Explicit is better than implicit • Simple is better than complex • Complex is better than complicated • Readability counts
6. WHY NOT URLLIB2?
7. urllib example import urllib2 gh_url = 'https://api.github.com' req = urllib2.Request(gh_url) password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm() password_manager.add_password(None, gh_url, 'user', 'pass') auth_manager = urllib2.HTTPBasicAuthHandler(password_manager) opener = urllib2.build_opener(auth_manager) urllib2.install_opener(opener) handler = urllib2.urlopen(req) print(handler.getcode()) print(handler.headers.getheader('content-type'))
8. requests example import requests r = requests.get( 'https://api.github.com', auth=('user', 'pass')) print(r.status_code) print(r.headers['content-type'])
9. MAKING A REQUEST
10. GET import requests r = requests.get('https://api.github.com/events')
11. GET with parameters import requests payload = {'key1': 'value1', 'key2': 'value2'} r = requests.get('http://httpbin.org/get', params=payload)
12. POST payload = {'key1': 'value1', 'key2': 'value2'} r = requests.post("http://httpbin.org/post", data=payload)
13. POST and automatically encode JSON url = 'https://api.github.com/some/endpoint' payload = {'some': 'data'} r = requests.post(url, json=payload)
14. Custom headers url = 'https://api.github.com/some/endpoint' headers = {'user-agent': 'my-app/0.0.1'} r = requests.get(url, headers=headers)
15. ALL OTHER METHODS ARE SUPPORTED • PUT • PATCH • HEAD • OPTIONS • DELETE (don’t worry if you don’t know some of them)
16. RESPONSES
17. Checking response status code >>> r = requests.get('http://httpbin.org/get') >>> r.status_code 200
18. Compare statuses with built-in constants >>> r.status_code == requests.codes.ok True
19. Raise exceptions from error status >>> bad_r = requests.get('http://httpbin.org/status/404') >>> bad_r.status_code # 404 >>> bad_r.raise_for_status() Traceback (most recent call last): File "requests/models.py", line 832, in raise_for_status raise http_error requests.exceptions.HTTPError: 404 Client Error
20. Built in JSON decoder in responses >>> import requests >>> r = requests.get('https://api.github.com/events') >>> r.json() [{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...
21. Headers in response >>> r.headers { 'content-encoding': 'gzip', 'connection': 'close', 'server': 'nginx/1.0.4', 'content-type': 'application/json' }
22. Headers in response (2) >>> r.headers['Content-Type'] 'application/json' >>> r.headers.get('content-type') 'application/json' You can access them using any casing
23. Cookies >>> url = 'http://example.com/some/cookie/setting/url' >>> r = requests.get(url) >>> r.cookies['example_cookie_name'] 'example_cookie_value'
24. Sessions s = requests.Session() s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') r = s.get('http://httpbin.org/cookies') print(r.text) # '{"cookies": {"sessioncookie": "123456789"}}'
25. Timeouts >>> requests.get('http://github.com', timeout=0.001) Traceback (most recent call last): File "<stdin>", line 1, in <module> requests.exceptions.Timeout: HTTPConnectionPool(host='github.com', port=80): Request timed out. (timeout=0.001)
26. Multiple requests • grequests: https://github.com/kennethreitz/grequests • frequests: https://github.com/i-trofimtschuk/frequests • requests-futures: https://github.com/ross/requests-futures
27. grequests import grequests urls = ['http://www.heroku.com', 'http://python-tablib.org'] rs = (grequests.get(u) for u in urls) >>> grequests.map(rs) [<Response [200]>, <Response [200]>]
28. ERRORS AND EXCEPTIONS • ConnectionError: network problems, refused connection ... • HTTPError: raised by raise_for_status() in case of unsuccessful status • TimeOut: in case the request times out • TooManyRedirects: when request reaches the maximum number of redirections
29. TESTING
30. Method example import requests def get_example(): r = requests.get('https://httpbin.org/get') return r.json()
31. Testing with Mock class TestClient(TestCase): @mock.patch('requests.get') def test_get_example_mock(self, mock_request): mocked_response = mock.Mock() mocked_response.status_code = 200 expected_dict = {"example": "foo"} mocked_response.json.return_value = expected_dict mock_request.return_value = mocked_response response = client.get_example() self.assertEqual(response, expected_dict)
32. Testing with responses @responses.activate def test_get_example_responses(self): responses.add(responses.GET, 'https://httpbin.org/get', body='{"example": "foo"}', status=200, content_type='application/json') response = client.get_example() self.assertEqual(response, {"example": "foo"})
33. How to stay in touch • blog: https://www.andreagrandi.it • Twitter: @andreagrandi • GitHub: https://github.com/andreagrandi • email: a.grandi@gmail.com • IRC: Andy80 on FreeNode (#python, #django, #python-uk ) • PGP: 7238 74F6 886D 5994 323F 1781 8CFB 47AD C384 F0CC
34. References • http://docs.python-requests.org • https://github.com/andreagrandi/requests-examples
35. Just one more thing... https://jobs.jobvite.com/gds / We are hiring!
36. Thanks!