Building a Wrapper API

The case for abstraction

Transcript

1. The case for abstraction Building a Wrapper API
2. Introduction
3. Wrapper APIs
4. Define the problem... • I need a way to manage Elasticsearch indices • For some actions, it takes at least 9 API calls to complete • These API calls share a lot of redundant configuration • Reduce redundant API calls and code • Simplify/Abstract the way they're called so it's easier
5. What is a Wrapper API? • Makes a low-level API more accessible • To you (because you create it) • To others (because you share it) • In effect, it becomes a high-level API
6. What is a Wrapper API? • Abstract frequently used code • Simplify complex code snippets • Extends a low-level API • Catch exceptions or conditions differently
7. import unwrapped1, unwrapped2 my_list = [] my_object = unwrapped(*args, **kwargs) value1 = my_object.method(*args) my_object.method2(value1) value2 = function1(my_object.accessor) for value in function2(value2): my_list.append(value) other_obj = unwrapped2(my_object,my_list) ... ... Lots of code goes here ... including all necessary functions ... and it gets hard to read now ... end_goal = functionN(*args, **kwargs)
8. import wrappername my_object = wrappername(*args, **kwargs) my_object.method(*args, **kwargs) end_goal = my_object.accessor
9. What isn't a Wrapper API? • Don't repackage existing API calls for no reason • If it takes just as many lines to re-make the call, it's not a simplification, and is a waste of time.
10. Coding That's why you're here, right?
11. Plan it out • Build on the low-level API/module/class • What behaviors are needed? • What does the final result need to be? • How can I test it? (Unit and/or Integration)
12. File layout my_project ├── docs │ ├── Makefile │ ├── conf.py │ └── index.rst ├── my_project │ ├── __init__.py │ ├── _version.py │ └── my_first_module.py ├── LICENSE.txt ├── MANIFEST.in ├── README.rst ├── requirements.txt ├── setup.py └── test ├── __init__.py ├── integration ├── unit └── run_tests.py
13. Identify classes & methods • Define an IndexList object • Automatically populate with all current indices • Filter that list of indices any which way we like • Make the result available as an accessor
14. Identify classes & methods • Define an Action object • Receives an IndexList object as an argument • Perform the designated action against the IndexList • All Actions should have similar API structure
15. class IndexList(object): def __init__(self, client): # other code here self.client = client self.indices = [] self.__get_indices() def __get_indices(self): # code here to populate self.indices def __internal_method(self, *args, **kwargs): # code here to process whatever is needed def class_method(self, *args, **kwargs): # code here to process whatever is needed def filter_methodN(self, *args, **kwargs): # code here to filter self.indices
16. class IndexList: def __init__(self, client): # code here def filter_by_regex(self, *args, **kwargs): # code here def filter_by_age(self, *args, **kwargs): # code here ilo = IndexList(client) ilo.filter_by_regex(*args, **kwargs) ilo.filter_by_age(*args, **kwargs) # Add as many more filters as there are (loop them, even) ilo.indices = the list of indices left after filtering.
17. # Pretend the IndexList stuff is above here class Action: def __init__(self, ilo): # setup instance vars here # code here def do_dry_run(self): # code here def do_action(self): # code here ilo = IndexList(client) ilo.filter_by_regex(*args, **kwargs) ilo.filter_by_age(*args, **kwargs) # Add as many more filters as there are (loop them, even) action = Action(ilo) action.do_dry_run() action.do_action()
18. Classes or functions? • Does your wrapper need classes? • Can it work with utility functions alone? • Why not both?
19. Testing Don't get caught off-guard...
20. Why test? • Demonstrate code reliability to yourself and others • Quick way to test & guarantee version compatibility • Python versions (2.x, 3.x) • Endpoint versions • Catch issues before they are in production
21. setup( name = "elasticsearch-curator", version = get_version(), author = "Elastic", author_email = "info@elastic.co", description = "Tending your Elasticsearch indices", long_description=fread('README.rst'), url = "http://github.com/elastic/curator", license = "Apache License, Version 2.0", install_requires = get_install_requires(), keywords = "elasticsearch time-series indexed index-expiry", packages = ["curator"], classifiers=[ "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", ], test_suite = "test.run_tests.run_all", tests_require = ["mock", "nose", "coverage", "nosexcover"], ...
22. #!/usr/bin/env python from __future__ import print_function import sys from os.path import dirname, abspath import nose def run_all(argv=None): sys.exitfunc = lambda: sys.stderr.write('Shutting down....\n') # always insert coverage when running tests through setup.py if argv is None: argv = [ 'nosetests', '--with-xunit', '--logging-format=%(levelname)s %(name)22s %(funcName)22s:%(lineno)-4d %(message)s', '--with-xcoverage', '--cover-package=curator', '--cover-erase', '--verbose', ] nose.run_exit( argv=argv, defaultTest=abspath(dirname(__file__)) ) if __name__ == '__main__': run_all(sys.argv)
23. Test Path Add integration and unit directories
24. Unit test from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionClose(TestCase): def test_init_raise(self): self.assertRaises(TypeError, curator.Close, 'invalid') def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '5.0.0'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) co = curator.Close(ilo) self.assertEqual(ilo, co.index_list) self.assertEqual(client, co.client)
25. Integration test import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200
26. Integration test class TestCLIClose(CuratorTestCase): def test_close_opened(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.optionless_proto.format('close')) self.create_index('my_index') self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) )
27. Integration test self.assertEquals( 'close', self.client.cluster.state( index='my_index', metric='metadata', )['metadata']['indices']['my_index']['state'] ) self.assertNotEqual( 'close', self.client.cluster.state( index='dummy', metric='metadata', )['metadata']['indices']['dummy']['state'] )
28. $ /usr/bin/python setup.py test running test running egg_info writing requirements to elasticsearch_curator.egg-info/requires.txt writing elasticsearch_curator.egg-info/PKG-INFO writing top-level names to elasticsearch_curator.egg-info/top_level.txt writing dependency_links to elasticsearch_curator.egg-info/dependency_links.txt writing entry points to elasticsearch_curator.egg-info/entry_points.txt reading manifest file 'elasticsearch_curator.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'elasticsearch_curator.egg-info/SOURCES.txt' running build_ext test_add_and_remove (test.integration.test_alias.TestCLIAlias) ... ok test_add_only (test.integration.test_alias.TestCLIAlias) ... ok test_add_only_skip_closed (test.integration.test_alias.TestCLIAlias) ... ok test_add_only_with_extra_settings (test.integration.test_alias.TestCLIAlias) ... ok test_add_with_empty_list (test.integration.test_alias.TestCLIAlias) ... ok test_add_with_empty_remove (test.integration.test_alias.TestCLIAlias) ... ok test_alias_remove_only (test.integration.test_alias.TestCLIAlias) ... ok test_extra_options (test.integration.test_alias.TestCLIAlias) ... ok test_no_add_remove (test.integration.test_alias.TestCLIAlias) ... ok test_no_alias (test.integration.test_alias.TestCLIAlias) ... ok ...
29. ---------------------------------------------------------------------- XML: /Users/buh/Dropbox/Development/curator/nosetests.xml Name Stmts Miss Cover Missing ------------------------------------------------------------------- curator.py 10 0 100% curator/_version.py 1 0 100% curator/actions.py 641 10 98% 1213-1220, 1269, 1289-1292 curator/cli.py 111 0 100% curator/config_utils.py 34 5 85% 24-29 ... SEVERAL MORE LINES HERE ... curator/repomgrcli.py 78 1 99% 16 curator/snapshotlist.py 186 0 100% curator/utils.py 640 14 98% 728-730, 739-762 curator/validators.py 1 0 100% curator/validators/actions.py 19 0 100% curator/validators/config_file.py 4 0 100% curator/validators/filters.py 32 0 100% curator/validators/options.py 13 0 100% curator/validators/schemacheck.py 41 0 100% ------------------------------------------------------------------- TOTAL 2491 30 99% ---------------------------------------------------------------------- Ran 563 tests in 125.423s FAILED (SKIP=4, failures=1) Shutting down....
30. The broken test def test_filter_by_array_of_aliases(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.filter_by_alias.format(' [ testalias, foo ]', False)) self.create_index('my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(1, len(curator.get_indices(self.client)))
31. ====================================================================== FAIL: test_filter_by_array_of_aliases (test.integration.test_integrations.TestFilters) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/buh/Dropbox/Development/curator/test/integration/ test_integrations.py", line 55, in test_filter_by_array_of_aliases self.assertEquals(1, len(curator.get_indices(self.client))) AssertionError: 1 != 2 -------------------- >> begin captured logging << -------------------- ... DEBUG curator.indexlist iterate_filters:904 Parsed filter args: {'exclude': False, 'filtertype': 'alias', 'aliases': ['testalias', 'foo']} DEBUG curator.utils iterate_filters:913 Filter args: {'exclude': False, 'aliases': ['testalias', 'foo']} DEBUG curator.utils iterate_filters:914 Pre-instance: [u'dummy', u'my_index'] DEBUG curator.indexlist filter_by_alias:694 Filtering indices matching aliases: "['testalias', 'foo']" DEBUG curator.indexlist empty_list_check:183 Checking for empty list DEBUG curator.indexlist __not_actionable:39 Index dummy is not actionable, removing from list. DEBUG curator.indexlist __excludify:58 Removed from actionable list: dummy is not associated with aliases: ['testalias', 'foo'] DEBUG curator.indexlist __not_actionable:39 Index my_index is not actionable, removing from list. DEBUG curator.indexlist __excludify:58 Removed from actionable list: my_index is not associated with aliases: ['testalias', 'foo']
32. ====================================================================== FAIL: test_filter_by_array_of_aliases (test.integration.test_integrations.TestFilters) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/buh/Dropbox/Development/curator/test/integration/ test_integrations.py", line 55, in test_filter_by_array_of_aliases self.assertEquals(1, len(curator.get_indices(self.client))) AssertionError: 1 != 2 -------------------- >> begin captured logging << -------------------- ... DEBUG curator.indexlist iterate_filters:904 Parsed filter args: {'exclude': False, 'filtertype': 'alias', 'aliases': ['testalias', 'foo']} DEBUG curator.utils iterate_filters:913 Filter args: {'exclude': False, 'aliases': ['testalias', 'foo']} DEBUG curator.utils iterate_filters:914 Pre-instance: [u'dummy', u'my_index'] DEBUG curator.indexlist filter_by_alias:694 Filtering indices matching aliases: "['testalias', 'foo']" DEBUG curator.indexlist empty_list_check:183 Checking for empty list DEBUG curator.indexlist __not_actionable:39 Index dummy is not actionable, removing from list. DEBUG curator.indexlist __excludify:58 Removed from actionable list: dummy is not associated with aliases: ['testalias', 'foo'] DEBUG curator.indexlist __not_actionable:39 Index my_index is not actionable, removing from list. DEBUG curator.indexlist __excludify:58 Removed from actionable list: my_index is not associated with aliases: ['testalias', 'foo']
33. elasticsearch: DEBUG: < {"error":"alias [foo] missing","status":404,"dummy": {"aliases":{"testalias":{}}}} curator.indexlist: DEBUG: Index dummy is not actionable, removing from list. curator.indexlist: DEBUG: Removed from actionable list: dummy is not associated with aliases: ['testalias', 'foo'] curator.indexlist: DEBUG: Index my_index is not actionable, removing from list. curator.indexlist: DEBUG: Removed from actionable list: my_index is not associated with aliases: ['testalias', 'foo']
36. Travis CI on GitHub language: python python: - "2.7" - "3.4" - "3.5" - "3.6" env: - ES_VERSION=5.0.2 - ES_VERSION=5.1.2 - ES_VERSION=5.2.2 - ES_VERSION=5.3.3 - ES_VERSION=5.4.1 os: linux cache: pip: true
37. Travis CI on GitHub jdk: - oraclejdk8 install: - pip install -r requirements.txt - pip install . script: - sudo apt-get update && sudo apt-get install oracle-java8-installer - java -version - sudo update-alternatives --set java /usr/lib/jvm/java-8-oracle/jre/bin/java - java -version - ./travis-run.sh
38. #!/bin/bash set -ex expected_skips=1 setup_es() { # code to download, uncompress and configure elasticsearch } start_es() { # code to start elasticsearch } setup_es https://ES.URL/elasticsearch-$ES_VERSION.tar.gz java_home='/usr/lib/jvm/java-8-oracle' start_es $java_home "-d -Epath.conf=$LC" 9200 "local" python setup.py test result=$(head -1 nosetests.xml | awk '{print $6 " " $7 " " $8}' | awk -F\> '{print $1}' | tr -d '"') echo "Result = $result" errors=$(echo $result | awk '{print $1}' | awk -F\= '{print $2}') failures=$(echo $result | awk '{print $2}' | awk -F\= '{print $2}') skips=$(echo $result | awk '{print $3}' | awk -F\= '{print $2}') if [[ $errors -gt 0 ]]; then exit 1 elif [[ $failures -gt 0 ]]; then exit 1 elif [[ $skips -gt $expected_skips ]]; then exit 1 fi
40. Documentation You do want others to be able to use this, don't you?
41. Why publish docs? • Users will want to know how to use your API • It can help answer direct questions. • Ah! That's at <link> in the manual... • Clearly define acceptable argument values
42. rST inline docs class Close(object): def __init__(self, ilo, delete_aliases=False): """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg delete_aliases: If `True`, will delete any associated aliases before closing indices. :type delete_aliases: bool """ verify_index_list(ilo) #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo #: Instance variable. #: Internal reference to `delete_aliases` self.delete_aliases = delete_aliases #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client
44. """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg request_body: The body to send to :class:`elasticsearch.Indices.Reindex`, which must be complete and usable, as Curator will do no vetting of the request_body. If it fails to function, Curator will return an exception. :arg refresh: Whether to refresh the entire target index after the operation is complete. (default: `True`) :type refresh: bool :arg requests_per_second: The throttle to set on this request in sub-requests per second. ``-1`` means set no throttle as does ``unlimited`` which is the only non-float this accepts. (default: ``-1``) :arg wait_for_active_shards: Sets the number of shard copies that must be active before proceeding with the reindex operation. (default: ``1``) means the primary shard only. Set to ``all`` for all shard copies, otherwise set to any non-negative value less than or equal to the total number of copies for the shard (number of replicas + 1) :arg wait_for_completion: Wait (or not) for the operation to complete before returning. (default: `True`) :type wait_for_completion: bool :arg wait_interval: How long in seconds to wait between checks for completion. :arg max_wait: Maximum number of seconds to `wait_for_completion` :arg remote_url_prefix: `Optional` url prefix, if needed to reach the Elasticsearch API (i.e., it's not at the root level) :type remote_url_prefix: str
46. import sys, os, re def fread(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() def get_version(): VERSIONFILE="../curator/_version.py" verstrline = fread(VERSIONFILE).strip() vsre = r"^__version__ = ['\"]([^'\"]*)['\"]" mo = re.search(vsre, verstrline, re.M) if mo: VERSION = mo.group(1) else: raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE,)) build_number = os.environ.get('CURATOR_BUILD_NUMBER', None) if build_number: return VERSION + "b{}".format(build_number) return VERSION sys.path.insert(0, os.path.abspath('../')) extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx'] # continued on next slide...
47. intersphinx_mapping = { 'python': ('https://docs.python.org/3.6', None), 'elasticsearch': ('http://elasticsearch-py.readthedocs.io/en/5.4.0', None), autoclass_content = "both" templates_path = ['_templates'] source_suffix = '.rst' master_doc = 'index' project = u'Elasticsearch Curator' copyright = u'2011-2017, Elasticsearch' release = get_version() version = '.'.join(release.split('.')[:2]) exclude_patterns = ['_build'] pygments_style = 'sphinx' html_theme = 'default' html_static_path = ['_static'] htmlhelp_basename = 'Curatordoc' latex_elements = { latex_documents = [ ('index', 'ES_Curator.tex', u'Elasticsearch Curator Documentation', u'Aaron Mildenstein', 'manual'), man_pages = [ ('index', 'curator', u'Elasticsearch Curator Documentation', [u'Aaron Mildenstein'], 1) texinfo_documents = [ ('index', 'Curator', u'Elasticsearch Curator Documentation', u'Aaron Mildenstein', 'Curator', 'One line description of project.', 'Miscellaneous'),
48. rST doc definition Contents -------- .. toctree:: :maxdepth: 2 objectclasses actionclasses filters utilities examples Changelog
49. L o g o It's magic... Automatically break out two-deep submenus.
50. .. _objectclasses: Object Classes ============== * `IndexList`_ * `SnapshotList`_ IndexList --------- .. autoclass:: curator.indexlist.IndexList :members: SnapshotList ------------ .. autoclass:: curator.snapshotlist.SnapshotList :members:
52. Get started with Sphinx $ cd /path/to/my_project/doc $ sphinx-quickstart Welcome to the Sphinx 1.6.3 quickstart utility. Please enter values for the following settings (just press Enter to accept a default value, if one is given in brackets). Enter the root path for documentation. > Root path for the documentation [.]:
53. The project name will occur in several places in the built documentation. > Project name: Curator > Author name(s): Aaron Mildenstein Sphinx has the notion of a "version" and a "release" for the software. Each version can have multiple releases. For example, for Python the version is something like 2.5 or 3.0, while the release is something like 2.5.1 or 3.0a1. If you don't need this dual structure, just set both to the same value. > Project version []: 5.1 > Project release [5.1]: 5.1.1 Please indicate if you want to use one of the following Sphinx extensions: > autodoc: automatically insert docstrings from modules (y/n) [n]: y ... > intersphinx: link between Sphinx documentation of different projects (y/n) [n]: y > todo: write "todo" entries that can be shown or hidden on build (y/n) [n]: n A Makefile and a Windows command file can be generated for you so that you only have to run e.g. `make html' instead of invoking sphinx-build directly. > Create Makefile? (y/n) [y]: > Create Windows command file? (y/n) [y]: n
54. Creating file ./conf.py. Creating file ./index.rst. Creating file ./Makefile. Finished: An initial directory structure has been created. You should now populate your master file ./index.rst and create other documentation source files. Use the Makefile to build the docs, like so: make builder where "builder" is one of the supported builders, e.g. html, latex or linkcheck. $ ls -l total 32 -rw-r--r--@ 1 buh staff 607 Jul 7 18:19 Makefile drwxr-xr-x@ 2 buh staff 68 Jul 7 18:19 _build drwxr-xr-x@ 2 buh staff 68 Jul 7 18:19 _static drwxr-xr-x@ 2 buh staff 68 Jul 7 18:19 _templates -rw-r--r--@ 1 buh staff 5340 Jul 7 18:19 conf.py -rw-r--r--@ 1 buh staff 437 Jul 7 18:19 index.rst
55. edit conf.py # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # import os import sys # sys.path.insert(0, os.path.abspath('.')) sys.path.insert(0, os.path.abspath('../'))
56. edit conf.py # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # # html_theme = 'alabaster' html_theme = 'classic'
57. edit conf.py # Custom sidebar templates, must be a dictionary that maps document names # to template names. # # This is required for the alabaster theme # refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars #html_sidebars = { # '**': [ # 'about.html', # 'navigation.html', # 'relations.html', # needs 'show_related': True theme option to display # 'searchbox.html', # 'donate.html', # ] #}
58. edit conf.py # Example configuration for intersphinx: refer to the Python standard library. intersphinx_mapping = { 'python': ('https://docs.python.org/', None), 'elasticsearch': ('http://elasticsearch-py.readthedocs.io/en/5.4.0', None), }
59. build it! $ make html Running Sphinx v1.6.3 loading pickled environment... not yet created loading intersphinx inventory from https://docs.python.org/objects.inv... intersphinx inventory has moved: https://docs.python.org/objects.inv -> https:// docs.python.org/2/objects.inv loading intersphinx inventory from https://elasticsearch-py.readthedocs.io/en/ 5.4.0/objects.inv... ... SEVERAL MORE LINES HERE build succeeded. Build finished. The HTML pages are in _build/html. $
60. Publish it! • https://readthedocs.org • https://docs.readthedocs.io/en/latest/getting_started.html • Use web hooks to automatically publish new versions!
62. Multi-version You can host multiple versions of your API documentation here. Consider donating to ReadTheDocs if you find this service useful!
63. Publishing Make that code available!
64. L o g o PyPI • It's dead simple • for you • and for your users • pip install -U my_project
65. Build your dist $ python setup.py sdist $ ls dist/ my_project-X.Y.Z.tar.gz $
66. Get twine $ pip install -U twine
67. Use twine to register $ twine register usage: twine register [-h] [-r REPOSITORY] [--repository-url REPOSITORY_URL] [-u USERNAME] [-p PASSWORD] [-c COMMENT] [--config-file CONFIG_FILE] [--cert path] [--client-cert path] package $ twine register --repository-url=https://pypi.python.org/pypi -u USER -p PASS dist/my_project-X.Y.Z.tar.gz
68. Use twine to upload $ twine upload usage: twine upload [-h] [-r REPOSITORY] [--repository-url REPOSITORY_URL] [-s] [--sign-with SIGN_WITH] [-i IDENTITY] [-u USERNAME] [-p PASSWORD] [-c COMMENT] [--config-file CONFIG_FILE] [--skip-existing] [--cert path] [--client-cert path] dist [dist ...] $ twine upload --repository-url=https://pypi.python.org/pypi -u USER -p PASS dist/ my_project-X.Y.Z.tar.gz
70. Consider RPM/DEB • Pro: • Easier upgrades for users • Wider distribution • Con: • A lot more complexity, potentially
71. Important Considerations Points to ponder as you contemplate a future with a Wrapper API
72. Who will this help? • If this is definable, it will make some of your decisions easier • It's okay if the answer is, "only me." • It's also okay if the answer is, "I don't know."
73. Am I even doing this right? • Worried that you might get it wrong? Don't be! • Tests and test-driven development help a lot. • Coding skills improve over time. • Fix things and improve as you learn. • It's okay to iterate.
74. Impostor? No sir! • Don't get stuck! • You can do it! • Asking for help doesn't make you weak, or a bad coder. • You don't have to know it all • Learn from your mistakes, and allow others to help.
75. That's All Folks!