StrongSteam at BrightonPy by Kyran Dale

From: http://brightonpy.org/meetings/2011-12-13/

Transcript

1. Building Strongsteam  What it looks like  How Python enabled it  Brain damage so you don't have to
2. Cafe-house sketches
3. Basic Strongsteam architecture
4. A module for (almost) everything
5. Choose your modules gentlemen  Liberal selection from standard library  Flask web-framework  MongoDB NoSQL database  Requests for building web-client  Concurrent Log-handler  Nosetests – nicer Unit-testing  Numpy all over the place
6. The death of print import logging import sys, os from cloghandler import ConcurrentRotatingFileHandler from strongsteam import cfg def get_logger(name='ss', log_filename=cfg.Config.LOG_FILE, debug=True, to_stdout=False): log = logging.getLogger(name) ch = logging.StreamHandler() if os.path.exists(os.path.dirname(log_filename)): fh = logging.handlers.ConcurrentRotatingFileHandler(log_filename, maxBytes=1000000, backupCount=0) else: raise IOError("log directory does not exist (" + os.path.dirname(log_filename) + ")") sys.exit(1) if to_stdout: log.addHandler(ch) log.addHandler(fh) ch_fmt = logging.Formatter("%(levelname)s\t: %(message)s") fh_fmt = logging.Formatter("%(asctime)s %(filename)s %(funcName)s %(lineno)d (%(levelname)s)\t: % (message)s") ch.setFormatter(ch_fmt) fh.setFormatter(fh_fmt) if debug: log.setLevel(logging.DEBUG) else: log.setLevel(logging.INFO) return log # In module.py: from strongsteam.ss_logger import get_logger log = get_logger(__name__) log.debug('Oops … ')
7. A little code... class Process(object): name = 'something_like_surf_or_ocr' [...] available_processes = {} def register_processes(): """This gets available processes, auto-imported by walking the processes package. We walk the user package beforehand to check for newly uploaded processes. """ global available_processes available_processes.clear() walk_packages() for p in Process.__subclasses__(): try: if getattr(p, 'name', None) is not None: available_processes[p.name] = p except TypeError: continue log.debug('Available processes: ' + repr(available_processes)) def walk_packages(pkg=sys.modules[__name__], comp_str='process_'): """Recursively auto-import all modules beginning with the comparison string.""" for importer, name, ispkg in pkgutil.walk_packages(pkg.__path__, prefix=pkg.__name__+'.'): if name.split('.')[-1].startswith(comp_str): __import__(name) log.info('Imported ' + name)