Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ticketea internals

ticketea internals

Talk at PyCON ES 2014 (Spanish)

00927a856336d961bdc7028722fe5897?s=128

Ticketea Engineering

November 09, 2014
Tweet

Transcript

  1. internals @ticketeaEng

  2. ticketea ¿Quiénes somos? @esanchezm @adiazher @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco

    @Ethervoid @imanolcg @loquedigairune @Leidan @RafaRM20
  3. None
  4. ticketea Al Principio… ✤ Primera web, usando .NET ✤ Reescritura

    en PHP y Yii ✤ Reescritura del frontal en PHP (framework propio)
  5. None
  6. ticketea Actualmente… ✤ De 4 personas a 12, en un

    año y medio ✤ Sin perfil de sistemas/dba puro ✤ Sistemas en AWS ✤ Hemos llegado a varios países
  7. Cambio de Estrategia

  8. No ha sido un camino fácil…

  9. odin

  10. ticketea Herramienta de informes ✤ Business Intelligence ✤ Exportación de

    informes (PDF/XLSX) ✤ Compatibilidad de sesiones con PHP
  11. None
  12. ticketea Compatibilidad con lo anterior… class SessionStore(CacheStore): def load(self): pass

    def save(self, must_create=False): pass @property def cache_key(self): pass def decode(self, session_data): pass def encode(self, session_dict): pass
  13. ticketea SessionStore -> CacheStore def load(self): try: session_data = self.decode(

    self._cache.get(self.cache_key, None) ) except Exception: session_data = None if session_data is not None and 'user' in session_data: try: session_data[SESSION_KEY] = session_data['user']['id'] except KeyError: pass return session_data self.create() return {}
  14. ticketea SessionStore -> CacheStore def save(self, must_create=False): if must_create: func

    = self._cache.add else: func = self._cache.set result = func( self.cache_key, self.encode(self._get_session(no_load=must_create)), self.get_expiry_age() ) if must_create and not result: raise CreateError
  15. ticketea SessionStore -> CacheStore @property def cache_key(self): return "memc.sess.key." +

    self._get_or_create_session_key() def decode(self, session_data): if session_data is not None: return PHPUnserialize().session_decode(session_data) return session_data def encode(self, session_dict): return PHPSerialize().session_encode(session_dict)
  16. ticketea Usando el admin sin modelos… # __init__.py from .

    import admin_models if 'django.contrib.admin.models' in sys.modules: del sys.modules['django.contrib.admin.models'] sys.modules['django.contrib.admin.models'] = admin_models from django.contrib.admin.sites import AdminSite AdminSite.check_dependencies = nothing from django.core.management.base import BaseCommand BaseCommand.validate = lambda x, *args, **kwargs: None
  17. pylongtable

  18. None
  19. ticketea

  20. Python Era Begins

  21. forseti

  22. ticketea ¿Qué es forseti? ✤ Gestionar grupos de auto-escalado ✤

    Generación de AMIs ✤ Despliegue de nuevo código ✤ Gestión de alarmas
  23. ticketea ¿Qué es forseti? FORSETI BOTO BOTO-CORE EC2AutoScaleConfig ELBalancer CloudWatchMetricAlarm

    EC2AutoScaleGroup EC2AutoScalePolicy GoldenEC2Image EC2Instance
  24. ticketea RELEASE THE KRAKEN

  25. frontend

  26. ticketea Arquitectura de sistemas API Load Balancer API API API

    { Frontend Frontend Frontend Frontend Load Balancer DB { Usuario
  27. ticketea Arquitectura del frontend supervisor chaussette

  28. ticketea Nuestro “django” Modelos Contrib auth ModelForms DB Templates Sesiones

    CBVs NO SÍ
  29. ticketea Resultado ✤ La planificación fue un desastre: 2x tiempo.

    ✤ Mejor logging y reporting de errores. ✤ Tiempo de respuesta reducido a la mitad. ✤ Cobertura de tests > 85%
  30. ticketea django-configurations settings ├── base.py ├── bundles │ ├── crispy.py

    │ ├── gunicorn.py │ ├── pipeline.py │ ├── raven.py │ ├── statsd.py │ └── tkt.py ├── development.py ├── production.py └── testing.py
  31. ticketea django-configurations class Base(CrispyForms, Settings): DJANGO_APPS = [] BUNDLE_APPS =

    [] TKT_APPS = [] @property def INSTALLED_APPS(self): return self.DJANGO_APPS + self.BUNDLE_APPS + self.TKT_APPS @property def DATABASES(self): return { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'db_name', 'USER': 'db_user', 'PASSWORD': self.DB_PASSWORD, 'HOST': 'localhost', } }
  32. ticketea django-configurations class Production(Gunicorn, Base): DB_PASSWORD = values.Value() class Development(Base):

    DB_PASSWORD = 'pass'
  33. ticketea django-configurations class CrispyForms(object): def __init__(self): super(CrispyForms, self).__init__() self.BUNDLE_APPS +=

    ( 'crispy_forms', )
  34. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)

  35. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)

  36. ticketea Uso del api client api.user(6077).event(1).update({ 'name': name, 'description': description

    })
  37. ticketea Creando un endpoint… class User(CRUDEndpoint): def __init__(self, pk): super(User,

    self).__init__(pk) self.add_endpoint(Event) def activate(self, activation_key, email): return self._http_client.request.post( '%s/activate' % self.url, data={ 'activation_key': activation_key, 'email': email } )
  38. ticketea Internals del api-client class RestCRUDEndpoint(object): . . . def

    add_endpoint(self, klass): url_endpoint = klass.__name__.lower() python_name = from_camel_to_snake(url_endpoint) list_name = plural_name(python_name) create_name = 'create_' + python_name klass.endpoint = '%s/%s' % (self.url, url_endpoint) klass._http_client = self._http_client setattr(self, python_name, klass) setattr( self, create_name, self._create_object(url_endpoint) )
  39. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)

    TicketeaApiMiddleware TicketeaApiException
  40. ticketea Inyectando el api-client en el request class TicketeaApiMiddleware(object): def

    process_request(self, request): if not hasattr(request, 'user'): raise ImproperlyConfigured('error') if isinstance(request.user, AnonymousUser): self._login_as_guest(request) request.api = create_api_client(...) def process_response(self, request, response): # Check if the token was refreshed return response
  41. ticketea Manejando errores del API class TicketeaApiExceptions(object): def process_exception(self, request,

    exception): if isinstance(exception, ApiException): if exception.code == self.REQUEST_TOKEN_EXPIRED: return self._manage_expired_token(request) elif isinstance(exception, ForbiddenException): raise PermissionDenied elif isinstance(exception, NotFoundException): raise Http404
  42. ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)

    ApiDispatchMixin TktTemplateView
  43. ticketea Mixins class ApiDispatchMixin(object): def dispatch(self, *args, **kwargs): if hasattr(self,

    'get_api_data'): result = self.get_api_data() if isinstance(result, HttpResponse): return result return super(ApiDispatchMixin, self).dispatch( *args, **kwargs )
  44. ticketea Mixins # FAIL class AccessKwargsMixin(object): def get_context_data(self, **kwargs): kw

    = super(AccessKwargsMixin, self) \ .get_context_data(**kwargs) kw['url_kwargs'] = self.kwargs return kw
  45. ticketea Nuestras vistas genéricas class TktView( LoginRequiredMixin, UserInjectorMixin, RoleRequiredMixin, ApiDispatchMixin,

    View ) class TktTemplateView( LoginRequiredMixin, UserInjectorMixin, RoleRequiredMixin, ApiDispatchMixin, TemplateView )
  46. ticketea Clase base de nuestros FormView class TktFormView(FormView): def post(self,

    request, *args, **kwargs): form_class = self.get_form_class() form = self.get_form(form_class) if form.is_valid(): validate_and_commit = getattr(self, 'validate_and_commit', lambda x: True) validate_result = validate_and_commit(form) if validate_result is None: raise ValueError('validate_and_commit must return something') if isinstance(validate_result, HttpResponse): return validate_result elif validate_result: return self.form_valid(form) else: return self.form_invalid(form) else: messages.error(request, _('there has been an error')) return self.form_invalid(form)
  47. ticketea Clase base de nuestros Form @decorator def handle_api_errors(form_save_method, self,

    *args, **kwargs): try: save_method = form_save_method(self, *args, **kwargs) return save_method if save_method is not None else True except ApiException as e: self.fill_form_errors(e.attrs) return False class TktForm(forms.Form): api_field_names = {} def fill_form_errors(self, errors): for field, errno in errors.items(): real_field_name = self.api_field_names.get(field, field) error_description = API_ERRORS_NAMES.get(errno) self._errors[real_field_name] = self.error_class( [error_description] )
  48. ticketea Ejemplo de formulario class AccountInfoForm(TktForm): email = forms.EmailField() def

    __init__(self, *args, **kwargs): super(AccountInfoForm, self).__init__(*args, **kwargs) self.helper = AccountInfoHelper() @handle_api_errors def save(self, user): user.update(**self.cleaned_data)
  49. ticketea Ejemplo de vista class AccountInfoView(TktFormView): form_class = AccountInfoForm template_name

    = 'users_config/users/profile.html' min_role = 'organizer' def get_api_data(self): self.user = self.request.api.user(self.kwargs['id']) def validate_and_commit(self, form): committed = form.save(self.user) if committed: messages.success(self.request, _('OK')) else: messages.error(self.request, _('KO')) return committed def get_success_url(self): return reverse('profile_url')
  50. ticketea third-party apps ✤ django-crispy-forms ✤ django-braces ✤ django-configurations ✤

    django-pipeline ✤ django-secure ✤ django-debug-toolbar ✤ werkzeug ✤ HTTPretty
  51. heracles

  52. ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery {

  53. ticketea Tarea base class BaseTask(Task): abstract = True ignore_result =

    True max_retries = 5 def retry_time(self, retries): RETRIES = [10, 15, 20, 30] try: return RETRIES[retries] except IndexError: return RETRIES[-1] def retry(self, *args, **kwargs): retries = self.request.retries + 1 max_retries = kwargs.get('max_retries', self.max_retries) if retries > max_retries * 0.3: logger.warning(...) kwargs['countdown'] = self.retry_time(retries - 1) return super(BaseTask, self).retry(*args, **kwargs)
  54. ticketea Tarea base class BaseTask(Task): def on_success(self, *args, **kwargs): statsd.incr('tasks.%s'

    % self.name) def on_failure(self, exc, task_id, args, kwargs, einfo): logger.warning(...) def __call__(self, *args, **kwargs): with transaction.atomic(): try: return super(BaseTask, self).__call__( *args, **kwargs ) except Exception as exc: if isinstance(exc, exceptions.Retry): raise exc else: logger.exception('Unhandled exception') self.retry(exc=exc)
  55. no hay solución perfecta

  56. thor

  57. ticketea Objetivos ✤ Pasar a sistemas distribuídos ✤ Permitir el

    funcionamiento degradado del sistema ✤ Poder desarrollar cada proyecto a una velocidad distinta ✤ Poder ajustar la infraestructura usada de cada proyecto ✤ Reducir puntos únicos de fallo
  58. ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery { THOR

    DBLS Tasks
  59. ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery { THOR

    DBLS Tasks Views X
  60. Más importante que tener un equipo A…

  61. … es tener un plan B

  62. graphing && logging

  63. ticketea Herramientas usadas ✤ sentry ✤ graphite ✤ grafana ✤

    rsyslog ✤ logstash ✤ cabot
  64. ticketea Sabina, Noviembre 2014

  65. ticketea PREGUNTAS

  66. @ticketeaEng GRACIAS