Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ticketea internals

ticketea internals

Talk at PyCON ES 2014 (Spanish)

Ticketea Engineering

November 09, 2014
Tweet

More Decks by Ticketea Engineering

Other Decks in Programming

Transcript

  1. internals
    @ticketeaEng

    View full-size slide

  2. ticketea
    ¿Quiénes somos?
    @esanchezm @adiazher @JavierHdez3 @maraujop
    @igalarzab @sullymorland @patoroco @Ethervoid
    @imanolcg @loquedigairune @Leidan
    @RafaRM20

    View full-size slide

  3. ticketea
    Al Principio…
    ✤ Primera web, usando .NET
    ✤ Reescritura en PHP y Yii
    ✤ Reescritura del frontal en PHP (framework propio)

    View full-size slide

  4. ticketea
    Actualmente…
    ✤ De 4 personas a 12, en un año y medio
    ✤ Sin perfil de sistemas/dba puro
    ✤ Sistemas en AWS
    ✤ Hemos llegado a varios países

    View full-size slide

  5. Cambio de
    Estrategia

    View full-size slide

  6. No ha sido un camino fácil…

    View full-size slide

  7. ticketea
    Herramienta de informes
    ✤ Business Intelligence
    ✤ Exportación de informes (PDF/XLSX)
    ✤ Compatibilidad de sesiones con PHP

    View full-size slide

  8. ticketea
    Compatibilidad con lo anterior…
    class SessionStore(CacheStore):
    def load(self):
    pass
    def save(self, must_create=False):
    pass
    @property
    def cache_key(self):
    pass
    def decode(self, session_data):
    pass
    def encode(self, session_dict):
    pass

    View full-size slide

  9. ticketea
    SessionStore -> CacheStore
    def load(self):
    try:
    session_data = self.decode(
    self._cache.get(self.cache_key, None)
    )
    except Exception:
    session_data = None
    if session_data is not None and 'user' in session_data:
    try:
    session_data[SESSION_KEY] = session_data['user']['id']
    except KeyError:
    pass
    return session_data
    self.create()
    return {}

    View full-size slide

  10. ticketea
    SessionStore -> CacheStore
    def save(self, must_create=False):
    if must_create:
    func = self._cache.add
    else:
    func = self._cache.set
    result = func(
    self.cache_key,
    self.encode(self._get_session(no_load=must_create)),
    self.get_expiry_age()
    )
    if must_create and not result:
    raise CreateError

    View full-size slide

  11. ticketea
    SessionStore -> CacheStore
    @property
    def cache_key(self):
    return "memc.sess.key." + self._get_or_create_session_key()
    def decode(self, session_data):
    if session_data is not None:
    return PHPUnserialize().session_decode(session_data)
    return session_data
    def encode(self, session_dict):
    return PHPSerialize().session_encode(session_dict)

    View full-size slide

  12. ticketea
    Usando el admin sin modelos…
    # __init__.py
    from . import admin_models
    if 'django.contrib.admin.models' in sys.modules:
    del sys.modules['django.contrib.admin.models']
    sys.modules['django.contrib.admin.models'] = admin_models
    from django.contrib.admin.sites import AdminSite
    AdminSite.check_dependencies = nothing
    from django.core.management.base import BaseCommand
    BaseCommand.validate = lambda x, *args, **kwargs: None

    View full-size slide

  13. Python Era Begins

    View full-size slide

  14. ticketea
    ¿Qué es forseti?
    ✤ Gestionar grupos de auto-escalado
    ✤ Generación de AMIs
    ✤ Despliegue de nuevo código
    ✤ Gestión de alarmas

    View full-size slide

  15. ticketea
    ¿Qué es forseti?
    FORSETI
    BOTO
    BOTO-CORE
    EC2AutoScaleConfig
    ELBalancer
    CloudWatchMetricAlarm
    EC2AutoScaleGroup
    EC2AutoScalePolicy
    GoldenEC2Image
    EC2Instance

    View full-size slide

  16. ticketea
    RELEASE THE KRAKEN

    View full-size slide

  17. ticketea
    Arquitectura de sistemas
    API
    Load
    Balancer
    API
    API
    API
    {
    Frontend
    Frontend
    Frontend
    Frontend
    Load
    Balancer
    DB
    {
    Usuario

    View full-size slide

  18. ticketea
    Arquitectura del frontend
    supervisor
    chaussette

    View full-size slide

  19. ticketea
    Nuestro “django”
    Modelos
    Contrib auth
    ModelForms
    DB
    Templates
    Sesiones
    CBVs
    NO SÍ

    View full-size slide

  20. ticketea
    Resultado
    ✤ La planificación fue un desastre: 2x tiempo.
    ✤ Mejor logging y reporting de errores.
    ✤ Tiempo de respuesta reducido a la mitad.
    ✤ Cobertura de tests > 85%

    View full-size slide

  21. ticketea
    django-configurations
    settings
    ├── base.py
    ├── bundles
    │ ├── crispy.py
    │ ├── gunicorn.py
    │ ├── pipeline.py
    │ ├── raven.py
    │ ├── statsd.py
    │ └── tkt.py
    ├── development.py
    ├── production.py
    └── testing.py

    View full-size slide

  22. ticketea
    django-configurations
    class Base(CrispyForms, Settings):
    DJANGO_APPS = []
    BUNDLE_APPS = []
    TKT_APPS = []
    @property
    def INSTALLED_APPS(self):
    return self.DJANGO_APPS + self.BUNDLE_APPS + self.TKT_APPS
    @property
    def DATABASES(self):
    return {
    'default': {
    'ENGINE': 'django.db.backends.mysql',
    'NAME': 'db_name',
    'USER': 'db_user',
    'PASSWORD': self.DB_PASSWORD,
    'HOST': 'localhost',
    }
    }

    View full-size slide

  23. ticketea
    django-configurations
    class Production(Gunicorn, Base):
    DB_PASSWORD = values.Value()
    class Development(Base):
    DB_PASSWORD = 'pass'

    View full-size slide

  24. ticketea
    django-configurations
    class CrispyForms(object):
    def __init__(self):
    super(CrispyForms, self).__init__()
    self.BUNDLE_APPS += (
    'crispy_forms',
    )

    View full-size slide

  25. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)

    View full-size slide

  26. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)

    View full-size slide

  27. ticketea
    Uso del api client
    api.user(6077).event(1).update({
    'name': name,
    'description': description
    })

    View full-size slide

  28. ticketea
    Creando un endpoint…
    class User(CRUDEndpoint):
    def __init__(self, pk):
    super(User, self).__init__(pk)
    self.add_endpoint(Event)
    def activate(self, activation_key, email):
    return self._http_client.request.post(
    '%s/activate' % self.url,
    data={
    'activation_key': activation_key,
    'email': email
    }
    )

    View full-size slide

  29. ticketea
    Internals del api-client
    class RestCRUDEndpoint(object):
    . . .
    def add_endpoint(self, klass):
    url_endpoint = klass.__name__.lower()
    python_name = from_camel_to_snake(url_endpoint)
    list_name = plural_name(python_name)
    create_name = 'create_' + python_name
    klass.endpoint = '%s/%s' % (self.url, url_endpoint)
    klass._http_client = self._http_client
    setattr(self, python_name, klass)
    setattr(
    self,
    create_name,
    self._create_object(url_endpoint)
    )

    View full-size slide

  30. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)
    TicketeaApiMiddleware
    TicketeaApiException

    View full-size slide

  31. ticketea
    Inyectando el api-client en el request
    class TicketeaApiMiddleware(object):
    def process_request(self, request):
    if not hasattr(request, 'user'):
    raise ImproperlyConfigured('error')
    if isinstance(request.user, AnonymousUser):
    self._login_as_guest(request)
    request.api = create_api_client(...)
    def process_response(self, request, response):
    # Check if the token was refreshed
    return response

    View full-size slide

  32. ticketea
    Manejando errores del API
    class TicketeaApiExceptions(object):
    def process_exception(self, request, exception):
    if isinstance(exception, ApiException):
    if exception.code == self.REQUEST_TOKEN_EXPIRED:
    return self._manage_expired_token(request)
    elif isinstance(exception, ForbiddenException):
    raise PermissionDenied
    elif isinstance(exception, NotFoundException):
    raise Http404

    View full-size slide

  33. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)
    ApiDispatchMixin
    TktTemplateView

    View full-size slide

  34. ticketea
    Mixins
    class ApiDispatchMixin(object):
    def dispatch(self, *args, **kwargs):
    if hasattr(self, 'get_api_data'):
    result = self.get_api_data()
    if isinstance(result, HttpResponse):
    return result
    return super(ApiDispatchMixin, self).dispatch(
    *args,
    **kwargs
    )

    View full-size slide

  35. ticketea
    Mixins
    # FAIL
    class AccessKwargsMixin(object):
    def get_context_data(self, **kwargs):
    kw = super(AccessKwargsMixin, self) \
    .get_context_data(**kwargs)
    kw['url_kwargs'] = self.kwargs
    return kw

    View full-size slide

  36. ticketea
    Nuestras vistas genéricas
    class TktView(
    LoginRequiredMixin, UserInjectorMixin,
    RoleRequiredMixin, ApiDispatchMixin, View
    )
    class TktTemplateView(
    LoginRequiredMixin, UserInjectorMixin,
    RoleRequiredMixin, ApiDispatchMixin,
    TemplateView
    )

    View full-size slide

  37. ticketea
    Clase base de nuestros FormView
    class TktFormView(FormView):
    def post(self, request, *args, **kwargs):
    form_class = self.get_form_class()
    form = self.get_form(form_class)
    if form.is_valid():
    validate_and_commit = getattr(self, 'validate_and_commit', lambda x: True)
    validate_result = validate_and_commit(form)
    if validate_result is None:
    raise ValueError('validate_and_commit must return something')
    if isinstance(validate_result, HttpResponse):
    return validate_result
    elif validate_result:
    return self.form_valid(form)
    else:
    return self.form_invalid(form)
    else:
    messages.error(request, _('there has been an error'))
    return self.form_invalid(form)

    View full-size slide

  38. ticketea
    Clase base de nuestros Form
    @decorator
    def handle_api_errors(form_save_method, self, *args, **kwargs):
    try:
    save_method = form_save_method(self, *args, **kwargs)
    return save_method if save_method is not None else True
    except ApiException as e:
    self.fill_form_errors(e.attrs)
    return False
    class TktForm(forms.Form):
    api_field_names = {}
    def fill_form_errors(self, errors):
    for field, errno in errors.items():
    real_field_name = self.api_field_names.get(field, field)
    error_description = API_ERRORS_NAMES.get(errno)
    self._errors[real_field_name] = self.error_class(
    [error_description]
    )

    View full-size slide

  39. ticketea
    Ejemplo de formulario
    class AccountInfoForm(TktForm):
    email = forms.EmailField()
    def __init__(self, *args, **kwargs):
    super(AccountInfoForm, self).__init__(*args, **kwargs)
    self.helper = AccountInfoHelper()
    @handle_api_errors
    def save(self, user):
    user.update(**self.cleaned_data)

    View full-size slide

  40. ticketea
    Ejemplo de vista
    class AccountInfoView(TktFormView):
    form_class = AccountInfoForm
    template_name = 'users_config/users/profile.html'
    min_role = 'organizer'
    def get_api_data(self):
    self.user = self.request.api.user(self.kwargs['id'])
    def validate_and_commit(self, form):
    committed = form.save(self.user)
    if committed:
    messages.success(self.request, _('OK'))
    else:
    messages.error(self.request, _('KO'))
    return committed
    def get_success_url(self):
    return reverse('profile_url')

    View full-size slide

  41. ticketea
    third-party apps
    ✤ django-crispy-forms
    ✤ django-braces
    ✤ django-configurations
    ✤ django-pipeline
    ✤ django-secure
    ✤ django-debug-toolbar
    ✤ werkzeug
    ✤ HTTPretty

    View full-size slide

  42. ticketea
    Arquitectura
    RabbitMQ
    API
    (PHP)
    Celery
    Celery
    Celery
    {

    View full-size slide

  43. ticketea
    Tarea base
    class BaseTask(Task):
    abstract = True
    ignore_result = True
    max_retries = 5
    def retry_time(self, retries):
    RETRIES = [10, 15, 20, 30]
    try:
    return RETRIES[retries]
    except IndexError:
    return RETRIES[-1]
    def retry(self, *args, **kwargs):
    retries = self.request.retries + 1
    max_retries = kwargs.get('max_retries', self.max_retries)
    if retries > max_retries * 0.3:
    logger.warning(...)
    kwargs['countdown'] = self.retry_time(retries - 1)
    return super(BaseTask, self).retry(*args, **kwargs)

    View full-size slide

  44. ticketea
    Tarea base
    class BaseTask(Task):
    def on_success(self, *args, **kwargs):
    statsd.incr('tasks.%s' % self.name)
    def on_failure(self, exc, task_id, args, kwargs, einfo):
    logger.warning(...)
    def __call__(self, *args, **kwargs):
    with transaction.atomic():
    try:
    return super(BaseTask, self).__call__(
    *args, **kwargs
    )
    except Exception as exc:
    if isinstance(exc, exceptions.Retry):
    raise exc
    else:
    logger.exception('Unhandled exception')
    self.retry(exc=exc)

    View full-size slide

  45. no hay solución perfecta

    View full-size slide

  46. ticketea
    Objetivos
    ✤ Pasar a sistemas distribuídos
    ✤ Permitir el funcionamiento degradado del sistema
    ✤ Poder desarrollar cada proyecto a una velocidad distinta
    ✤ Poder ajustar la infraestructura usada de cada proyecto
    ✤ Reducir puntos únicos de fallo

    View full-size slide

  47. ticketea
    Arquitectura
    RabbitMQ
    API
    (PHP)
    Celery
    Celery
    Celery
    {
    THOR
    DBLS
    Tasks

    View full-size slide

  48. ticketea
    Arquitectura
    RabbitMQ
    API
    (PHP)
    Celery
    Celery
    Celery
    {
    THOR
    DBLS
    Tasks
    Views
    X

    View full-size slide

  49. Más importante que tener un equipo A…

    View full-size slide

  50. … es tener un plan B

    View full-size slide

  51. graphing
    &&
    logging

    View full-size slide

  52. ticketea
    Herramientas usadas
    ✤ sentry
    ✤ graphite
    ✤ grafana
    ✤ rsyslog
    ✤ logstash
    ✤ cabot

    View full-size slide

  53. ticketea
    Sabina, Noviembre 2014

    View full-size slide

  54. ticketea
    PREGUNTAS

    View full-size slide

  55. @ticketeaEng
    GRACIAS

    View full-size slide