Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ticketea internals

ticketea internals

Talk at PyCON ES 2014 (Spanish)

Ticketea Engineering

November 09, 2014
Tweet

More Decks by Ticketea Engineering

Other Decks in Programming

Transcript

  1. internals
    @ticketeaEng

    View Slide

  2. ticketea
    ¿Quiénes somos?
    @esanchezm @adiazher @JavierHdez3 @maraujop
    @igalarzab @sullymorland @patoroco @Ethervoid
    @imanolcg @loquedigairune @Leidan
    @RafaRM20

    View Slide

  3. View Slide

  4. ticketea
    Al Principio…
    ✤ Primera web, usando .NET
    ✤ Reescritura en PHP y Yii
    ✤ Reescritura del frontal en PHP (framework propio)

    View Slide

  5. View Slide

  6. ticketea
    Actualmente…
    ✤ De 4 personas a 12, en un año y medio
    ✤ Sin perfil de sistemas/dba puro
    ✤ Sistemas en AWS
    ✤ Hemos llegado a varios países

    View Slide

  7. Cambio de
    Estrategia

    View Slide

  8. No ha sido un camino fácil…

    View Slide

  9. odin

    View Slide

  10. ticketea
    Herramienta de informes
    ✤ Business Intelligence
    ✤ Exportación de informes (PDF/XLSX)
    ✤ Compatibilidad de sesiones con PHP

    View Slide

  11. View Slide

  12. ticketea
    Compatibilidad con lo anterior…
    class SessionStore(CacheStore):
    def load(self):
    pass
    def save(self, must_create=False):
    pass
    @property
    def cache_key(self):
    pass
    def decode(self, session_data):
    pass
    def encode(self, session_dict):
    pass

    View Slide

  13. ticketea
    SessionStore -> CacheStore
    def load(self):
    try:
    session_data = self.decode(
    self._cache.get(self.cache_key, None)
    )
    except Exception:
    session_data = None
    if session_data is not None and 'user' in session_data:
    try:
    session_data[SESSION_KEY] = session_data['user']['id']
    except KeyError:
    pass
    return session_data
    self.create()
    return {}

    View Slide

  14. ticketea
    SessionStore -> CacheStore
    def save(self, must_create=False):
    if must_create:
    func = self._cache.add
    else:
    func = self._cache.set
    result = func(
    self.cache_key,
    self.encode(self._get_session(no_load=must_create)),
    self.get_expiry_age()
    )
    if must_create and not result:
    raise CreateError

    View Slide

  15. ticketea
    SessionStore -> CacheStore
    @property
    def cache_key(self):
    return "memc.sess.key." + self._get_or_create_session_key()
    def decode(self, session_data):
    if session_data is not None:
    return PHPUnserialize().session_decode(session_data)
    return session_data
    def encode(self, session_dict):
    return PHPSerialize().session_encode(session_dict)

    View Slide

  16. ticketea
    Usando el admin sin modelos…
    # __init__.py
    from . import admin_models
    if 'django.contrib.admin.models' in sys.modules:
    del sys.modules['django.contrib.admin.models']
    sys.modules['django.contrib.admin.models'] = admin_models
    from django.contrib.admin.sites import AdminSite
    AdminSite.check_dependencies = nothing
    from django.core.management.base import BaseCommand
    BaseCommand.validate = lambda x, *args, **kwargs: None

    View Slide

  17. pylongtable

    View Slide

  18. View Slide

  19. ticketea

    View Slide

  20. Python Era Begins

    View Slide

  21. forseti

    View Slide

  22. ticketea
    ¿Qué es forseti?
    ✤ Gestionar grupos de auto-escalado
    ✤ Generación de AMIs
    ✤ Despliegue de nuevo código
    ✤ Gestión de alarmas

    View Slide

  23. ticketea
    ¿Qué es forseti?
    FORSETI
    BOTO
    BOTO-CORE
    EC2AutoScaleConfig
    ELBalancer
    CloudWatchMetricAlarm
    EC2AutoScaleGroup
    EC2AutoScalePolicy
    GoldenEC2Image
    EC2Instance

    View Slide

  24. ticketea
    RELEASE THE KRAKEN

    View Slide

  25. frontend

    View Slide

  26. ticketea
    Arquitectura de sistemas
    API
    Load
    Balancer
    API
    API
    API
    {
    Frontend
    Frontend
    Frontend
    Frontend
    Load
    Balancer
    DB
    {
    Usuario

    View Slide

  27. ticketea
    Arquitectura del frontend
    supervisor
    chaussette

    View Slide

  28. ticketea
    Nuestro “django”
    Modelos
    Contrib auth
    ModelForms
    DB
    Templates
    Sesiones
    CBVs
    NO SÍ

    View Slide

  29. ticketea
    Resultado
    ✤ La planificación fue un desastre: 2x tiempo.
    ✤ Mejor logging y reporting de errores.
    ✤ Tiempo de respuesta reducido a la mitad.
    ✤ Cobertura de tests > 85%

    View Slide

  30. ticketea
    django-configurations
    settings
    ├── base.py
    ├── bundles
    │ ├── crispy.py
    │ ├── gunicorn.py
    │ ├── pipeline.py
    │ ├── raven.py
    │ ├── statsd.py
    │ └── tkt.py
    ├── development.py
    ├── production.py
    └── testing.py

    View Slide

  31. ticketea
    django-configurations
    class Base(CrispyForms, Settings):
    DJANGO_APPS = []
    BUNDLE_APPS = []
    TKT_APPS = []
    @property
    def INSTALLED_APPS(self):
    return self.DJANGO_APPS + self.BUNDLE_APPS + self.TKT_APPS
    @property
    def DATABASES(self):
    return {
    'default': {
    'ENGINE': 'django.db.backends.mysql',
    'NAME': 'db_name',
    'USER': 'db_user',
    'PASSWORD': self.DB_PASSWORD,
    'HOST': 'localhost',
    }
    }

    View Slide

  32. ticketea
    django-configurations
    class Production(Gunicorn, Base):
    DB_PASSWORD = values.Value()
    class Development(Base):
    DB_PASSWORD = 'pass'

    View Slide

  33. ticketea
    django-configurations
    class CrispyForms(object):
    def __init__(self):
    super(CrispyForms, self).__init__()
    self.BUNDLE_APPS += (
    'crispy_forms',
    )

    View Slide

  34. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)

    View Slide

  35. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)

    View Slide

  36. ticketea
    Uso del api client
    api.user(6077).event(1).update({
    'name': name,
    'description': description
    })

    View Slide

  37. ticketea
    Creando un endpoint…
    class User(CRUDEndpoint):
    def __init__(self, pk):
    super(User, self).__init__(pk)
    self.add_endpoint(Event)
    def activate(self, activation_key, email):
    return self._http_client.request.post(
    '%s/activate' % self.url,
    data={
    'activation_key': activation_key,
    'email': email
    }
    )

    View Slide

  38. ticketea
    Internals del api-client
    class RestCRUDEndpoint(object):
    . . .
    def add_endpoint(self, klass):
    url_endpoint = klass.__name__.lower()
    python_name = from_camel_to_snake(url_endpoint)
    list_name = plural_name(python_name)
    create_name = 'create_' + python_name
    klass.endpoint = '%s/%s' % (self.url, url_endpoint)
    klass._http_client = self._http_client
    setattr(self, python_name, klass)
    setattr(
    self,
    create_name,
    self._create_object(url_endpoint)
    )

    View Slide

  39. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)
    TicketeaApiMiddleware
    TicketeaApiException

    View Slide

  40. ticketea
    Inyectando el api-client en el request
    class TicketeaApiMiddleware(object):
    def process_request(self, request):
    if not hasattr(request, 'user'):
    raise ImproperlyConfigured('error')
    if isinstance(request.user, AnonymousUser):
    self._login_as_guest(request)
    request.api = create_api_client(...)
    def process_response(self, request, response):
    # Check if the token was refreshed
    return response

    View Slide

  41. ticketea
    Manejando errores del API
    class TicketeaApiExceptions(object):
    def process_exception(self, request, exception):
    if isinstance(exception, ApiException):
    if exception.code == self.REQUEST_TOKEN_EXPIRED:
    return self._manage_expired_token(request)
    elif isinstance(exception, ForbiddenException):
    raise PermissionDenied
    elif isinstance(exception, NotFoundException):
    raise Http404

    View Slide

  42. ticketea
    Nuestro flujo…
    tkt-auth
    (django app)
    Frontend
    (Django)
    api-client
    (request.api)
    ApiDispatchMixin
    TktTemplateView

    View Slide

  43. ticketea
    Mixins
    class ApiDispatchMixin(object):
    def dispatch(self, *args, **kwargs):
    if hasattr(self, 'get_api_data'):
    result = self.get_api_data()
    if isinstance(result, HttpResponse):
    return result
    return super(ApiDispatchMixin, self).dispatch(
    *args,
    **kwargs
    )

    View Slide

  44. ticketea
    Mixins
    # FAIL
    class AccessKwargsMixin(object):
    def get_context_data(self, **kwargs):
    kw = super(AccessKwargsMixin, self) \
    .get_context_data(**kwargs)
    kw['url_kwargs'] = self.kwargs
    return kw

    View Slide

  45. ticketea
    Nuestras vistas genéricas
    class TktView(
    LoginRequiredMixin, UserInjectorMixin,
    RoleRequiredMixin, ApiDispatchMixin, View
    )
    class TktTemplateView(
    LoginRequiredMixin, UserInjectorMixin,
    RoleRequiredMixin, ApiDispatchMixin,
    TemplateView
    )

    View Slide

  46. ticketea
    Clase base de nuestros FormView
    class TktFormView(FormView):
    def post(self, request, *args, **kwargs):
    form_class = self.get_form_class()
    form = self.get_form(form_class)
    if form.is_valid():
    validate_and_commit = getattr(self, 'validate_and_commit', lambda x: True)
    validate_result = validate_and_commit(form)
    if validate_result is None:
    raise ValueError('validate_and_commit must return something')
    if isinstance(validate_result, HttpResponse):
    return validate_result
    elif validate_result:
    return self.form_valid(form)
    else:
    return self.form_invalid(form)
    else:
    messages.error(request, _('there has been an error'))
    return self.form_invalid(form)

    View Slide

  47. ticketea
    Clase base de nuestros Form
    @decorator
    def handle_api_errors(form_save_method, self, *args, **kwargs):
    try:
    save_method = form_save_method(self, *args, **kwargs)
    return save_method if save_method is not None else True
    except ApiException as e:
    self.fill_form_errors(e.attrs)
    return False
    class TktForm(forms.Form):
    api_field_names = {}
    def fill_form_errors(self, errors):
    for field, errno in errors.items():
    real_field_name = self.api_field_names.get(field, field)
    error_description = API_ERRORS_NAMES.get(errno)
    self._errors[real_field_name] = self.error_class(
    [error_description]
    )

    View Slide

  48. ticketea
    Ejemplo de formulario
    class AccountInfoForm(TktForm):
    email = forms.EmailField()
    def __init__(self, *args, **kwargs):
    super(AccountInfoForm, self).__init__(*args, **kwargs)
    self.helper = AccountInfoHelper()
    @handle_api_errors
    def save(self, user):
    user.update(**self.cleaned_data)

    View Slide

  49. ticketea
    Ejemplo de vista
    class AccountInfoView(TktFormView):
    form_class = AccountInfoForm
    template_name = 'users_config/users/profile.html'
    min_role = 'organizer'
    def get_api_data(self):
    self.user = self.request.api.user(self.kwargs['id'])
    def validate_and_commit(self, form):
    committed = form.save(self.user)
    if committed:
    messages.success(self.request, _('OK'))
    else:
    messages.error(self.request, _('KO'))
    return committed
    def get_success_url(self):
    return reverse('profile_url')

    View Slide

  50. ticketea
    third-party apps
    ✤ django-crispy-forms
    ✤ django-braces
    ✤ django-configurations
    ✤ django-pipeline
    ✤ django-secure
    ✤ django-debug-toolbar
    ✤ werkzeug
    ✤ HTTPretty

    View Slide

  51. heracles

    View Slide

  52. ticketea
    Arquitectura
    RabbitMQ
    API
    (PHP)
    Celery
    Celery
    Celery
    {

    View Slide

  53. ticketea
    Tarea base
    class BaseTask(Task):
    abstract = True
    ignore_result = True
    max_retries = 5
    def retry_time(self, retries):
    RETRIES = [10, 15, 20, 30]
    try:
    return RETRIES[retries]
    except IndexError:
    return RETRIES[-1]
    def retry(self, *args, **kwargs):
    retries = self.request.retries + 1
    max_retries = kwargs.get('max_retries', self.max_retries)
    if retries > max_retries * 0.3:
    logger.warning(...)
    kwargs['countdown'] = self.retry_time(retries - 1)
    return super(BaseTask, self).retry(*args, **kwargs)

    View Slide

  54. ticketea
    Tarea base
    class BaseTask(Task):
    def on_success(self, *args, **kwargs):
    statsd.incr('tasks.%s' % self.name)
    def on_failure(self, exc, task_id, args, kwargs, einfo):
    logger.warning(...)
    def __call__(self, *args, **kwargs):
    with transaction.atomic():
    try:
    return super(BaseTask, self).__call__(
    *args, **kwargs
    )
    except Exception as exc:
    if isinstance(exc, exceptions.Retry):
    raise exc
    else:
    logger.exception('Unhandled exception')
    self.retry(exc=exc)

    View Slide

  55. no hay solución perfecta

    View Slide

  56. thor

    View Slide

  57. ticketea
    Objetivos
    ✤ Pasar a sistemas distribuídos
    ✤ Permitir el funcionamiento degradado del sistema
    ✤ Poder desarrollar cada proyecto a una velocidad distinta
    ✤ Poder ajustar la infraestructura usada de cada proyecto
    ✤ Reducir puntos únicos de fallo

    View Slide

  58. ticketea
    Arquitectura
    RabbitMQ
    API
    (PHP)
    Celery
    Celery
    Celery
    {
    THOR
    DBLS
    Tasks

    View Slide

  59. ticketea
    Arquitectura
    RabbitMQ
    API
    (PHP)
    Celery
    Celery
    Celery
    {
    THOR
    DBLS
    Tasks
    Views
    X

    View Slide

  60. Más importante que tener un equipo A…

    View Slide

  61. … es tener un plan B

    View Slide

  62. graphing
    &&
    logging

    View Slide

  63. ticketea
    Herramientas usadas
    ✤ sentry
    ✤ graphite
    ✤ grafana
    ✤ rsyslog
    ✤ logstash
    ✤ cabot

    View Slide

  64. ticketea
    Sabina, Noviembre 2014

    View Slide

  65. ticketea
    PREGUNTAS

    View Slide

  66. @ticketeaEng
    GRACIAS

    View Slide