Tweet
Tweet

Slide 1

Slide 1 text

internals @ticketeaEng

Slide 2

Slide 2 text

ticketea ¿Quiénes somos? @esanchezm @adiazher @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg @loquedigairune @Leidan @RafaRM20

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

ticketea Al Principio… ✤ Primera web, usando .NET ✤ Reescritura en PHP y Yii ✤ Reescritura del frontal en PHP (framework propio)

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

ticketea Actualmente… ✤ De 4 personas a 12, en un año y medio ✤ Sin perfil de sistemas/dba puro ✤ Sistemas en AWS ✤ Hemos llegado a varios países

Slide 7

Slide 7 text

Cambio de Estrategia

Slide 8

Slide 8 text

No ha sido un camino fácil…

Slide 9

Slide 9 text

odin

Slide 10

Slide 10 text

ticketea Herramienta de informes ✤ Business Intelligence ✤ Exportación de informes (PDF/XLSX) ✤ Compatibilidad de sesiones con PHP

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

ticketea Compatibilidad con lo anterior… class SessionStore(CacheStore): def load(self): pass def save(self, must_create=False): pass @property def cache_key(self): pass def decode(self, session_data): pass def encode(self, session_dict): pass

Slide 13

Slide 13 text

ticketea SessionStore -> CacheStore def load(self): try: session_data = self.decode( self._cache.get(self.cache_key, None) ) except Exception: session_data = None if session_data is not None and 'user' in session_data: try: session_data[SESSION_KEY] = session_data['user']['id'] except KeyError: pass return session_data self.create() return {}

Slide 14

Slide 14 text

ticketea SessionStore -> CacheStore def save(self, must_create=False): if must_create: func = self._cache.add else: func = self._cache.set result = func( self.cache_key, self.encode(self._get_session(no_load=must_create)), self.get_expiry_age() ) if must_create and not result: raise CreateError

Slide 15

Slide 15 text

ticketea SessionStore -> CacheStore @property def cache_key(self): return "memc.sess.key." + self._get_or_create_session_key() def decode(self, session_data): if session_data is not None: return PHPUnserialize().session_decode(session_data) return session_data def encode(self, session_dict): return PHPSerialize().session_encode(session_dict)

Slide 16

Slide 16 text

ticketea Usando el admin sin modelos… # __init__.py from . import admin_models if 'django.contrib.admin.models' in sys.modules: del sys.modules['django.contrib.admin.models'] sys.modules['django.contrib.admin.models'] = admin_models from django.contrib.admin.sites import AdminSite AdminSite.check_dependencies = nothing from django.core.management.base import BaseCommand BaseCommand.validate = lambda x, *args, **kwargs: None

Slide 17

Slide 17 text

pylongtable

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

ticketea

Slide 20

Slide 20 text

Python Era Begins

Slide 21

Slide 21 text

forseti

Slide 22

Slide 22 text

ticketea ¿Qué es forseti? ✤ Gestionar grupos de auto-escalado ✤ Generación de AMIs ✤ Despliegue de nuevo código ✤ Gestión de alarmas

Slide 23

Slide 23 text

ticketea ¿Qué es forseti? FORSETI BOTO BOTO-CORE EC2AutoScaleConfig ELBalancer CloudWatchMetricAlarm EC2AutoScaleGroup EC2AutoScalePolicy GoldenEC2Image EC2Instance

Slide 24

Slide 24 text

ticketea RELEASE THE KRAKEN

Slide 25

Slide 25 text

frontend

Slide 26

Slide 26 text

ticketea Arquitectura de sistemas API Load Balancer API API API { Frontend Frontend Frontend Frontend Load Balancer DB { Usuario

Slide 27

Slide 27 text

ticketea Arquitectura del frontend supervisor chaussette

Slide 28

Slide 28 text

ticketea Nuestro “django” Modelos Contrib auth ModelForms DB Templates Sesiones CBVs NO SÍ

Slide 29

Slide 29 text

ticketea Resultado ✤ La planificación fue un desastre: 2x tiempo. ✤ Mejor logging y reporting de errores. ✤ Tiempo de respuesta reducido a la mitad. ✤ Cobertura de tests > 85%

Slide 30

Slide 30 text

ticketea django-configurations settings ├── base.py ├── bundles │ ├── crispy.py │ ├── gunicorn.py │ ├── pipeline.py │ ├── raven.py │ ├── statsd.py │ └── tkt.py ├── development.py ├── production.py └── testing.py

Slide 31

Slide 31 text

ticketea django-configurations class Base(CrispyForms, Settings): DJANGO_APPS = [] BUNDLE_APPS = [] TKT_APPS = [] @property def INSTALLED_APPS(self): return self.DJANGO_APPS + self.BUNDLE_APPS + self.TKT_APPS @property def DATABASES(self): return { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'db_name', 'USER': 'db_user', 'PASSWORD': self.DB_PASSWORD, 'HOST': 'localhost', } }

Slide 32

Slide 32 text

ticketea django-configurations class Production(Gunicorn, Base): DB_PASSWORD = values.Value() class Development(Base): DB_PASSWORD = 'pass'

Slide 33

Slide 33 text

ticketea django-configurations class CrispyForms(object): def __init__(self): super(CrispyForms, self).__init__() self.BUNDLE_APPS += ( 'crispy_forms', )

Slide 34

Slide 34 text

ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)

Slide 35

Slide 35 text

ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api)

Slide 36

Slide 36 text

ticketea Uso del api client api.user(6077).event(1).update({ 'name': name, 'description': description })

Slide 37

Slide 37 text

ticketea Creando un endpoint… class User(CRUDEndpoint): def __init__(self, pk): super(User, self).__init__(pk) self.add_endpoint(Event) def activate(self, activation_key, email): return self._http_client.request.post( '%s/activate' % self.url, data={ 'activation_key': activation_key, 'email': email } )

Slide 38

Slide 38 text

ticketea Internals del api-client class RestCRUDEndpoint(object): . . . def add_endpoint(self, klass): url_endpoint = klass.__name__.lower() python_name = from_camel_to_snake(url_endpoint) list_name = plural_name(python_name) create_name = 'create_' + python_name klass.endpoint = '%s/%s' % (self.url, url_endpoint) klass._http_client = self._http_client setattr(self, python_name, klass) setattr( self, create_name, self._create_object(url_endpoint) )

Slide 39

Slide 39 text

ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api) TicketeaApiMiddleware TicketeaApiException

Slide 40

Slide 40 text

ticketea Inyectando el api-client en el request class TicketeaApiMiddleware(object): def process_request(self, request): if not hasattr(request, 'user'): raise ImproperlyConfigured('error') if isinstance(request.user, AnonymousUser): self._login_as_guest(request) request.api = create_api_client(...) def process_response(self, request, response): # Check if the token was refreshed return response

Slide 41

Slide 41 text

ticketea Manejando errores del API class TicketeaApiExceptions(object): def process_exception(self, request, exception): if isinstance(exception, ApiException): if exception.code == self.REQUEST_TOKEN_EXPIRED: return self._manage_expired_token(request) elif isinstance(exception, ForbiddenException): raise PermissionDenied elif isinstance(exception, NotFoundException): raise Http404

Slide 42

Slide 42 text

ticketea Nuestro flujo… tkt-auth (django app) Frontend (Django) api-client (request.api) ApiDispatchMixin TktTemplateView

Slide 43

Slide 43 text

ticketea Mixins class ApiDispatchMixin(object): def dispatch(self, *args, **kwargs): if hasattr(self, 'get_api_data'): result = self.get_api_data() if isinstance(result, HttpResponse): return result return super(ApiDispatchMixin, self).dispatch( *args, **kwargs )

Slide 44

Slide 44 text

ticketea Mixins # FAIL class AccessKwargsMixin(object): def get_context_data(self, **kwargs): kw = super(AccessKwargsMixin, self) \ .get_context_data(**kwargs) kw['url_kwargs'] = self.kwargs return kw

Slide 45

Slide 45 text

ticketea Nuestras vistas genéricas class TktView( LoginRequiredMixin, UserInjectorMixin, RoleRequiredMixin, ApiDispatchMixin, View ) class TktTemplateView( LoginRequiredMixin, UserInjectorMixin, RoleRequiredMixin, ApiDispatchMixin, TemplateView )

Slide 46

Slide 46 text

ticketea Clase base de nuestros FormView class TktFormView(FormView): def post(self, request, *args, **kwargs): form_class = self.get_form_class() form = self.get_form(form_class) if form.is_valid(): validate_and_commit = getattr(self, 'validate_and_commit', lambda x: True) validate_result = validate_and_commit(form) if validate_result is None: raise ValueError('validate_and_commit must return something') if isinstance(validate_result, HttpResponse): return validate_result elif validate_result: return self.form_valid(form) else: return self.form_invalid(form) else: messages.error(request, _('there has been an error')) return self.form_invalid(form)

Slide 47

Slide 47 text

ticketea Clase base de nuestros Form @decorator def handle_api_errors(form_save_method, self, *args, **kwargs): try: save_method = form_save_method(self, *args, **kwargs) return save_method if save_method is not None else True except ApiException as e: self.fill_form_errors(e.attrs) return False class TktForm(forms.Form): api_field_names = {} def fill_form_errors(self, errors): for field, errno in errors.items(): real_field_name = self.api_field_names.get(field, field) error_description = API_ERRORS_NAMES.get(errno) self._errors[real_field_name] = self.error_class( [error_description] )

Slide 48

Slide 48 text

ticketea Ejemplo de formulario class AccountInfoForm(TktForm): email = forms.EmailField() def __init__(self, *args, **kwargs): super(AccountInfoForm, self).__init__(*args, **kwargs) self.helper = AccountInfoHelper() @handle_api_errors def save(self, user): user.update(**self.cleaned_data)

Slide 49

Slide 49 text

ticketea Ejemplo de vista class AccountInfoView(TktFormView): form_class = AccountInfoForm template_name = 'users_config/users/profile.html' min_role = 'organizer' def get_api_data(self): self.user = self.request.api.user(self.kwargs['id']) def validate_and_commit(self, form): committed = form.save(self.user) if committed: messages.success(self.request, _('OK')) else: messages.error(self.request, _('KO')) return committed def get_success_url(self): return reverse('profile_url')

Slide 50

Slide 50 text

ticketea third-party apps ✤ django-crispy-forms ✤ django-braces ✤ django-configurations ✤ django-pipeline ✤ django-secure ✤ django-debug-toolbar ✤ werkzeug ✤ HTTPretty

Slide 51

Slide 51 text

heracles

Slide 52

Slide 52 text

ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery {

Slide 53

Slide 53 text

ticketea Tarea base class BaseTask(Task): abstract = True ignore_result = True max_retries = 5 def retry_time(self, retries): RETRIES = [10, 15, 20, 30] try: return RETRIES[retries] except IndexError: return RETRIES[-1] def retry(self, *args, **kwargs): retries = self.request.retries + 1 max_retries = kwargs.get('max_retries', self.max_retries) if retries > max_retries * 0.3: logger.warning(...) kwargs['countdown'] = self.retry_time(retries - 1) return super(BaseTask, self).retry(*args, **kwargs)

Slide 54

Slide 54 text

ticketea Tarea base class BaseTask(Task): def on_success(self, *args, **kwargs): statsd.incr('tasks.%s' % self.name) def on_failure(self, exc, task_id, args, kwargs, einfo): logger.warning(...) def __call__(self, *args, **kwargs): with transaction.atomic(): try: return super(BaseTask, self).__call__( *args, **kwargs ) except Exception as exc: if isinstance(exc, exceptions.Retry): raise exc else: logger.exception('Unhandled exception') self.retry(exc=exc)

Slide 55

Slide 55 text

no hay solución perfecta

Slide 56

Slide 56 text

thor

Slide 57

Slide 57 text

ticketea Objetivos ✤ Pasar a sistemas distribuídos ✤ Permitir el funcionamiento degradado del sistema ✤ Poder desarrollar cada proyecto a una velocidad distinta ✤ Poder ajustar la infraestructura usada de cada proyecto ✤ Reducir puntos únicos de fallo

Slide 58

Slide 58 text

ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery { THOR DBLS Tasks

Slide 59

Slide 59 text

ticketea Arquitectura RabbitMQ API (PHP) Celery Celery Celery { THOR DBLS Tasks Views X

Slide 60

Slide 60 text

Más importante que tener un equipo A…

Slide 61

Slide 61 text

… es tener un plan B

Slide 62

Slide 62 text

graphing && logging

Slide 63

Slide 63 text

ticketea Herramientas usadas ✤ sentry ✤ graphite ✤ grafana ✤ rsyslog ✤ logstash ✤ cabot

Slide 64

Slide 64 text

ticketea Sabina, Noviembre 2014

Slide 65

Slide 65 text

ticketea PREGUNTAS

Slide 66

Slide 66 text

@ticketeaEng GRACIAS