Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Travisify - Project life cycle in the own repository

Tuenti
November 09, 2014

Travisify - Project life cycle in the own repository

Miguel Ángel's (@magmax9) talk at PyConEs'14

Tuenti

November 09, 2014
Tweet

More Decks by Tuenti

Other Decks in Technology

Transcript

  1. TESTING EVOLUTION No testing at all Some manual testing performed

    by QA team Some Acceptation tests written by QA team Unit, Integration and Acceptation tests written by developers (no QA team)
  2. TESTING Ad hoc testing environment One team to manage the

    testing environment: Testing Framework (TFW)
  3. PROBLEMS It doesn't scale to other repositories/projects The team was

    not able to deal with problems developers are used to. Developers want to improve to that environment
  4. PROBLEMS Impossible to be managed by just one team Too

    different technologies: PHP: fpm, hphpi Java: tomcat, jboss, android Erlang, Python, Ruby, ... make, rake, ant, maven, gradle, ...
  5. TRAVIS IDEA The repository contains more than code: The definition

    of the testing environment The test execution commands
  6. OUR REQUIREMENTS One job per branch Dynamic reconfiguration Special jobs

    for releases Just one testing environment per project Some projects require several environments (client- server)
  7. APROACH 1. Create a XML template 2. Parameterize the XML

    template 3. Use a YAML file in the repo to store values
  8. Later on, we found that our library must communicate with

    Jenkins in order to: ERRORS Check if a plugin is installed Check a plugin version Still to be done...
  9. USE CASE 1: CREATION We created the "tu-ci" (Tuenti Continuous

    Integration) command: Reads information about working copy Sends a message to Flow to create the repository Flow reads the .ci.yml and reconfigures the Jenkins Job
  10. USE CASE 2: UPDATE We added a hook in our

    git and mercurial repositories: It just sends a curl to Flow Flow regenerates the Jenkins Job
  11. USE CASE 3: LAUNCH The Jenkins Job: 1. Generates a

    VirtualEnv environment 2. Installs cigen 3. Uses the cigen runner to parse(again) the .ci.yml
  12. TELL ME WHAT TO DO And not where, when or

    how Write meta-commands in order to help developers: Create resources for servers (BBDDs, memcache, ...) Deploy the server side for clients Launch VMs
  13. META FEATURES We discovered that: Some commands must be executed

    to setup a job Some commands should be executed on success/failure
  14. SCHEMA Platform where it should run Workflows are lists of

    Stages Stages are lists of Steps Steps are shell scripts Plugins are... a best effort :D
  15. SCHEMA PYTHON RULES!! d e f p r o j

    e c t _ f r o m _ s t r e a m ( s t r e a m , p r o j e c t = N o n e , y a m l _ l i b = N o n e ) : y a m l _ l i b = y a m l _ l i b o r y a m l p r o j e c t = p r o j e c t o r P r o j e c t ( ) c o n t e n t = y a m l _ l i b . l o a d ( s t r e a m ) p r o j e c t . p l a t f o r m = c o n t e n t . g e t ( ' p l a t f o r m ' , ' d e f a u l t ' ) p r o j e c t . t i m e o u t = c o n t e n t . g e t ( ' t i m e o u t ' , p r o j e c t . t i m e o u t ) p r o j e c t . p a r a m e t e r s = c o n t e n t . g e t ( ' p a r a m e t e r s ' , { } ) p r o j e c t . p l u g i n s = c o n t e n t . g e t ( ' p l u g i n s ' , { } ) p r o j e c t . s t a g e s = c o n t e n t . g e t ( ' s t a g e s ' , { } ) p r o j e c t . w o r k f l o w s = c o n t e n t . g e t ( ' w o r k f l o w s ' , { } ) r e t u r n p r o j e c t
  16. SCHEMA EXAMPLE p l a t f o r m

    : a n d r o i d w o r k f l o w s : f a s t : [ c o m p i l e , u n i t ] f u l l : [ c o m p i l e , u n i t , i n t e g r a t i o n ] r e l e a s e : [ c o m p i l e , u n i t , p u b l i s h ] s t a g e s : c o m p i l e : - f i n d - n a m e " * . p y c " - d e l e t e - m a k e u n i t : p y . t e s t t e s t s / u n i t i n t e g r a t i o n : p y . t e s t t e s t s / i n t e g r a t i o n p u b l i s h : p y t h o n s e t u p . p y b d i s t _ w h e e l u p l o a d p l u g i n s : c o v e r a g e : f i l e : c o v e r a g e . x m l s o n a r :
  17. c l a s s T e m p l

    a t e L o a d e r ( o b j e c t ) : d e f g e t _ t e m p l a t e ( s e l f , n a m e ) : a s s e r t i s i n s t a n c e ( n a m e , s t r ) t e m p l a t e _ f i l e = s e l f . _ t e m p l a t e _ f i l e ( n a m e ) l o g g e r . d e b u g ( " L o a d i n g t e m p l a t e % s " , t e m p l a t e _ f i l e ) w i t h o p e n ( t e m p l a t e _ f i l e ) a s f d : r e t u r n T e m p l a t e ( f d . r e a d ( ) ) d e f _ t e m p l a t e _ f i l e ( s e l f , n a m e ) : c u r r e n t = o s . p a t h . d i r n a m e ( _ _ f i l e _ _ ) f i l e n a m e = n a m e + ' . j i n j a ' r e t u r n o s . p a t h . j o i n ( c u r r e n t , ' t e m p l a t e s ' , f i l e n a m e ) c l a s s G e n e r a t o r ( o b j e c t ) : T E M P L A T E = N o n e d e f _ _ i n i t _ _ ( s e l f , p r o j e c t , l o a d e r = N o n e ) : s e l f . p r o j e c t = p r o j e c t s e l f . _ l o a d e r = l o a d e r o r T e m p l a t e L o a d e r ( ) d e f g e n e r a t e ( s e l f , r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w ) : t e m p l a t e = s e l f . _ l o a d e r . g e t _ t e m p l a t e ( s e l f . T E M P L A T E ) r e t u r n t e m p l a t e . r e n d e r ( p r o j e c t = s e l f . p r o j e c t , c r e a t i o n _ t i m e = t i m e . t i m e ( ) , p r o j e c t _ f i l e _ p a t h = r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w = w o r k f l o w , c i g e n = C I G E N , )
  18. c l a s s J e n k i

    n s G e n e r a t o r ( G e n e r a t o r ) : T E M P L A T E N A M E = ' j e n k i n s '
  19. p r o j e c t = c i

    g e n . l o a d e r . p r o j e c t _ f r o m _ f i l e ( a r g s . p r o j e c t _ f i l e ) p r o j e c t . r e p o s i t o r y . u r l = ' h t t p : / / e x a m p l e . o r g / f o o / b a r ' p r o j e c t . r e p o s i t o r y . k i n d = c i g e n . p r o j e c t . R e p o s i t o r y . G I T p r o j e c t . r e p o s i t o r y . b r a n c h = a r g s . r e p o _ b r a n c h g e n e r a t o r = c i g e n . g e n e r a t o r . J e n k i n s G e n e r a t o r ( p r o j e c t ) p r i n t g e n e r a t o r . g e n e r a t e ( a r g s . p r o j e c t _ f i l e , a r g s . w o r k f l o w )
  20. d e f r u n ( s e l

    f ) : s e l f . _ c o n f i g u r e _ p l u g i n s ( ) s e l f . _ u p d a t e _ e n v i r o n m e n t ( ) i f ' s e t u p ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f a u l t S e t u p ' , s e l f . _ c o n f i g [ ' s e t u p ' ] ) t r y : f o r s t a g e i n s e l f . _ p r o j e c t . w o r k f l o w s [ s e l f . _ w o r k f l o w ] : s t e p s = s e l f . _ p r o j e c t . s t a g e s . g e t ( s t a g e , N o n e ) i f s t e p s i s N o n e : l o g g e r . w a r n i n g ( ' E r r o r o n S t a g e " % s " ' , s t a g e ) c o n t i n u e s e l f . _ r u n _ s t a g e ( s t a g e , s t e p s ) s e l f . _ r u n _ s t a g e ( ' D e f a u l t S u c c e s s ' , s e l f . _ c o n f i g . g e t ( ' o n _ s u c c e s s ' , [ ] ) , p l u g i n = ' o n _ s ' ) l o g g e r . i n f o ( ' S U C C E S S ' ) e x c e p t E x c e p t i o n : i f ' o n _ f a i l u r e ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . F a i l u r e ' , s e l f . _ c o n f i g [ ' o n _ f ' ] ) r a i s e f i n a l l y : i f ' t e a r d o w n ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . T e a r d o w n ' , s e l f . _ c o n f i g [ ' t . ' ] ) s e l f . _ p r i n t _ r e p o r t ( )
  21. NEW REQUIREMENTS Cigen should communicate to Jenkins Check if a

    plugin is installed Check plugin version and compatibility Modularize the current (HUGE) template Move logic to code instead of template Improve integration: Go-CI, Travis, ... Jira, Taiga, ...