Share
Embed
Travisify - Project life cycle in the own repository
Miguel Ángel's (@magmax9) talk at PyConEs'14
Tuenti
November 09, 2014
Transcript
58 slides
-
ABOUT ME Miguel Ángel García SRE (Systems Reliability Engineer) at
@magmax9 http://magmax.org -
TESTING EVOLUTION No testing at all Some manual testing performed
by QA team Some Acceptation tests written by QA team Unit, Integration and Acceptation tests written by developers (no QA team) -
The bad guy at the begining Time saver later Deployments
center right now -
TESTING Ad hoc testing environment One team to manage the
testing environment: Testing Framework (TFW) -
PROBLEMS It doesn't scale to other repositories/projects The team was
not able to deal with problems developers are used to. Developers want to improve to that environment -
PROBLEMS Impossible to be managed by just one team Too
different technologies: PHP: fpm, hphpi Java: tomcat, jboss, android Erlang, Python, Ruby, ... make, rake, ant, maven, gradle, ... -
TRAVIS IDEA The repository contains more than code: The definition
of the testing environment The test execution commands -
OUR REQUIREMENTS One job per branch Dynamic reconfiguration Special jobs
for releases Just one testing environment per project Some projects require several environments (client- server) -
APROACH 1. Create a XML template 2. Parameterize the XML
template 3. Use a YAML file in the repo to store values -
Later on, we found that our library must communicate with
Jenkins in order to: ERRORS Check if a plugin is installed Check a plugin version Still to be done... -
USE CASE 1: CREATION We created the "tu-ci" (Tuenti Continuous
Integration) command: Reads information about working copy Sends a message to Flow to create the repository Flow reads the .ci.yml and reconfigures the Jenkins Job -
FLOW Is our centralized deployments tool Django You must be
tired to hear us talking about it -
USE CASE 2: UPDATE We added a hook in our
git and mercurial repositories: It just sends a curl to Flow Flow regenerates the Jenkins Job -
USE CASE 3: LAUNCH The Jenkins Job: 1. Generates a
VirtualEnv environment 2. Installs cigen 3. Uses the cigen runner to parse(again) the .ci.yml -
TELL ME WHAT TO DO And not where, when or
how Write meta-commands in order to help developers: Create resources for servers (BBDDs, memcache, ...) Deploy the server side for clients Launch VMs -
META FEATURES We discovered that: Some commands must be executed
to setup a job Some commands should be executed on success/failure -
SCHEMA Platform where it should run Workflows are lists of
Stages Stages are lists of Steps Steps are shell scripts Plugins are... a best effort :D -
SCHEMA PYTHON RULES!! d e f p r o j
e c t _ f r o m _ s t r e a m ( s t r e a m , p r o j e c t = N o n e , y a m l _ l i b = N o n e ) : y a m l _ l i b = y a m l _ l i b o r y a m l p r o j e c t = p r o j e c t o r P r o j e c t ( ) c o n t e n t = y a m l _ l i b . l o a d ( s t r e a m ) p r o j e c t . p l a t f o r m = c o n t e n t . g e t ( ' p l a t f o r m ' , ' d e f a u l t ' ) p r o j e c t . t i m e o u t = c o n t e n t . g e t ( ' t i m e o u t ' , p r o j e c t . t i m e o u t ) p r o j e c t . p a r a m e t e r s = c o n t e n t . g e t ( ' p a r a m e t e r s ' , { } ) p r o j e c t . p l u g i n s = c o n t e n t . g e t ( ' p l u g i n s ' , { } ) p r o j e c t . s t a g e s = c o n t e n t . g e t ( ' s t a g e s ' , { } ) p r o j e c t . w o r k f l o w s = c o n t e n t . g e t ( ' w o r k f l o w s ' , { } ) r e t u r n p r o j e c t -
SCHEMA EXAMPLE p l a t f o r m
: a n d r o i d w o r k f l o w s : f a s t : [ c o m p i l e , u n i t ] f u l l : [ c o m p i l e , u n i t , i n t e g r a t i o n ] r e l e a s e : [ c o m p i l e , u n i t , p u b l i s h ] s t a g e s : c o m p i l e : - f i n d - n a m e " * . p y c " - d e l e t e - m a k e u n i t : p y . t e s t t e s t s / u n i t i n t e g r a t i o n : p y . t e s t t e s t s / i n t e g r a t i o n p u b l i s h : p y t h o n s e t u p . p y b d i s t _ w h e e l u p l o a d p l u g i n s : c o v e r a g e : f i l e : c o v e r a g e . x m l s o n a r : -
c l a s s T e m p l
a t e L o a d e r ( o b j e c t ) : d e f g e t _ t e m p l a t e ( s e l f , n a m e ) : a s s e r t i s i n s t a n c e ( n a m e , s t r ) t e m p l a t e _ f i l e = s e l f . _ t e m p l a t e _ f i l e ( n a m e ) l o g g e r . d e b u g ( " L o a d i n g t e m p l a t e % s " , t e m p l a t e _ f i l e ) w i t h o p e n ( t e m p l a t e _ f i l e ) a s f d : r e t u r n T e m p l a t e ( f d . r e a d ( ) ) d e f _ t e m p l a t e _ f i l e ( s e l f , n a m e ) : c u r r e n t = o s . p a t h . d i r n a m e ( _ _ f i l e _ _ ) f i l e n a m e = n a m e + ' . j i n j a ' r e t u r n o s . p a t h . j o i n ( c u r r e n t , ' t e m p l a t e s ' , f i l e n a m e ) c l a s s G e n e r a t o r ( o b j e c t ) : T E M P L A T E = N o n e d e f _ _ i n i t _ _ ( s e l f , p r o j e c t , l o a d e r = N o n e ) : s e l f . p r o j e c t = p r o j e c t s e l f . _ l o a d e r = l o a d e r o r T e m p l a t e L o a d e r ( ) d e f g e n e r a t e ( s e l f , r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w ) : t e m p l a t e = s e l f . _ l o a d e r . g e t _ t e m p l a t e ( s e l f . T E M P L A T E ) r e t u r n t e m p l a t e . r e n d e r ( p r o j e c t = s e l f . p r o j e c t , c r e a t i o n _ t i m e = t i m e . t i m e ( ) , p r o j e c t _ f i l e _ p a t h = r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w = w o r k f l o w , c i g e n = C I G E N , ) -
c l a s s J e n k i
n s G e n e r a t o r ( G e n e r a t o r ) : T E M P L A T E N A M E = ' j e n k i n s ' -
p r o j e c t = c i
g e n . l o a d e r . p r o j e c t _ f r o m _ f i l e ( a r g s . p r o j e c t _ f i l e ) p r o j e c t . r e p o s i t o r y . u r l = ' h t t p : / / e x a m p l e . o r g / f o o / b a r ' p r o j e c t . r e p o s i t o r y . k i n d = c i g e n . p r o j e c t . R e p o s i t o r y . G I T p r o j e c t . r e p o s i t o r y . b r a n c h = a r g s . r e p o _ b r a n c h g e n e r a t o r = c i g e n . g e n e r a t o r . J e n k i n s G e n e r a t o r ( p r o j e c t ) p r i n t g e n e r a t o r . g e n e r a t e ( a r g s . p r o j e c t _ f i l e , a r g s . w o r k f l o w ) -
d e f r u n ( s e l
f ) : s e l f . _ c o n f i g u r e _ p l u g i n s ( ) s e l f . _ u p d a t e _ e n v i r o n m e n t ( ) i f ' s e t u p ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f a u l t S e t u p ' , s e l f . _ c o n f i g [ ' s e t u p ' ] ) t r y : f o r s t a g e i n s e l f . _ p r o j e c t . w o r k f l o w s [ s e l f . _ w o r k f l o w ] : s t e p s = s e l f . _ p r o j e c t . s t a g e s . g e t ( s t a g e , N o n e ) i f s t e p s i s N o n e : l o g g e r . w a r n i n g ( ' E r r o r o n S t a g e " % s " ' , s t a g e ) c o n t i n u e s e l f . _ r u n _ s t a g e ( s t a g e , s t e p s ) s e l f . _ r u n _ s t a g e ( ' D e f a u l t S u c c e s s ' , s e l f . _ c o n f i g . g e t ( ' o n _ s u c c e s s ' , [ ] ) , p l u g i n = ' o n _ s ' ) l o g g e r . i n f o ( ' S U C C E S S ' ) e x c e p t E x c e p t i o n : i f ' o n _ f a i l u r e ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . F a i l u r e ' , s e l f . _ c o n f i g [ ' o n _ f ' ] ) r a i s e f i n a l l y : i f ' t e a r d o w n ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . T e a r d o w n ' , s e l f . _ c o n f i g [ ' t . ' ] ) s e l f . _ p r i n t _ r e p o r t ( ) -
NEW REQUIREMENTS Cigen should communicate to Jenkins Check if a
plugin is installed Check plugin version and compatibility Modularize the current (HUGE) template Move logic to code instead of template Improve integration: Go-CI, Travis, ... Jira, Taiga, ...