TESTING EVOLUTION No testing at all Some manual testing performed by QA team Some Acceptation tests written by QA team Unit, Integration and Acceptation tests written by developers (no QA team)
PROBLEMS It doesn't scale to other repositories/projects The team was not able to deal with problems developers are used to. Developers want to improve to that environment
PROBLEMS Impossible to be managed by just one team Too different technologies: PHP: fpm, hphpi Java: tomcat, jboss, android Erlang, Python, Ruby, ... make, rake, ant, maven, gradle, ...
OUR REQUIREMENTS One job per branch Dynamic reconfiguration Special jobs for releases Just one testing environment per project Some projects require several environments (client- server)
Later on, we found that our library must communicate with Jenkins in order to: ERRORS Check if a plugin is installed Check a plugin version Still to be done...
USE CASE 1: CREATION We created the "tu-ci" (Tuenti Continuous Integration) command: Reads information about working copy Sends a message to Flow to create the repository Flow reads the .ci.yml and reconfigures the Jenkins Job
TELL ME WHAT TO DO And not where, when or how Write meta-commands in order to help developers: Create resources for servers (BBDDs, memcache, ...) Deploy the server side for clients Launch VMs
SCHEMA PYTHON RULES!! d e f p r o j e c t _ f r o m _ s t r e a m ( s t r e a m , p r o j e c t = N o n e , y a m l _ l i b = N o n e ) : y a m l _ l i b = y a m l _ l i b o r y a m l p r o j e c t = p r o j e c t o r P r o j e c t ( ) c o n t e n t = y a m l _ l i b . l o a d ( s t r e a m ) p r o j e c t . p l a t f o r m = c o n t e n t . g e t ( ' p l a t f o r m ' , ' d e f a u l t ' ) p r o j e c t . t i m e o u t = c o n t e n t . g e t ( ' t i m e o u t ' , p r o j e c t . t i m e o u t ) p r o j e c t . p a r a m e t e r s = c o n t e n t . g e t ( ' p a r a m e t e r s ' , { } ) p r o j e c t . p l u g i n s = c o n t e n t . g e t ( ' p l u g i n s ' , { } ) p r o j e c t . s t a g e s = c o n t e n t . g e t ( ' s t a g e s ' , { } ) p r o j e c t . w o r k f l o w s = c o n t e n t . g e t ( ' w o r k f l o w s ' , { } ) r e t u r n p r o j e c t
SCHEMA EXAMPLE p l a t f o r m : a n d r o i d w o r k f l o w s : f a s t : [ c o m p i l e , u n i t ] f u l l : [ c o m p i l e , u n i t , i n t e g r a t i o n ] r e l e a s e : [ c o m p i l e , u n i t , p u b l i s h ] s t a g e s : c o m p i l e : - f i n d - n a m e " * . p y c " - d e l e t e - m a k e u n i t : p y . t e s t t e s t s / u n i t i n t e g r a t i o n : p y . t e s t t e s t s / i n t e g r a t i o n p u b l i s h : p y t h o n s e t u p . p y b d i s t _ w h e e l u p l o a d p l u g i n s : c o v e r a g e : f i l e : c o v e r a g e . x m l s o n a r :
c l a s s T e m p l a t e L o a d e r ( o b j e c t ) : d e f g e t _ t e m p l a t e ( s e l f , n a m e ) : a s s e r t i s i n s t a n c e ( n a m e , s t r ) t e m p l a t e _ f i l e = s e l f . _ t e m p l a t e _ f i l e ( n a m e ) l o g g e r . d e b u g ( " L o a d i n g t e m p l a t e % s " , t e m p l a t e _ f i l e ) w i t h o p e n ( t e m p l a t e _ f i l e ) a s f d : r e t u r n T e m p l a t e ( f d . r e a d ( ) ) d e f _ t e m p l a t e _ f i l e ( s e l f , n a m e ) : c u r r e n t = o s . p a t h . d i r n a m e ( _ _ f i l e _ _ ) f i l e n a m e = n a m e + ' . j i n j a ' r e t u r n o s . p a t h . j o i n ( c u r r e n t , ' t e m p l a t e s ' , f i l e n a m e ) c l a s s G e n e r a t o r ( o b j e c t ) : T E M P L A T E = N o n e d e f _ _ i n i t _ _ ( s e l f , p r o j e c t , l o a d e r = N o n e ) : s e l f . p r o j e c t = p r o j e c t s e l f . _ l o a d e r = l o a d e r o r T e m p l a t e L o a d e r ( ) d e f g e n e r a t e ( s e l f , r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w ) : t e m p l a t e = s e l f . _ l o a d e r . g e t _ t e m p l a t e ( s e l f . T E M P L A T E ) r e t u r n t e m p l a t e . r e n d e r ( p r o j e c t = s e l f . p r o j e c t , c r e a t i o n _ t i m e = t i m e . t i m e ( ) , p r o j e c t _ f i l e _ p a t h = r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w = w o r k f l o w , c i g e n = C I G E N , )
p r o j e c t = c i g e n . l o a d e r . p r o j e c t _ f r o m _ f i l e ( a r g s . p r o j e c t _ f i l e ) p r o j e c t . r e p o s i t o r y . u r l = ' h t t p : / / e x a m p l e . o r g / f o o / b a r ' p r o j e c t . r e p o s i t o r y . k i n d = c i g e n . p r o j e c t . R e p o s i t o r y . G I T p r o j e c t . r e p o s i t o r y . b r a n c h = a r g s . r e p o _ b r a n c h g e n e r a t o r = c i g e n . g e n e r a t o r . J e n k i n s G e n e r a t o r ( p r o j e c t ) p r i n t g e n e r a t o r . g e n e r a t e ( a r g s . p r o j e c t _ f i l e , a r g s . w o r k f l o w )
d e f r u n ( s e l f ) : s e l f . _ c o n f i g u r e _ p l u g i n s ( ) s e l f . _ u p d a t e _ e n v i r o n m e n t ( ) i f ' s e t u p ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f a u l t S e t u p ' , s e l f . _ c o n f i g [ ' s e t u p ' ] ) t r y : f o r s t a g e i n s e l f . _ p r o j e c t . w o r k f l o w s [ s e l f . _ w o r k f l o w ] : s t e p s = s e l f . _ p r o j e c t . s t a g e s . g e t ( s t a g e , N o n e ) i f s t e p s i s N o n e : l o g g e r . w a r n i n g ( ' E r r o r o n S t a g e " % s " ' , s t a g e ) c o n t i n u e s e l f . _ r u n _ s t a g e ( s t a g e , s t e p s ) s e l f . _ r u n _ s t a g e ( ' D e f a u l t S u c c e s s ' , s e l f . _ c o n f i g . g e t ( ' o n _ s u c c e s s ' , [ ] ) , p l u g i n = ' o n _ s ' ) l o g g e r . i n f o ( ' S U C C E S S ' ) e x c e p t E x c e p t i o n : i f ' o n _ f a i l u r e ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . F a i l u r e ' , s e l f . _ c o n f i g [ ' o n _ f ' ] ) r a i s e f i n a l l y : i f ' t e a r d o w n ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . T e a r d o w n ' , s e l f . _ c o n f i g [ ' t . ' ] ) s e l f . _ p r i n t _ r e p o r t ( )
NEW REQUIREMENTS Cigen should communicate to Jenkins Check if a plugin is installed Check plugin version and compatibility Modularize the current (HUGE) template Move logic to code instead of template Improve integration: Go-CI, Travis, ... Jira, Taiga, ...