Introduction Proposed approach Experimental results Conclusion A genetically optimized graph-based people extraction method for embedded transportation systems C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST, F59650, Villeneuve d’Ascq, France 2 Normandie Univ., UNICAEN, ENSICAEN, GREYC UMR CNRS 6072, Caen, France October −,  1/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Plan 1 Introduction Context In transport environnement 2 Proposed approach Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization 3 Experimental results Database Example of people extraction results Proposed method VS state of art foreground detection Comparison Precision/Recall Example : Full displacement of one person 4 Conclusion 2/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Context In transport environnement Introduction Context Video surveillance systems are widespread in transport applications in order to enhance the comfort and security of the infrastructure, as well as that of people. • People detection (in case of potentially dangerous situations) • Re-identification and tracking of people (optimize passenger flows) • Passagers Counting (queue) • Action recognition (falling) • ... Observation The common feature of these researches is the detection of moving objects 3/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Context In transport environnement In transport environnement Application Optimize the management of passenger flows in multimodal context Plan : Follow the trajectory of passengers (bus, tramway, subway) → People re-identification (in several cameras) → People extraction (of the scene) Our database contains many scientific locks (fast brightness changes, noise, shadow, scrolling background, etc.) Problem Many (foreground detection) methods exist in the literature but do not give satisfactory results in complex transport environments 4/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Purpose of the application Mean Idea Use foreground detection (based on state of art) to initialize a clustering graph-cut method and extract people Method is divided into 4 blocks 1 Pre-processing : mitigate brightness changes and noise 2 Foreground detection : Retrieving useful information 3 Post-treatment : Delete shadows and artefacts 4 People extraction using clustering method Optimization The complete proposed method (methods used and setting parameters) is optimized by a genetic algorithm 5/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Pre-processing Objective Combine image filters and invariants in order to mitigate : Effect of fast brightness changes Noise Methods implemented • 3 filters (blur, gaussian blur, median blur, bilateral) • 6 colorimetric invariants (greyworld, l1l2l3, m1m2m3, reduced coordinates, affine normalization, RGB rank) • Each pre-processing step is independent Optimization Best colorimetric invariant, filter and parameters will be automatically determined by the genetic algorithm 6/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Foreground Detection Objective Using foreground detection in order to approximately determine the position of people (on our database : no method gives very good results) Approaches implemented • Gaussian mixture model method • fuzzy based method • statistical method using color/texture features • neural networks method • non parametric method Optimization The best foreground detection method is performed by the genetic algorithm 7/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Foreground Detection Original Fuz Gaussian Fuz Sugeno Int MOG Adaptive SOM VuMeter Figure : Results of foreground detection methods on our database (BOSS project) 8/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Post-treatment Objective Delete shadows and others artefacts Original Foreground detection 9/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Post-treatment Methods implemented Robust detection of shadow Chromaticity based method Physical method Geometry based method Texture based method Background information is used to improve detection Background and foreground detections are dissociate Morphological filtering is used to delete artifacts Optimization The best shadows detection method, the best background detection method and the settings of morphological filtering are performed by the genetic algorithm 10/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization People Extraction Objective Refine the rough foreground detection obtained at the end of post-treatment block (increase the smoothness of the final graph clustering) Proposed strategy with 3 major steps 1 Convert image to superpixel graph by SLIC method 2 A bounding box is delimitated in order to limit the number of pixel to be considered Original SLIC segmentation Graph Figure : Example of graph calculated on a bounding box of interest 11/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization People Extraction Proposed strategy with 3 major steps 3 Graph clustering into 2 classes with a graph cut method : formulating the clustering problem as an energy minimization in the form of a labeling problem S(Ri , Rj ) is the similarity between two nodes (regions) S(Ri , Rj ) = exp − d(Ri , Rj ) 2θ2 . 1 dist(Ri , Rj ) The capacity W fg (Ri ) of the source (foreground) class for a region Ri is given by : W fg (Ri ) =    K if |Ri ∩ IFG | > α 0 if Ri ∩ IMASK = 0 λPfg (Ri )K otherwise The capacity Wbg(Ri ) of the sink (background) class for a region Ri is given by : W bg (Ri ) =    0 if |Ri ∩ IFG | > α K if Ri ∩ IMASK = 0 λPbg (Ri )K otherwise 12/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Graph Cut clustering Improve belonging unknown node Predict belonging of unknown node with color histograms 2 colors histograms (background/foreground) are calculated Belonging regions foreground/background Pfg (Ri ) = 3 c=1 pi ∈Ri Pfg c (pi , R(i)) 3 × m Pbg (Ri ) = 3 c=1 pi ∈Ri Pbg c (pi , R(i)) 3 × m 13/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Purpose of the application Pre-processing Foreground Detection Post-treatment People Extraction Genetic algorithm optimization Genetic algorithm optimization Problem Too many methods are implemented to have them tuned by hand Methods of optimization • Genetic Algorithm • 30 genes by chromosomes • Training use a population of 256 chromosomes • F-Measures is chosen as fitness • The training is done with the 2 first people (12 peoples in the complete sequence) Figure : Chromosome description 14/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Database Example of people extraction results Proposed method VS state of art foreground detection Comparison Precision/Recall Example : Full displacement of one person Database BOSS European project database Sequence recorded inside a train (people moves in front of camera, 4258 frames) Many scientific locks (brightness changes, shadows, scrolling background, noise) Evaluation with F-Measures : F1 = 2 . precision . recall precision + recall precision = |well detected foreground pixels| | foreground pixels | recall = | well detected foreground pixels | | detected pixels | 15/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Database Example of people extraction results Proposed method VS state of art foreground detection Comparison Precision/Recall Example : Full displacement of one person Example of people extraction results Original image Ground truth State of the art Our method Figure : People extraction results on the BOSS project database (the best method of the state of the art is determined by the genetical algorithm VS the proposed method) 16/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Database Example of people extraction results Proposed method VS state of art foreground detection Comparison Precision/Recall Example : Full displacement of one person Proposed method VS state of art foreground detection F-Measures corresponds to an average on all the displacements of several persons The best foreground detection is used (determined by the genetic algorithm) Score F-Measures means : Proposed method = 0.82 Foreground detection = 0.59 Observations Results are alway better with our proposed method In difficult cases, the proposed method increases widely the results 17/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Database Example of people extraction results Proposed method VS state of art foreground detection Comparison Precision/Recall Example : Full displacement of one person Comparison Precision/Recall Figure : People extraction results (with the state of art) Figure : People extraction results (with the proposed method) Observations Foreground detection based on state of art get poor recall results Proposed method get very good precision and recall scores Proposed method get small gap between precision and recall scores 18/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Database Example of people extraction results Proposed method VS state of art foreground detection Comparison Precision/Recall Example : Full displacement of one person Example : Full displacement of one person Proposed method is always better than state of art (except the 43th frame) t − 1 43th frame (t) t + 1 Figure : illustration of the bad result (the 43th frame) The bad detection will be corrected in future works by integrated tracking method Figure : F-Measure of one people for his displacement in front of the camera 19/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Conclusion • The best foreground detection methods of the literature are not enough robust and accurate to obtain good people extraction in complex transport environment • We propose a new method of people extraction based on superpixel segmentation coupled with graph cut clustering initialized by the foreground detection • We propose to use robust pre-processing (based on filters and colorimetric invariants) and post-treatment steps (shadow removal and morphological filtering) in order to increase the quality of the foreground detection. • The best settings of the proposed method are determined by a genetic algorithm and evaluated with the F-Measure criteria = 0.82 VS the state of art = 0.59 Future works • Introduce a temporal information in the graph cut clustering • Develop the people re-identification step 20/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method 

Introduction Proposed approach Experimental results Conclusion Thank you for your attention 21/21 C. Coniglio 1, C. Meurie 1, O. Lezoray 2, M. Berbineau 1 1 Univ Lille Nord de France, F-59000 Lille, IFSTTAR, LEOST A genetically graph-based people extraction method