Interactive Streaming of High Resolution JPEG2000 Image Sequences J.J. S´ anchez Hern´ andez1 J.P. Garc´ ıa-Ortiz1 Carmelo Maturana-Espinosa1 V. Gonz´ alez-Ruiz1 D. M¨ uller2 1Department of Informatics, University of Almer´ ıa, Spain 2European Space Agency, ESTEC, Noordwijk, Netherlands 1/25 

Outline 1. Introduction. 2. Motivation. 3. Proposal. 4. Evaluation. 5. Conclusions and Future Work. 6. References. 2/25 

Introduction • There is a need to create large repositories of images. • The ESA/NASA JHelioviewer Project. • SDO/AIA takes images in 10 channels, every 12 seconds. • Image size: 4096 × 4096 pixels. AIA 193 AIA 304 AIA 171 AIA 211 AIA 131 AIA 335 AIA 094 AIA 1600 AIA 1700 AIA 4500 3/25 

Introduction • Huge amount of data: 1.4 TB/day = 511 TB/year. • JPEG2000 can compress 4K × 4K image to 1 MB. (lossy) • 10 channels at 36 sec cadence: 24 GB/day = 8.8 TB/year. • Can keep comprehensive data set of browse data online. JPIP and JPEG2000 seems to offer the best solution. 4/25 

ESA JPIP Server Architecture • Hybrid model combining process and thread approaches. • Fault-tolerant and robust approach. Index manager File manager Image cache files Configuration (server.cfg) App. shared information Father process . . . UNIX domain socket Child process JPEG2000 images Client manager Client manager Data-bin server Data-bin server 5/25 

ESA JPIP Server https://launchpad.net/esajpip 6/25 

JHelioviewer Architecture JHelioviewer Browser Web Service JPEG2000 Metadata catalog data search query metadata image request JPIP data stream and headers event catalog search query event data Event server Event catalog repository interface User Metadata handler Solar event manager Layer manager Image renderer/decoder Local cache image repository JPEG2000 Client cache Parser model JHelioviewer Server 7/25 

JHelioviewer https://launchpad.net/jhelioviewer 8/25 

JHelioviewer http://jhelioviewer.org 9/25 

JHelioviewer Demo 10/25 

Stop-and-Wait Retrieval Technique ... GET Client GET JPIP Server precinct data-bins precinct data-bins image.jp2?...&len=2000... image.jp2?...&len=2000... How to retrieve a single image 11/25 

Stop-and-Wait Retrieval Technique Client RTT0 RTT1 . . . . . . Server precinct data-bins precinct data-bins precinct data-bins t(len0 ) t(len1 ) GET context=jpxl<0-14>&len0=46557... GET context=jpxl<150-159>&len10=77597... GET context=jpxl<15-29>&len1=58196... How to retrieve an image sequence 12/25 

Motivation This technique, based on a stop-and-wait data-flow control algorithm, has a drawback when we are performing an interactive retrieval of a sequence of images. Let ki the number of images of the i-th request, P the playback image-rate, t(leni ) the reception time of the group of images, RTTi+1 the Round Trip Time at the iteration i + 1. If ki P < t(leni ) + RTTi+1 , (1) the user will experience a pause in the playback at the instant of time: i−1 x=0 t(lenx ) + ki /P. (2) Our proposal guarantees that the frame-rate of the visualization can be maintained even under severe bandwidth variations and reduced buffer sizes. [Video] 13/25 

Pipeline-based Retrieval Technique Client GET GET GET JPIP Server precincts data-bins context=jpxl<0-159>&pref=mbw:10M&srate=20... ...&pref=mbw=11M... ...&pref=srate=18... 14/25 

Estimation of the Bit-rate. Client side Input: s(Ri ): Size of reply Ri t(Ri ): Time neccesary to receive Ri mbwi−1 : Previous bit-rate estimation λ: Buffering time selected by the user (5 seconds) P: Playback image-rate (20 images/second) w: A correction factor (0.5) α: Weight of the current bit-rate measurement (0.1) Output: mbwi : Bit-rate estimation for the i-th request Algorithm 1: N ← P λ # Buffer size in images 2: B ← s(Ri ) t(Ri ) # Bit-rate measurement 3: mbwi ← αB + (1 − α)mbwi−1 # Weighted average bit-rate 4: i ← i−1 + |B−mbwi| max{B,mbwi} 2 # Average bit-rate error 5: λ ← N P # Current buffer time 6: IF (λ < λ) THEN 7: mbwi ← mbwi 1 − w(λ λ + 1) # mbwi−1 was too high 8: ELSE IF (λ > λ) THEN 9: mbwi ← mbwi (1 + j ) # mbwi−1 was too low 15/25 

Bit-rate control. Server side Input: mbwi : Bit-rate estimation at request i sratei : Playback image-rate at request i Output: Ri i-th reply Algorithm 1: δ ← mbwi sratei # Bit-rate/image 2: Ri ← ∅ # The reply 3: FOR EACH image f 4: FOR EACH layer l 5: FOR EACH resolution r 6: FOR EACH component c 7: FOR EACH precinct p 8: Ri ← Ri ∪ data-bin ζf,l,r,c,p 9: IF (s(Ri ) ≥ δ) THEN 11: SEND Ri 10: GOTO step 1 16/25 

Evaluation The experiments have been performed in two real network scenarios: • S1 , Download bandwidth: 8 MB/s. (High bandwidth connection) • S2 , Download bandwidth: 120 KB/s. (Low bandwidth connection) The video sequences were two linked JPX files: • Sun: 1000 images (4096 × 4096). • Stockholm: 604 images (1280 × 720). 17/25 

Evaluation S1 . With precincts [avg: 46.50] S1 . Without precincts [avg: 32.78] S2 . With precincts [avg: 33.17] S2 . Without precincts [avg: 28.42] S1 . With precincts [avg: 41.81] S1 . Without precincts [avg: 38.20] S2 . With precincts [avg: 35.49] S2 . Without precincts [avg: 33.31] Experiment 1. Sun sequence with and without precincts 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 PSNR [dB] Frame Number WOI: (0,(1024,1024),(512,512)) 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 Frame Number WOI: (1,(0,0),(2048,2048)) Experiment 1. Study of the impact of the use of precincts on scenarios S1 and S2 . The image sequence used was Sun sequence using precincts of 128 × 128 and without precincts. Only the pipelined retrieval technique was used with a frame-rate of 10fps at the client side and not using buffering. 18/25 

Evaluation 1 layer [avg: 40.95] 8 layers [avg: 41.50] 16 layers [avg: 42.31] 32 layers [avg: 42.00] 1 layer [avg: 34.20] 8 layers [avg: 39.70] 16 layers [avg: 39.71] 32 layers [avg: 38.56] Experiment 2. Sun sequence. WOI: (2,(0,0),(1024,1024)) 30 32 34 36 38 40 42 44 0 200 400 600 800 1000 PSNR [dB] Frame Number Scenario S1 30 32 34 36 38 40 42 44 0 200 400 600 800 1000 Frame Number Scenario S2 Experiment 2. Study of the impact of the number of the quality layers on scenarios S1 and S2 . The image sequence used was Sun sequence encoded with different number of layers. Only the pipelined retrieval technique was used with a frame-rate of 5fps at the client side and not using buffering. 19/25 

Evaluation Pipelined. 5fps [avg: 40.25] Pipelined. 10fps [avg: 35.95] Pipelined. 15fps [avg: 33.80] Pipelined. 20fps [avg: 31.76] Stop-and-Wait. 5fps [avg: 30.74] Stop-and-Wait. 10fps [avg: 27.13] Stop-and-Wait. 15fps [avg: 26.87] Stop-and-Wait. 20fps [avg: 27.87] Pipelined. 5fps [avg: 26.34] Pipelined. 10fps [avg: 24.02]] Pipelined. 15fps [avg: 22.92] Pipelined. 20fps [avg: 22.64] Stop-and-Wait. 5fps [avg: 23.30] Stop-and-Wait. 10fps [avg: 22.54] Stop-and-Wait. 15fps [avg: 22.40] Stop-and-Wait. 20fps [avg: 22.47] Experiment 3. Stockholm sequence. WOI: (0,(0,0),(1280,720)) 10 15 20 25 30 35 40 0 100 200 300 400 500 600 PSNR [dB] Frame Number Scenario S1 10 15 20 25 30 35 40 0 100 200 300 400 500 600 Frame Number Scenario S2 Experiment 3. A comparative between both versions of the server on scenarios S1 and S2 . The image sequence used was Stockholm sequence without buffering at the client side. WOI located in the coordinates (0, (0, 0), (1280, 720)) with a resolution image of 1280 × 720. The WOI corresponds to a full version of the image in a specific level resolution. 20/25 

Evaluation S1 . λ = 1s [avg: 50.19] λ = 2s [avg: 50.23] λ = 3s [avg: 50.27] λ = 4s [avg: 50.23] λ = 5s [avg: 50.21] S2 . λ = 1s [avg: 39.50] λ = 2s [avg: 39.82] λ = 3s [avg: 39.82] λ = 4s [avg: 39.85] λ = 5s [avg: 39.82] S1 . λ = 1s [avg: 34.50] λ = 2s [avg: 34.68] λ = 3s [avg: 35.73] λ = 4s [avg: 35.67] λ = 5s [avg: 35.81] S2 . λ = 1s [avg: 24.06] λ = 2s [avg: 24.04] λ = 3s [avg: 24.02] λ = 4s [avg: 24.01] λ = 5s [avg: 24.05] Experiment 4. Sun and Stockholm sequences. 10fps 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 PSNR [dB] Frame Number 20 25 30 35 40 45 50 55 60 0 100 200 300 400 500 600 Frame Number Stockholm sequence. WOI: (0,(0,0),(1280,720)) Sun sequence. WOI: (2,(0,0),(1024,1024)) Experiment 4. Study of the impact of the use of a buffer with different sizes. Only the pipelined retrieval technique was used with a frame-rate of 10fps at the client side and a buffer of different sizes. 21/25 

Conclusions and Future Work • Our proposal guarantees that the frame-rate of the visualization can be maintained even under severe bandwidth variations and reduced buffer sizes. • Our proposal is effective and fully compliant with the JPIP standard, because it is based on the use of parameters which are defined to control the data-flow of the transmission of Motion JPEG2000 video. • As future work we could improve the rate allocation method so that the server transmits more bytes for those frames that are most important in the sequence because they are more complicated to compress, or because these frames contain regions with greater motion than other frames of the sequence. 22/25 

References • D. S. Taubman and R. Prandolim. (2003). Architecture, Philosophy and Perfo- mance of JPIP: Internet Protocol Standard for JPEG2000, in International Sympo- sium on Visual Communications and Image Processing, vol. 5150, pp. 649–663. • Devaux, F.-O., Meessen, J., Parisot, C., Delaigle, J.-F., Macq, B. and Vleeschouwer C.D., (2009). Remote Interactive Browsing of Video Surveillance Content Based on JPEG 2000. IEEE Transactionf on Circuits and Systems for Video Technology, 19(8):1143 - 1157. • International Organization for Standardization, (2004). Information Technology - JPEG 2000 Image Coding System - Core Coding System. ISO/IEC 15444-1:2004. • International Organization for Standardization, (2005). Information Technology - JPEG 2000 Image Coding System - Interactivity Tools, APIs and Protocols. ISO/IEC 15444-9:2005. • Garc´ ıa Ortiz J.P., Gonz´ alez Ruiz V., Garc´ ıa I., M¨ uller D. and G. Dimitoglou, (2010). Interactive Browsing of Remote JPEG 2000 Image Sequences. In IEEE International Conference on Pattern Recognition, pages 3179 - 3182, Istambul, Turkey. IEEE press. 23/25 

24/25