Interactive Streaming of Sequences of High Resolution JPEG2000 Images

This work presents the Continue data-flow control strategy, a JPIP-compliant solution for the interactive streaming of sequences of images that are transmitted over time-varying communication channels. In this context, the random fluctuation of the capacity of the transmission channel over the time forces the clients to prefetch a minimal amount of the code-stream of each image of the beginning of the transmitted sequence before the playback starts,
and the server to decide, in real-time, which amount of the code-stream of each compressed image is going to be transmitted. The estimated channel capacity is performed by clients and the rate-control at the server is straightforward, resulting in a highly scalable image retrieval system.

Barcelona, June 2015.

Transcript

1. Interactive Streaming of Sequences of High Resolution JPEG2000 Images J.J. S´ anchez Hern´ andez1 J.P. Garc´ ıa-Ortiz1 V. Gonz´ alez-Ruiz1 D. M¨ uller2 1Department of Informatics, University of Almer´ ıa, Spain 2European Space Agency, ESTEC, Noordwijk, Netherlands 1/24
2. Outline 1. Introduction. 2. Motivation. 3. Proposal. 4. Evaluation. 5. Conclusions and Future Work. 6. References. 2/24
3. Introduction • There is a need to create large repositories of images. • The ESA/NASA JHelioviewer Project. • SDO/AIA takes images in 10 channels, every 12 seconds. • Image size: 4096 × 4096 pixels. AIA 193 AIA 304 AIA 171 AIA 211 AIA 131 AIA 335 AIA 094 AIA 1600 AIA 1700 AIA 4500 3/24
4. Introduction • Huge amount of data: 1.4 TB/day = 511 TB/year. • JPEG2000 can compress 4K × 4K image to 1 MB. (lossy) • 10 channels at 36 sec cadence: 24 GB/day = 8.8 TB/year. • Can keep comprehensive data set of browse data online. JPIP and JPEG2000 seems to offer the best solution. 4/24
5. ESA JPIP Server Architecture • Hybrid model combining process and thread approaches. • Fault-tolerant and robust approach. App. shared JPEG2000 images Client manager Client manager Index manager File manager Image cache files Configuration (server.cfg) information Father process . . . UNIX domain socket Child process Data-bin server Data-bin server 5/24
6. ESA JPIP Server https://launchpad.net/esajpip 6/24
7. ESA JPIP Server https://tracker.debian.org/pkg/esajpip 7/24
8. JHelioviewer Architecture JHelioviewer Browser Web Service JPEG2000 Metadata catalog data search query metadata image request JPIP data stream and headers event catalog search query event data Event server Event catalog repository interface User Metadata handler Solar event manager Layer manager Image renderer/decoder Local cache image repository JPEG2000 Client cache Parser model JHelioviewer Server 8/24
9. JHelioviewer https://github.com/jhelioviewer 9/24
10. JHelioviewer http://jhelioviewer.org 10/24
11. JHelioviewer Demo 11/24
12. Stop-and-Wait Flow Control ... R1 .s R1 .s data-bins data-bins Time Client JPIP Server R0 .s R0 .e GET image.jp2?...&len=2000... GET image.jp2?...&len=2000... Time-line example of how to retrieve a single WOI using the S&W flow control strategy. 12/24
13. Stop-and-Wait Flow Control Time t0 t1 data-bins data-bins data-bins Client RTT0 RTT1 . . . . . . Server GET context=jpxl<0-14>&len=46557... GET context=jpxl<15-29>&len=58196... GET context=jpxl<150-159>&len=77597... Example of how to retrieve a sequence of 160 WOIs using the S&W flow control strategy, transmitting 15 images in each request. 13/24
14. Continue Flow Control change bandwidth Estimated Time Interaction User Client Server data-bins (20 → 18 FPS) GET context=jpxl<0-159>&pref=mbw=10M&srate=20... 1 second GET ...&pref=mbw=11M... GET ...&pref=srate=18... 1 second An example of three iterations of the Continue flow control algorithm. 14/24
15. Continue flow control (Client) Input: ri−1 .mbw: # Estimated bandwidth (bits) for the prev. request. Ri−1 .len: # Number of bits received in the prev. reply. λ: # Prefetch time selected by the user (5 seconds). λ : # Current buffering size (in seconds). α: # Weight of the current transmission bit-rate measurement (0.1). i−1 : # Average bit-rate estimated error in the previous iteration. Output: ri .mbw: # Estimated bandwidth (bits) for the current request. 1: x ← αRi−1 .len + (1 − α)ri−1 .mbw 2: i ← 1 2 i−1 + |Ri−1 .len−x| max{Ri−1 .len,x} 3: IF (λ < λ) THEN 4: ri .mbw ← x 1 − 0.5(−λ λ + 1) 5: ELSE IF (λ > λ) THEN 6: ri .mbw ← x(1 + i ) 7: ELSE 8: ri .mbw ← x 15/24
16. Continue rate control (Server) Input: ri .srate: # Playback frame-rate at iteration i. ri .mbw: # Channel capacity estimate at iteration i. F: # Number of frames of the video. Output: Ri : # The i-th reply. 1: Ri ← ∅ 2: STATIC f ← 0 3: δ ← ri .mbw ri .srate # Bits/image. 4: DO ri .srate TIMES: 5: DO: 6: θ ← θ ∪ NEXT PACKET OF FRAME f 7: WHILE θ.len < δ: 8: f ← (f + 1) MOD F 9: Ri ← Ri ∪ θ 16/24
17. Evaluation The experiments have been performed in two real network scenarios: • S1 : high bit-rate (12 Mbps average), low-varibility (0.695 Mbps standard deviation), medium RTT (161 ms). • S2 : low bit-rate (0.96 Mbps average), high-variability (0.92 Mbps standard deviation), medium-to-low RTT (120 ms). The video sequences were two linked JPX files: • Sun: 1000 images (4096 × 4096). • Stockholm: 604 images (1280 × 720). 17/24
18. Evaluation Stockholm. WOI: (0, (0, 0), (1280, 720)) Sc.2, λ = 0 s (26.34) λ = 5 s (43.89) Sc.1, λ = 0 s (40.25) Lagrange (46.61) λ = 5 s (39.82) Sc.2, λ = 0 s (39.50) λ = 5 s (50.21) Sc.1, λ = 0 s (49.90) Lagrange (57.03) P = 5 frames/second 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 PSNR [dB] Image Sun. WOI: (2, (0, 0), (1024, 1024)) 20 25 30 35 40 45 50 55 60 0 100 200 300 400 500 600 Image λ = 5 s (26.08) Stockholm. WOI: (0,(0,0),(1280,720)) Sc2, λ = 0 s (24.02) λ = 5 s (36.05) Sc1, λ = 0 s (35.95) Lagrange (36.62) λ = 5 s (36.00) Sc.2, λ = 0 s (36.18) λ = 5 s (45.72) Sc.1, λ = 0 s (45.46) Lagrange (49.06) P = 10 frames/second 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 PSNR [dB] Image Sun. WOI: (2,(0,0),(1024,1024)) 20 25 30 35 40 45 50 55 60 0 100 200 300 400 500 600 Image λ = 5 s (24.05) Stockholm. WOI: (0,(0,0),(1280,720)) Sc.2, λ = 0 s (22.92) λ = 5 s (33.55) Sc.1, λ = 0 s (33.80) Lagrange (34.01) λ = 5 s (34.47) Sc.2, λ = 0 s (34.57) λ = 5 s (43.87) Sc.1, λ = 0 s (43.37) Lagrange (46.79) P = 15 frames/second 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 PSNR [dB] Image Sun. WOI: (2,(0,0),(1024,1024)) 20 25 30 35 40 45 50 55 60 0 100 200 300 400 500 600 Image λ = 5 s (23.01) Stockholm. WOI: (0,(0,0),(1280,720)) Sc2, λ = 0 s (22.64) λ = 5 s (31.93) Sc.1, λ = 0 s (31.76) Lagrange (32.20) λ = 5 s (33.83) Sc.2, λ = 0 s (33.72) λ = 5 s (42.54) Sc.1, λ = 0 s (42.28) Lagrange (45.16) P = 20 frames/second 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 PSNR [dB] Image Sun. WOI: (2,(0,0),(1024,1024)) 20 25 30 35 40 45 50 55 60 0 100 200 300 400 500 600 Image λ = 5 s (22.66) Impact of the prefetch time, λ, on PSNR in Continue. 18/24
19. Evaluation Stockholm (0, (0, 0), (1280, 720)) Continue, 20 FPS (31.93) S&W, 5 FPS (30.41) Continue, 5 FPS (43.89) Lagrange (46.61) S&W, 20 FPS (38.94) Continue, 20 FPS (42.54) S&W, 5 FPS (42.64) Continue, 5 FPS (50.21) Lagrange (57.03) Sc.1 20 25 30 35 40 45 50 55 60 0 200 400 600 800 1000 PSNR [dB] Image Sun (2, (0, 0), (1024, 1024)) 15 20 25 30 35 40 45 50 55 0 100 200 300 400 500 600 Image S&W, 20 FPS (26.71) Stockholm (0, (0, 0), (1280, 720)) Continue, 20 FPS (22.66) S&W, 5 FPS (22.29) Continue, 5 FPS (26.08) Lagrange (28.73) S&W, 20 FPS (29.66) Continue, 20 FPS (33.83) S&W, 5 FPS (36.05) Continue, 5 FPS (39.82) Lagrange (44.66) Sc.2 20 25 30 35 40 45 50 0 200 400 600 800 1000 PSNR [dB] Image Sun (2, (0, 0), (1024, 1024)) 20 22 24 26 28 30 0 100 200 300 400 500 600 Image S&W, 20 FPS (21.76) S&W vs. Continue. 19/24
20. Conclusions and Future Work • The experimental results demonstrate that Continue outperforms, in terms of QoE, other commonly used data-flow control strategies such as Stop-and-Wait. • This increase in QoE is expressed as an increase in the overall PSNR of the image reconstructions and a decrease in the variance of this PSNR during display time. • Continue is computationally scalable since almost all calculations involved in the client/server synchronization process are performed by the clients. These results demonstrate that Continue is a good candidate for being used in highly scalable Motion JPEG2000 streaming systems. • Future work is envisioned to further improve the rate control subsystem using, for example, the ideas proposed by [12] and [14], and to perform the rate allocation on the client side, following, e.g., the technique proposed by [15]. 20/24
21. References 1 D. M¨ uller, B. Fleck, G. Dimitoglou, B. W. Caplins, D. E. Amadigwe, J. P. G. Ortiz, A. A. B. Wamsler, V. K. Hughitt, and J. Ireland, “JHelioviewer: Visualizing large sets of solar images using JPEG 2000,” Computing in Science and Eng., vol. 11, no. 5, pp. 38-47, Sep. 2009. 2 “ESA-NASA Helioviewer Project”, http://wiki.helioviewer.org. 3 “ESA-NASA Helioviewer Project,” http://sohowww.nascom.nasa.gov. 4 W. D. Pesnell, B. J. Thompson, and P. C. Chamberlin, “The Solar Dynamics Observatory (SDO),” Solar Physics, vol. 275, pp. 3–15, Jan. 2012. 5 International Organization for Standardization, “Information Technology - JPEG 2000 Image Coding System - Part 2: Extensions,” 2004. 6 “Kakadu JPEG 2000 SDK”, http://www.kakadusoftware.com. 7 International Organization for Standardization, “Information Technology - JPEG 2000 Image Coding System - Part 1: Core Coding System,” 2004. 8 D. Taubman, “High performance scalable image compression with ebcot”, Multimedia, IEEE Transac- tions on, vol. 9, no. 7, pp. 1158 – 1170, 2002. 9 International Organization for Standardization, “Information Technology - JPEG 2000 Image Coding System - Part 9: Interactivity Tools, APIs and Protocols”, 2005. 10 J. C. Dagher, A. Bilgin, and M. W. Marcellin, “Resource-constrained rate control for Motion JPEG2000”, IEEE Trans. Image Process., vol. 12, no. 12, pp. 1522–1529, 2003. 21/24
22. References 11 F. Aul´ ı-Llin´ as, A. Bilgin, and M. W. Marcellin, “FAST Rate Allocation Through Steepest Descent for JPEG2000 Video Transmission,” IEEE Trans. Image Process., vol. 20, no. 4, pp. 1166–1173, 2011. 12 L. Jim´ enez-Rodr´ ıguez, F. Aul´ ı-Llin´ as, and M. W. Marcellin, “FAST Rate Allocation for JPEG2000 Video Transmission Over Time-Varying Channels,” IEEE Trans. Multimedia, vol. 15, pp. 15–26, 2013. 13 J. Garc´ ıa-Ortiz, C. Mart´ ın, V. Gonz´ alez-Ruiz, J. S´ anchez-Hern´ andez, I. Garc´ ıa, and D. M¨ uller , “Efficient and scalable open-source JPIP server for streaming of large volumes of image data,” in IEEE International Conference on Consumer Electronics, Berlin, September 2011, pp. 380– 384. 14 D. Taubman and R. Rosenbaum, “Rate-distortion optimized interactive browsing of jpeg2000 images,” in Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, vol. 3, Sept 2003, pp. III–765–8 vol.2. 15 F. Aul´ ı-Llin´ as and J. Serra-Sagrista, “JPEG2000 Quality Scalability Without Quality Layers,” IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 7, pp. 923–936, 2008. 16 J. L. Joseph Y. Hui, Ezhan Karasan and J. Zhang, “Client- server synchronization and buffering for variable rate multimedia retrievals,” IEEE Journal on Selected Areas in Communications, vol. 14, pp. 226–237, January 1996. 17 M. Daami and N. D. Georganas, “Client based synchronization control of coded data streams,” IEEE International Conference on Multimedia Computing and Systems, pp. 387–394, June 1997. 22/24
23. References 18 M. Mielke and A. Zhang, “A multi-level buffering and feedback scheme for distributed multimedia presentation systems,” Proceedings. 7th International Conference on Computer Communications and Networks, pp. 219–226, October 1998. 19 “ESA JPIP Server,” http://helioviewer.nascom.nasa.gov. 20 “Sun test image sequence,” http://delphi.nascom.nasa.gov/jp2/AIA/171/2012/04/10/. 21 “Stockholm test image sequence,” ftp://ftp.ldv.e-technik.tu-muenchen.de/pub/testsequences/720p/ 720p5994stockholmter.yuv. 22 J. P. Garc´ ıa-Ortiz, V. Gonz´ alez-Ruiz, I. Garc´ ıa, D. M¨ uller, and G. Dim- itoglou, “An Efficient Prefetching Strategy for Remote Browsing of JPEG 2000 Image Sequences,” ser. Analysis, Retrieval and Delivery of Multimedia Content, Lecture Notes in Electrical Engineering. New York: Springer, 2013, vol. 15, ch. 14, pp. 239–251. 23/24
24. 24/24