A manifesto on the future of image coding - JPEG Pleno

A manifesto on the future of image coding - JPEG Pleno
Touradj Ebrahimi
JPEG Convener

Tremendous progress has been achieved in the way consumers and professionals capture, store, distribute, display and ultimately use images. We have been witnessing an ever growing acceleration in creation and usage of images in all sectors, applications, products and services. This widespread and still growing use of images has brought new challenges for which solutions should be found. Among others, image annotation, search and management,  imaging security and in particular privacy, efficient image storage, seamless image communication, new imaging modalities, and enhanced imaging experiences are  just a few examples of challenges to which scientific community, industry, service providers, and entrepreneurs have responded in the past, including continuing improvement of existing solutions and creating new ones.

During 25 years, Joint Picture Experts Group (JPEG) has been an example of such efforts, and has offered image coding standards which can cope with some of the above challenges. This has resulted in a series of successful and widely adopted coding algorithms and file formats such as JPEG and JPEG 2000.

JPEG format is today a synonymous of modern digital imaging, and one of the most popular and widely used standards in the history. Images created in JPEG format now exceeds 1 billion per day in their number, and most of us can count a couple, if not more JPEG engines in devices we regularly use in our daily lives; in our mobile phones, in our computers, in our tablets, and of course in our cameras. JPEG ecosystem is strong and continues an exponential growth for the foreseeable future. A significant number of small and large companies created in the last two decades have been relying on JPEG format, and this trend will likely continue.

A question to ask our selves is: will we continue to have the same relationship to flat snapshots in time (the so-called Kodak moments) we call pictures, or could there be a different and enhanced experience created when capturing and using images, that could go beyond the experience images have been providing us for the last 120 years?

Several researchers, artists, professionals, and entrepreneurs have been asking this same question and attempting to find answers with more or less success. Stereoscopic and multi-view photography, panoramic and 360-degree imaging, image fusion, points cloud plus texture imaging, high dynamic range imaging, integral imaging, light field imaging, and holographic imaging are among exemples of solutions that have been proposed as future of imaging.

The offered experience based on the above solutions obviously depends largely on the state of the maturity of technologies behind them, and their implementation, including the content created by them, the type of usage and relationship they offer to professionals and consumers. It is normal to expect that with enough efforts and resources, as well as enough time, some will have the potential of reaching a level of maturity that can make them good candidates to offer an experience beyond what exists today.

However, the way such solutions are introduced has probably as much importance as the solutions themselves. The JPEG ecosystem is huge, and offers sufficient experience for what it is expected to offer. The legacy that JPEG brings should not be underestimated. In fact, several attempts have been made in the past by companies, as well as by standardisation organisations, including the JPEG committee itself, to replace the JPEG format by an alternative. Success has been limited if not inexistent in many of such attempts.

Introduction of new imaging experiences should therefore be done in a smooth and gradual way without disruption of the existing JPEG ecosystem, but by enhancing it for those wanting it, while still offering the old experience to those who don't. JPEG backward compatible standards are therefore essential in introduction of new imaging standards, and should be given the attention they deserve, when assessing future image coding standards.

Recently, JPEG standardisation committee has created an initiative called JPEG Pleno. “Pleno” is a reference to “Plenoptic” a mathematical representation, which not only provides color information of a specific point in a scene, but also how it changes when observed from different directions and distances. “Pleno” is also the latin word for “complete”, a reference to the vision of the JPEG committee that believes future imaging will provide a more complete description of scenes well beyond what is possible today.

The road-map for JPEG Pleno follows a path that starts in 2015 and will continue beyond 2020, with the objective of making the same type of impact that the original JPEG format has had on today's digital imaging starting from 20 years ago. Several milestones are in work to approach the ultimate image representation in well-thought, precise, and useful steps. Each step could potentially offer an enhanced experience when compared to the previous, immediately ready to be used in applications, with backward compatibility. Backward compatibility could be either at the coding or at the file format level, allowing an old JPEG decoder of 20 years ago to still be able to decode an image, even if that image won’t take full advantage of the intended experience, which will be only offered with a JPEG Pleno decoder. Examples of potential milestones include panoramic and 360-degree images, fused images, and light-field images. Stay tuned as JPEG committee clarifies and shares its plans and achievements for JPEG Pleno in the coming months and if you want to join this effort, do not hesitate to drop me an email to Touradj.Ebrahimi@epfl.ch.