Super resolution is the process of using AI/Machine Learning neural networks to increase the resolution of an image. These networks work by learning features of many thousands of images and then applying that knowledge to fill in the information gaps during resizing to a higher resolution. For example, because the network knows what trees ought to look like, it can scale a tree up and paint in missing gaps with leaves, bark or other features it has learned should be there. This knowledge is saved in a file, often called a “model.”
Traditional up-scaling techniques can often lead to “fuzzy” or unclear images. This happens because the algorithm used to resize often interpolates between nearby pixel values. Consider the following example of an image upscaled from 128px to 512px using imagemagick:
Super resolution often offers much better quality. This image has been upscaled 4x from the same 128px image as the above example:
The final images from super resolution are not perfect and often have mistakes. For example, the eye in the super resolution-version has some strangeness. The text in the lower right is also abnormal, but overall, it’s a clearer image.
Benefits to Game Developers
The main benefit to game developers is resource management. For live 3D games, super resolution is used to display high resolution images while only having to rasterize images at a lower resolution. This is lighter on the target hardware. For Ren’Py-like games, game developers can render images at a lower resolution saving time and increasing development speed and then use super resolution to upscale the image with minimal quality loss.
Consider the following scene rendered on a RTX titan (blender cycles engine, optix, 1024 samples, denoise):
This image took 4.5 minutes to render in 1080 resolution. Using super resolution, the image took less than 22 seconds to upscale it to 8k resolution (7680×4320). By contrast, it would have taken over 1hr to render this scene in 8k from blender directly. To do 4k resolution, rendering a 540p image took 1.5 mins, and only 5 seconds to upscale.
How to use Super Resolution
As of the time of this writing, super resolution is not yet available in many of the typical tool kits developers use. To use it, we have to dive into some pretty experimental projects. One such project is called “SRFBN_CVPR19” (Deepglugs’ personal fork of the project). Just like the name and its ability to roll off the tongue, this next part will be quite advanced. For one, I have only tried this on Linux. If you do manage to get this working on Windows, please let me and others know in the comments. I assume knowledge of command line use and python-pip.
First follow the quick-start instructions steps 1 and 2 found in the README.md file. This will get you the code and the model files. For this example, I used the “SRFBN” models (not “SRFBN-S”). The “-S” models are smaller networks and might be quicker with the tradeoff of quality. Feel free to experiment with both.
Second, if you haven’t installed the pre-requisites, do that now. Here’s a command that will help:
python3 -m pip install opencv-python Pillow tqdm torch pandas
With the pre-requisites installed, we can now run the “test.py” script to upscale some images:
python3 test.py -opt SRFBN_x2_BI.json
This command upscales the images from the folder “results/LR/LRBI/SetS/x2” and puts them in “results.” To use your own images, simply edit SRFBN_x2_BI.json and change the “dataroot_LR” attribute to the location of your images and the “output” attribute to the location you want the upscaled images. Pay attention to the “scale” and the “pretrained_path” attributes. The “scale” attribute is the upscaling multiplier. A value of “2” will upscale the width and height of the image by 2. This value corresponds with the model file specified by “pretrained_path.” If you change “scale” to 4, for example, you will want to use the “SRFBN_x4_BI.pth” model file or else you will get an error when you run. You will not be able to use scale values without a corresponding model file (ie, scale 6.)
Bonus Compression Tip
Deepglugs’ personal fork of SRFBN has support for webp image format with lossless compression. To use this, just make sure your source images have the “.webp” file extension and SRFBN will automatically compress the final upscaled image.
A more user friendly way?
RunwayML is a commercial service that also has super resolution support among other types of networks. This is more user friendly than the method documented above, but can cost money if you use RunwayML rendering farms. If you have nvidia/CUDA devices on your machine, you can also use those for free.
Winning
Super resolution is a way of producing good-quality, high-resolution images faster than rendering high-resolution directly and better quality than the traditional methods. It can save you machine time and offer users higher resolution images than what they normally might get (ie 4k, 8k). Super resolution is not perfect, so you may have to look closely at what it produces to make sure there aren’t abnormalities introduced. Users may notice these abnormalities or they may not. One way to minimize this is to use direct-render techniques for still-images and use super resolution for animations where the user is less likely to notice abnormal results in a single frame. Animations can take an incredibly long time to render, so the savings by using super resolution are valuable.
