Style Transfer Generative Adversarial Networks take two images and apply the style from one image to the other image. Here are some sample results from here.
For a more technical explanation of how these work, you can refer to the following papers;
Image Style Transfer Using Convolutional Neural Networks
Artistic style transfer for videos
Preserving Color in Neural Artistic Style Transfer
Ever since first seeing this technique I wanted to add it as an image processing option within Visions of Chaos.
If you only want to play around with style transfer or only have a few photos you want to experiment with, then I recommend you use an online service like DeepArt because this can be a tedious process to setup and use on your own PC.
GPU with Cuda support
The methods in this blog will run without a graphics card processor (GPU) but are very slow using only the CPU (ten minutes for a tiny image with few iterations, hours for larger sizes with many iterations).
For fast results you need a Nvidia graphics card that supports Cuda. Check the list here. On board GPUs are not supported. AMD Radeon GPUs are not supported. You need Nvidia.
If you do not have a supported Nvidia graphics card you can continue to get the CPU supported version going if you are very patient and/or a masochist.
Version Numbers Are Important
There are multiple components and steps to get this working. Versions of software is important. The used versions mentioned in this post worked for me. Changing any of the software versions may cause any of the parts to fail. If this works for you, great. If you find another combo of Python, Tensorflow and Cuda/cuDNN works for you, please leave a comment. I had all sorts of hassles installing/uninstalling/testing multiple versions of software until this set worked.
For these steps I made a new C:\STGAN\ folder for all the downloads, so that is what you will see referenced in the steps and screenshots.
Download Python v3.6.4 from here.
Install Python. NOTE: you must check the “Add Python 3.6 to PATH” checkbox on the first Python installer screen.
TensorFlow is a machine learning platform.
To get TensorFlow CPU support in Python type the following inside a command prompt window
pip3 install --no-cache-dir --ignore-installed --upgrade tensorflow
pip3 install --ignore-installed --upgrade scipy
Download opencv_python-3.4.5-cp36-cp36m-win_amd64.whl (make sure that is the version you download). Save it to the C:\STGAN\ folder.
Back in the command prompt, change into the folder the whl was downloaded to and install it
pip3 install --no-cache-dir --ignore-installed --upgrade opencv_python-3.4.5-cp36-cp36m-win_amd64.whl
Now for the actual Python program that handles the style transfer.
Download and extract neural-style-tf (for this example I used the C:\STGAN\neural-style-tf-master\ folder )
Download this model and put it into the extracted neural-style-tf-master directory.
Change into the neural-style-tf-master folder with the command prompt
Now test each of these import lines (from neural-style-tf.py) one at a time to verify everything is OK
To do that just type “python” in your terminal and press enter (restart your terminal first if you haven’t after you finished installing everything). The line should now start with “>>>” instead of the directory. Copy and paste the following commands (you can copy paste them all at once to save typing them one at a time) into the Python prompt.
import tensorflow as tf
import numpy as np
They should come back without errors, ie
Here we go! If you got to here then it is time to do a quick (slow) CPU test run.
Go into a command prompt under the neural_style.py folder and run the following command
python neural_style.py --content_img golden_gate.jpg --style_imgs starry-night.jpg --max_size 1000 --max_iterations 100 --print_iterations 1 --original_colors --device /cpu:0 --verbose
You will see various stats and then after some time (on a not so new PC this took 15 minutes) you will see it finish.
You should then see the output under the C:\STGAN\neural-style-tf-master\image_output\ directory.
If you got to here then it is mostly working. The next step is to get GPU support working so the processing times can be much faster.
NVidia CUDA and cuDNN
Waiting 15 minutes per image really tests the patience. If you have a newer GPU then it can be used to speed up the calculations. Firstly you need to download the various support tools and drivers.
You need to download CUDA 9 + Update(s)
NOTE, you need to install the base version then the updates.
Select the “exe (local)” versions of the files.
NOTE: by default the Nvidia installer wants to install extra drivers etc, you only need the first CUDA related stuff. Uncheck the rest.
Download cuDNN version v7.2.1 from https://developer.nvidia.com/cudnn
You do have to register, but if you do not want to use your real name and email to register, use a fake name and 10 minute mail to get the verification email.
You want “cuDNN v7.2.1 Library for Windows 10” under “Download cuDNN v7.2.1 (August 7, 2018), for CUDA 9.2”
Yes, I know it says for 9.2 and we only downloaded 9.0, but this is the only version I got working.
Extract the zip to a temp folder and then copy the contents to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\ ie;
Copy cudnn64_6.dll into C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\ (which should be in the path now)
Copy cudnn.h into C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include
Copy cudnn.lib into C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\lib\x64
Now you want to remove the CPU tensorflow and install the GPU tensorflow.
Use these two commands (note the ==1.12.0 forces version 1.12.0 of Tensorflow as newer versions did not work for me).
pip3 uninstall tensorflow
pip3 install --no-cache-dir --ignore-installed --upgrade tensorflow-gpu==1.12.0
Re-run the test command (note that it now specifies GPU device to use). Also note that the max_size is small here. Larger sizes need more GPU memory and power and may fail, so best to start with a small sized image as a test.
python neural_style.py --content_img golden_gate.jpg --style_imgs starry-night.jpg --max_size 250 --max_iterations 100 --print_iterations 1 --original_colors --device /gpu:0 --verbose
and you should see it is MUCH faster.
If your GPU is not supported or it does not run, you are stuck with CPU so roll back to CPU support,
pip3 uninstall tensorflow-gpu
pip3 install --no-cache-dir --ignore-installed --upgrade tensorflow
Style Transfer in Visions of Chaos
If you made it this far you can now experiment with style tranfer GANs in Visions of Chaos. I have added some basic wrapper code that executes the python command to apply style transfer to any fractal or other image you can create.
Generate any image, then select Image->Image Processing->Style Transfer.
Start with smaller image sizes to get an idea of how long the process will take on your system before going for larger sized images.
You can also select any external image file to apply the style transfer to. So dig out those cat photos and have fun. Note that if you get tired of the limited style images that come with neural-style-tf you can put any image you like under the styles folder and use those. Grab an image of your favorite artist’s works and experiment.
For some examples I used the following photo of Miss Marple.
And applied some various transfer style images.
MC Escher Plane Filling II
A Mandelbrot fractal
Another Mandelbrot fractal
HR Giger Biomechanical Landscape
Kandinsky Composition VII
Picasso Les Femmes d’Alger
Picasso Seated Nude
Hokusai The Great Wave off Kanagawa
Munch The Scream
Turner The Wreck of a Transport Ship
van Gogh Starry Night
If you get a failed style transfer and an error message, here are a few things to try;
1. Smaller image size. Depending on the RAM in your PC and GPU you may have maxed out.
2. Reboot. Seems to always fix a stubborn error for me. The Cuda and/or cuDNN seem to be the main culprit. They get hung or locked or something and only a reboot will get them working again.