artist: unknown and forgotten

CNN as a Feature Extractor serving the artistic style and optimized harmony

VGG-Net is a convolutional neural network that has a depth of 19 layers. It was built and trained by K. Simonyan and A. Zisserman at the University of Oxford in 2014. It was the first and simplest of many deep ConvNet architectures that followed AlexNet

Back in my previous text, I wrote about Transfer Learning and I have described the VGG16 model and how it was used for classification. Since the VGG-type of the model is considered a convolutional classic regarding the compositions of the layer, groups of convolutions followed by pooling, and with Softmax classifier on top. Similar to the Transfer experiment, the Classifier group of layers will be removed so we have the whole classic convolutional structure, which we will not be training.

Yep, Neural Style Transfer requires no data, no training, no effort. The trick is using the formation of 5 stacks of convolutional layers as a Feature Extractor.

And now the bad news: There are 3 sets of Loss Function variants on relation style-content imagery and they are messy for grasping at once, but clearly, that led to several empirical conclusions. We will get to that, no worry, I just don’t want to be hyping about the method where the solution is a bit blurry for me, and it hides behind the common term: optimization.

Ok. Hope you are teased to find out where the Catch22 leads to…

No, this isn’t sushi with chopsticks served with a classifier. Sure you recognize the classic VGG-Net Extractor.

Regularly we would, as usual, import and load the VGG-Net architecture that is pre-trained on the ImageNet dataset. Note: weights will be set to “imagenet” and the trainable will be set to false. You are not training the network when performing the Neural and Fast Style Transfer: VGG-Net will simply act as a feature extraction unit. Load and freeze:

# load the pre-trained VGG   
vgg = tf.keras.applications.vgg19.VGG19(include_top=False, weights='imagenet')    
# freeze the weights of the model's layers   
vgg.trainable = False

Defining the input and output of the model, nothing unusual: input stays the same as the standard VGG-Net model, and output will be the set of layers that outputs the style and content layer activations.

The question arises: how to be certain which group of layers belongs to the content of an image, and how to distinguish them from the others that dictate the style of an image?

My first serious mental bug with this script. I wasn’t wasting much time on finding the solution because you can’t come up with the conclusion independently. The whole idea of gaining knowledge in Machine Learning and training the networks is to inform yourself and educate while “standing on the shoulder of giants” — some enthusiastic developer, practitioner, engineer, or simply curious giant tried and failed many times so that we can have the privilege to observe from above. Everything is clear from up here.

Oh yeah, the layer question! Sure. I have no idea, had to accept this as a source hint:

You’d have to be satisfied with metaphysical improvisation: If you want to know what differs style from content, you’d have to find out how the CNN “visualizes” the reality.

Sounds pretty Zen to me. Like deal with it.

CNN will learn general low-level features like lines, edges, and basic shapes in the layers that are closer to the input. As a general rule, the top layer has the lowest dimensionality as it approaches the higher level where the lower level features define the style of the image and the higher-level features define the content.

Fine-tuned and pre-trained, served with memories of Imagenet

Neural Style Transfer by definition an optimization technique used to take two images — a content image blended in the style reference image (i.e. artwork by a famous painter) — so the output of an image looks like the content image in the characteristically unique style reference.

Neural style transfer is a method of blending two images and generating a new image from a content image by copying the style of another image, called the style image. This newly created image is often referred to as the stylized image.

Another good thing is that we don’t need lots of training samples. To be more accurate, this technique can be done with just two pairs of images! Isn’t that neat?

Allow me to explain this simple trick before I assert optimization; that is where the true wizardry of NST unquestionably has to be.

I generated the content image as my input image to the VGG extractor just by prompting the Dall-e : [old-timer car in black and white] — to get an image where the shape of an object dominates, with a low RGB value, because I wanted to add the very specific painting technique for further blending.

For style images, I had no problem with choosing because I already generated hundreds of images where I was prompting for Moebius landscapes from another planet. More vivid -the better.

Style image I got by prompting GAN network and ordering specifically Moebius type of shading and coloring. Obviously, I didn’t go for many iterations because I wanted the dreamy output, a bit blurry where the shapes are not dominating, just the not- from- this- world vibe

OK. Style checked. Information can be extracted legitimately from every layer of the network. This diagram is the quickest possible instruction to develop a mental replica of a method in my mind of what’s going on.

Balance is the key to understanding the beauty and symmetry in Loss functions

Arrows on the diagram show how the weights from each stack of layers may be influential on the *Style Loss****.

When initializing generated image from the content image, the model compares the generated image with the content image by calculating a *Content Loss**.

Few important principles for obtaining the harmony

1-Total Loss would be zero if the images were identical. Thus, you don’t want the loss to get too high or you’d lose the valuable features and attributes of the original image. In addition, you want to reduce the style loss because we’d like some of its attributes to be in the generated image. It’s ultimately a balancing game.

2-The style loss is calculated by comparing each layer of the CNNs for both the generated image and the style image.

3-The style loss ends up being an average loss across the multiple layers that are being compared. In early iterations, as the images are very different, the style loss will be very high, but the goal is to reduce it over time as the generated image incorporates more of the style information.

4-You’d want a generated image to keep the overall loss low. You can combine the style loss with the content loss **to get a Total Loss*****

If the average loss across the multiple layers would be an image.

We can use an optimizer to update the generated image to get a newly generated image whose total loss is lower. The aim is to get closer to a merged image that is in between the two input images.

The process can go forever constantly comparing generated image against the style image = and if that is equal to style loss. Avoid this Catch22 by adding the two losses to the total loss, so the optimization should reduce the overall total loss.

“Blended with the Loss, blinded by the lights”

Content image and one style image can take many iterations before an image is stylized effectively and this can cause time loss (no pun intended)

The total loss is given by Ltotal=βLstyle+αLcontent, where β and α are weights given to the content and style features to generate the new image.

Fun Fact Not: VGG 16/VGG19:

• It is very slow to train (the original VGG model was trained on the Nvidia Titan GPU for 2–3 weeks).
• The size of VGG-16 trained ImageNet weights is 528 MB. So, it takes quite a lot of disk space and spare capacity, which makes it inefficient.

Resources: TF DOCs.