• Hi guest! As you can see, the new Wizard Forums has been revived, and we are glad to have you visiting our site! However, it would be really helpful, both to you and us, if you registered on our website! Registering allows you to see all posts, and make posts yourself, which would be great if you could share your knowledge and opinions with us! You could also make posts to ask questions!

Neural network style transfer in art

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
341
Reaction score
629
Awards
10
I have been interested in using neural networking to augment art in the past couple years. I've had varying success, but I feel like I'm starting to get a handle on it.

What is neural networking? I'll use IBM's definition:

Neural networks reflect the behavior of the human brain, allowing computer programs to recognize patterns and solve common problems in the fields of AI, machine learning, and deep learning.
The use cases are almost infinite, but it's regularly used for things like image classification, decoding handwritten text (eg, manuscripts), etc.

No doubt most, if not all, of you have heard something about neural networking being used in mainstream media and social networks. Deepfakes use neural networking, and so does Google's Deep Dream (
Please, Log in or Register to view URLs content!
). Deep Dream, which I'll talk about more in the next post, was my introduction to neural networking, and while it's cool, you can only see so many alien dogs before it gets boring.

I started using Caffe, but found it not as well supported and a bit more complicated. Now I use Tensorflow, which was developed by the Google Brain team for Google usage. It's now a free and open source.

I'm currently using Tensorflow in Windows 10 with a Nvidia GTX 1070 video card. I'm using Python 3.7.6 for all scripting. I'd prefer to use Linux, but I didn't feel like dual booting and honestly it's worked well, once you get through the pain of getting everything installed correctly.

So this post is specifically about using Tensorflow to translate the style of one image and apply it to another image. If I'm understanding correctly, the neural network is doing edge detection on the style image, but I'll admit I don't understand everything yet.

I started with the following tutorials, and eventually ended up with a custom written script, which I may post to Github some day. It's kind of messy and in flux right now as I trial to dial in settings to get the results I want.

Please, Log in or Register to view URLs content!
Please, Log in or Register to view URLs content!
Please, Log in or Register to view URLs content!
Please, Log in or Register to view URLs content!

As a bonus, I'm using a super resolution model from Tensorflow hub to size images up without loss of detail. For those using modern versions of Adobe Photoshop or Lightroom, you might have seen Adobe announce that they added a super resolution tool to Photoshop. I'm utilizing something similar, just free and open source. It's pretty amazing. I can render a 500x500 pixel image, send it through super res and end up with a 2000x2000 image, with virtually no loss of quality. Pretty amazing stuff. I based my scripting for super res on the following tutorials.

Please, Log in or Register to view URLs content!
Please, Log in or Register to view URLs content!

For the next post, I'll be using the following 2 images for content and style. Basically the same Paimon sigil, one black on white and the other white on black.

Both are 500x500 images.

I definitely do not have this process completely fleshed out yet, but I think I'm getting some interesting results already.
 

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
341
Reaction score
629
Awards
10
These first two are using the black and white image as the content image (the image to which the style will be applied to), and the white on black image as the style.

This is the raw output from my script.


I added to the script so that I could combine and blend the raw output with the original content image. To my eye, having it blended looks better, but to each their own. This is the raw image blended with the original content image (the black on white sigil).


I wish I could embed more than one image per post, but it is what it is.
 

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
341
Reaction score
629
Awards
10
These 2 images are using the white on black sigil as the content image and the black on white sigil as the style.


And the raw output blended with the original image.


You can get different effects by having the script resize the style image. In these examples, I'm resizing the style image to 100x100 before running through the neural network.

This is all cool, but let's do this to some real images.
 

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
341
Reaction score
629
Awards
10
This is using one of the Mandelbulb3D fractals from my other post. This already has the sigil in the actual image, but let's apply the sigil style also. Both use the original fractal as the content image.

This one is using the white on black sigil.


And this one is using the black on white sigil.

 

SkullTraill

Glorious Light of Knowledge and Power
Staff member
Custodian
Librarian
Joined
Apr 12, 2021
Messages
1,785
Reaction score
14,495
Awards
19
Interesting. I used to work with NN/ML/AI style transfer when it first came out because of how novel and grounbreaking it was at the time (several years ago). But since then, as the novelty wore off, I've been somewhat unimpressed by the technology for still image art. I am particularly turned off by the degradation in quality/resolution, and how it never looks "real". Over the years I've come to recognize that "style transfer look" and it instantly turns me off now from an artistic perspective.

However I will say that thinking of occult applications for it is piqueing my interest again.
 

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
341
Reaction score
629
Awards
10
Interesting. I used to work with NN/ML/AI style transfer when it first came out because of how novel and grounbreaking it was at the time (several years ago). But since then, as the novelty wore off, I've been somewhat unimpressed by the technology for still image art. I am particularly turned off by the degradation in quality/resolution, and how it never looks "real". Over the years I've come to recognize that "style transfer look" and it instantly turns me off now from an artistic perspective.

However I will say that thinking of occult applications for it is piqueing my interest again.
It's definitely not where I want it yet. This type of style transfer is doing edge detection, so the results can look very similar even if you're using different content and style images. I mostly do abstract art, so the unreal look works for me.

The quality degradation is interesting. It can be mitigated by using the right settings (ie, size of the style image vs content image). If one of the images has jagged edges, the result will definitely amplify that. I'm working on adding some Gaussian blur to the style image before applying it to smooth any jagged edges, but not sure how well that will work yet.

There's also a big difference between using Tensorflow hub style transfer vs doing it local network training. The TF hub version is so noisy and you have almost no control over anything. All you can really control is the content and style images, and the sizes of each, but nothing else. Training the network locally tho gives a ton of control, and better quality, but is more time consuming and complicated.

There are exceptions tho. This next image started with a 1080x800 monochrome random noise image and the style is a photo I took years ago at a local park. It was generated using TF hub rather than local training. It's noisy but it the content image is just random noise, so that makes sense and I kind of like how it generated.


The locally trained version of this came out terrible (wrong settings almost definitely).
 

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
341
Reaction score
629
Awards
10
Two more for today.

One of my favorites. Made from 2 of my fractal images (both done in JWildfire) using local training and custom scripting.


Content image in this one was made in JWildfire. The style image was created in Mandelbulb3D using Paimon sigil as a height map.

 

SkullTraill

Glorious Light of Knowledge and Power
Staff member
Custodian
Librarian
Joined
Apr 12, 2021
Messages
1,785
Reaction score
14,495
Awards
19
It's definitely not where I want it yet. This type of style transfer is doing edge detection, so the results can look very similar even if you're using different content and style images. I mostly do abstract art, so the unreal look works for me.

The quality degradation is interesting. It can be mitigated by using the right settings (ie, size of the style image vs content image). If one of the images has jagged edges, the result will definitely amplify that. I'm working on adding some Gaussian blur to the style image before applying it to smooth any jagged edges, but not sure how well that will work yet.

There's also a big difference between using Tensorflow hub style transfer vs doing it local network training. The TF hub version is so noisy and you have almost no control over anything. All you can really control is the content and style images, and the sizes of each, but nothing else. Training the network locally tho gives a ton of control, and better quality, but is more time consuming and complicated.

There are exceptions tho. This next image started with a 1080x800 monochrome random noise image and the style is a photo I took years ago at a local park. It was generated using TF hub rather than local training. It's noisy but it the content image is just random noise, so that makes sense and I kind of like how it generated.

-​

The locally trained version of this came out terrible (wrong settings almost definitely).
Yeah, you're right. It's the edge detection that makes lots of style transfers look the same and repetitive. Cause I have seen and am quite impressed with other types of ML image generation especially those that deal with video such as:
Please, Log in or Register to view URLs content!

But the still image ones (in the mainstream "style transfer" methods) usually look too repetitive (both within a single image, and across multiple images) for me.

Trust me, I've spent a lot of time tweaking and messing around with it locally and just never found results to be satisfactory for a personally reasonable amount of time and effort. I even tried gaussian blurring it and for my STs it looked worse 😂

Two more for today.

One of my favorites. Made from 2 of my fractal images (both done in JWildfire) using local training and custom scripting.


Content image in this one was made in JWildfire. The style image was created in Mandelbulb3D using Paimon sigil as a height map.

I do kinds like these (especially 2nd)

I see panda faces.
Omg yeah, I can't unsee it now 🐼
 
Top