Ex nihilo nihil fit

Ex nihilo nihil fit is the concept that nothing comes from nothing, which is one of the oldest principles in Western thought, dating back to Parmenides and Lucretius. Lucretius famously wrote a poem De rerum natura which pushes as its base concept that nothing ever comes from nothing. In short, you can always, in principle, trace the thing back to its ingredients.


Something that seems to break this covenant is exactly what a new paper in Nature Communications seems to claim in relation to AI-generated images.


Zheng Dai and David Gifford at MIT in simple terms built diffusion models where you can cleanly switch off the influence of any training image and see what the model would've made without it. Which in turns allows you to see the weight of a model image into the input, which led to an interesting finding, once the training set gets big enough, the results stayed largely the same thing.


TLDR: Remove one image, or even remove an entire artist's catalog, remove every photo of a person, and the output barely moves. The authors call it "attribution decay," and it follows an inverse power law, showing up already at tens of thousands of training images.


Which is pretty insane given there are billions of images in the common models today and will likely be trillions in the next few months.


While some would call this imagination, I would hypothesize with little evidence that maybe humans are very predictable, or there is some sort of cross-artist bleedover due to inspiration or overlapping concepts. In any case, this is pretty cool, and I am keeping an eye out for new research in this area. Give the paper a read, it's worth it!