How AI could change the fabric of architectural visualization

Lon: AI is evolving and it’s evolving fast. It's already had an impact on tools we use for architectural visualization like render denoisers. CGarchitect even uses artificial intelligence to analyze and promote the most eye-catching images found in the 1000+ submissions we get every month. But despite AI's benefits, it's also a polarizing topic. While some see the potential of AI-based tools immediately, others fear they will eventually replace skilled artists and designers. But like any new technology, the real story is always evolving, and these breakthroughs often open doors and workflows we never imagined.

Over the past few months, you've probably seen a lot of AI-generated artwork in the news and on social media. Maybe you’ve even experimented with it yourself. I have, and it’s been fascinating. Among the works I found particularly interesting was a series of studies on fabric architecture by Factory Fifteen’s Paul Nicholls. As a company willing to dive in and explore uncharted territories, Factory Fifteen has always been at the frontier of where art and technology intersect. To find out where artificial intelligence might be headed for architectural visualization, I invited Paul to share his thoughts and experiments with us.

Paul: First, I’d like to thank Lon and CGarchitect for inviting me to write about the use of AI in design and image-making. CGarchitect has been a great partner to us over the years, and we love any chance to collaborate.

In case you don't know me or our studio, I'm Paul Nicholls, creative director at Factory Fifteen. We are a directing collective and a design and animation studio specializing in architectural visualization, television, and film. Architecture and space are at the heart of the work we do for a wide range of clients.

Experimenting with artificial intelligence
The topic of AI in architectural visualization is complicated. It's experimental and new, and the landscape is constantly changing. So far, we’ve dabbled with several different AI technologies, including Disco Diffusion, NVIDIA Canvas, DALL-E 2, and Midjourney. This article focuses on a study I created in Midjourney and what AI tools like these might mean for our industry.

From the first time I tried Midjourney, I was hooked. The last time I experienced this kind of tech magic was in 2016, when I discovered the virtual reality paint application Tilt Brush. I’ve been impressed with Midjourney’s speed, process, and aesthetic quality of the results.

Midjourney's AI has been trained using over a billion sample images, which is more than any of the other tools I tested, and the results were more varied and less repetitive. It also appears to have a much larger photographic library. For example, you can get quite accurate results when you reference real places, and overall, the results are quite photorealistic. (Midjourney also excels at creating painterly results.) Most other AI generators produced results that looked like acid trips, but the results that Midjourney generates are much more useful.

To use Midjourney, you type a text prompt directly in Discord. As a massive gamer, I already use Discord daily, so the logic and interface feel familiar to me. Amazingly, when you type a prompt, “/imagine fabric building,” it generates four thumbnail options in about 15 seconds. Then you can choose to upscale any of the results to 2K, which adds a lot more detail to the image in about 15 more seconds. The process is way faster than any other AI technology I’ve used.

For example, here’s a look at the results for the prompt: /imagine a large space made of fabric and mist.

The engine creates four images, which very often can be quite different from one another. If you don’t like any of the thumbnails it creates, you can simply re-roll the dice to generate a new batch of thumbnails from the same prompt. If you like a particular thumbnail, you can have Midjourney refine it to create a new set of four based on the one you’ve selected. You continue this process of refining and adjusting until you’re happy with the result.

Painting with your mind
I've always been inspired by the fabric-wrapped installations of Christo and Jeanne-Claude and by the Diller & Scofidio Blur project, which questions thresholds of space through mist. This was my starting point for this study. It was my goal to create some kind of large-scale fabric architecture, something impossible but still somewhat tangible.

By this point, I had already done a few studies and engaged in the Discord community. I recognized what prompts worked for other people and began making my own database. You can achieve certain effects by using the right prompts, like ‘line art’, ‘interior’, ‘large-scale’, ‘exteriors’, and ‘aerial shot’. I could refer to this list later as needed to speed up the exploratory process.

In short, the iterative process of “painting with your mind” using words or even images as prompts is incredibly enjoyable. The engine is very responsive, and the results are generated quickly. Adding specific prompts like lasers, sails, and hangar to the mix helped me refine the results. I tried to retain some control over the randomness by only adding or changing one part of the prompt at a time.

Here's my final prompt for this series: large canopy sails with groups of people dancing, fabric, zaha hadid, rave, stage, concert, huge speakers, lasers, wasteland, lights, night, moonlit, fog, minimal, cinematic shot, high details, hyper quality, 8K, photography, dramatic lighting, festival crowd --ar 1:1

I also changed the aspect ratio from square to landscape to portrait by defining it with prompts like ‘--ar 1:1’ or ‘--ar 1:3’.

Most of the images were generated using this prompt or slight variations of it. Then I generated the final images using the images themselves as prompts. It can become a bit of a self-referential rabbit hole.

Finding the right balance
If you're creative, you can easily get lost in the process. Even though you’re only saving the images that you really like, you might generate hundreds along the way. It gives you a sense of real creation, and I was satisfied with the results because it felt like I was capturing what was in my head. But I've also had times when I didn't get what I was aiming for, and that’s frustrating. There's a tricky balance between control versus random slot machine magic. 

The future of AI
One of the first things I asked the Midjourney developers was if I could use an image to drive the composition. Sadly, this isn't possible yet, but if you could combine the composition tools of NVIDIA Canvas with the output of Midjourney, it would be a perfect solution for me. Then you could define the composition early on and spend more of your time defining the look, lighting, colors, and details. Ideally, you could paint the composition or even input a simple black-and-white drawing or blocking of the space. This could be a great tool for designers, concept artists, and architectural visualization artists.

Imagine a future version of Midjourney where an architect could input a basic screen grab or chalk render of their design and prompt: /imagine daytime, soft shadows, wooden floor, white walls with picture frames, chandelier, potted plants dotted around, 4K … and then the tool generates a version of this for them in five seconds! Sure it might take some iterations, but maybe there could be presets based on references of the architect’s favorite visualization studio, and suddenly, there’s a shift. The in-house design team could use AI like this to iterate themselves, and the relationships between the architectural firms and visualization studios could change as a result. I believe there will always be room for high-end, bespoke creative work in our industry, but AI could change the dynamic.

Using AI for concept design
I’ve already seen some great examples of AI being used by concept designers, using images of their own work as a prompt in Midjourney and then painting over them in Photoshop. For example, they might have AI generate different forms for a spaceship and then refine the result further by hand. It’s incredibly useful.

We can see using AI in this way ourselves. It could be very helpful for creating early design concepts, generating forms, and creating moodboards. As an example, this could be useful for creative jobs like ads and films where we need to provide a specific reference early on. Imagine if you need a picture of a cat wearing a red hat sitting on a box next to a blue car, then DALL-E 2 can already give you that. The power to communicate that concept easily to a client is a huge benefit, and it’s something we hope to experiment with more in the future.

Can AI images be copyrighted?
The question of authorship and ownership is an interesting one. At its core, Midjourney mashes together images that don’t belong to you, yet you can pass off the results as your own. I suppose it’s not that different from photo-bashing an image in Photoshop. Of course, there are other questions, too: Is it right to use an artist or designer in your prompt? Or ‘in the style of (insert famous artist)’? Personally, I prefer the default style Midjourney creates without tagging an artist’s style. There are a number of Discord threads dedicated to these types of copyright questions, and it will be interesting to see where it all leads. 

Looking forward to the future
I should mention that Midjourney's AI engine has already been updated since I created these studies a few weeks ago, resulting in better quality, more resolved images. It’s an exciting time and we need to embrace it—be the new pioneers of this AI-fueled creative process.

Thanks again, Lon, for the opportunity to showcase some AI experiments and talk about this fascinating topic with the CGarchitect community.
 
Factory Fifteen is hiring
I'd love to take this opportunity to say that we are actively growing the team and looking for several full time positions with flexible working currently:

Also, we are always looking for talented freelancers, so please contact jobs@factoryfifteen.com with the job title as the subject line.

About Factory Fifteen
Factory Fifteen are BAFTA award winning directors, heading up a boutique CGI and VFX studio.

Co-founded by Kibwe Tavares, Jonathan Gales and Paul Nicholls, their multi-disciplinary talents and distinctive approach to filmmaking has been sought after by high profile brands including Samsung, Channel 4 and The British Film Institute, along with world-leading architects and developers. 

Most recently, Factory Fifteen transported viewers to a magically reimagined Tokyo for the BBC’s coverage of the Olympic Games. They also brought to life the iconic cityscape for the BBC series adaptation of Malorie Black’s seminal piece, Noughts + Crosses and created the mesmerizing visuals for the breathtaking ballet ‘Aisha and Abhaya’ which debuted at the Royal Opera House.