Virtual Production at Cloud901 in the Memphis Central Library PUBLIC LIBRARIES LEADING THE WAY Virtual Production at Cloud901 in the Memphis Central Library David Mason and Alan Ji INFORMATION TECHNOLOGY AND LIBRARIES | MARCH 2023 https://doi.org/10.6017/ital.v42i1.16315 Alan Ji (GeeWhyAlan@outlook.com) is Youth Contributor. David Mason (davidpattenmason@gmail.com) is Program Specialist, Memphis Public Library. © 2023. BACKGROUND Cloud901 Cloud901 is a teen learning center in the Memphis Public Library (https://www.memphislibrary.org/cloud901/). In our space we give youth between the ages of 13–18 exclusive access to all the resources needed to produce anything from short films to visual art to 100+ pound robots. This includes specialty areas designated for subjects such as art, video production, and engineering, each staffed with expert facilitators. As an organization we aim to provide youth with the opportunity to discover many areas of study at only the cost of their time and attention. In addition to providing individual support, Cloud901 (dubbed “The Cloud”) runs its own instructional programs, acts as a venue for outside organizations to host their programs, and generally serves as a place where high schoolers can hang out after school. The nature of what we do places us at the intersection of many different fields and brings a wide variety of youth together. Despite this, The Cloud faces a challenge I see in many other institutions: Computer programming is taught in isolation from other fields such as film, so it is seen as an uncreative pursuit. Filmmaking and programming could each benefit from one another: film, by incorporating code to streamline production; and programming, by gaining a way to creatively collaborate with those seen traditionally as “artists.” In order to explore this connection at Cloud901, I am using my experience in film and programming to develop a “virtual production” initiative in our space. This project serves the purpose of teaching youth how to write programs within Unreal Engine while creating a platform where those interested in the film, programming, music, and visual art aspects of our space can collaborate. Virtual Production Virtual production is a new method of producing special effects for movies and TV that turns the traditional visual effects production pipeline on its head. Until now, the process of modern film making has required all digital compositing and special effects to take place after filming. This system relies on actors, directors, and the pre-production team to imagine what environments might look like and to make decisions about lighting and composition with little input from the visual effects team. If they make misguided decisions, the post-production process becomes significantly more complicated and the result may not measure up to expectations. To minimize this problem, Industrial Light & Magic’s production team for The Mandalorian used a circular room lined with light panels controlled by a real-time 3D rendering engine to light the physical set. Not only does this process allow visual effects artists to make immediate adjustments to the virtual surroundings in response to the director’s feedback during the shoot, but it also produces footage that requires a comparatively small amount of post-production work. mailto:GeeWhyAlan@outlook.com mailto:davidpattenmason@gmail.com INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2023 VIRTUAL PRODUCTION AT CLOUD901 2 MASON AND JI How Can We Do This? The Cloud faces many more constraints than Industrial Light & Magic (ILM). Nevertheless, we found a way to achieve a form of virtual production in our space now, as the tools have become quite accessible compared to even a few years ago. Base Station Nikon d5600, camera that captures our actor. Vive Tracker, used in conjunction with the Base Station to sync the location and rotation of our physical camera with our virtual camera Increasingly Accessible Tools On the software side we rely on Blender for creating virtual environments and we use Unreal Engine 5 for the virtual production logic and rendering. Both software applications are used in professional film production and both are completely free. In fact, the virtual production stage used in The Mandalorian also relies on Unreal Engine. Thanks to the gaming and live streaming industries, the hardware required has become much more affordable. In our production we acquired the latest graphics and VR tracking equipment for a few thousand dollars. Such equipment includes our workstation laptop for running Unreal Engine and the HTC Vive 3.0 tracking system for synchronizing the position and orientation of our physical camera with our virtual camera. For a full breakdown, refer to the documentation on our project’s GitHub page. https://www.vive.com/us/accessory/base-station/ https://www.nikonusa.com/en/nikon-products/product/dslr-cameras/d5600.html https://www.vive.com/us/accessory/tracker3/ https://github.com/dp-mason/Budget-Virtual-Production INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2023 VIRTUAL PRODUCTION AT CLOUD901 3 MASON AND JI OUR APPROACH The process used by ILM begins with the render of the environment in Unreal Engine, which is then sent to the LED-panels surrounding the stage, which light the stage and act as the backdrop that the camera captures. The actors on the stage are working with the same visual backgrounds and lighting that will ultimately be seen by the viewer at home. (Other effects are added in post- production.) This means that the real actors are illuminated by the virtual background via the LED displays, making the addition of purely virtual effects easier and more seamless. Example of the cylindrical computer-generated background with actors in the center, lit with the same illumination as is generated by the virtual set. Image licensed under the Creative Commons Attribution-Share Alike 4.0 International license. Source: https://commons.wikimedia.org/wiki/File:StageCraft_The_Mandalorian.jpg. In contrast, our effects process begins at our physical camera by sending the live video of our actor against a green screen to Unreal Engine, where the video is processed to remove the background. Then, with the help of a virtual object matching the dimensions of our physical green screen, we apply the actor’s image on top of the virtual environment. Instead of projecting our virtual environment to real space through LED panels, we do the inverse: we project our actor into the virtual environment, achieving a similar effect at a fraction of the cost. https://en.wikipedia.org/wiki/en:Creative_Commons https://creativecommons.org/licenses/by-sa/4.0/deed.en https://commons.wikimedia.org/wiki/File:StageCraft_The_Mandalorian.jpg INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2023 VIRTUAL PRODUCTION AT CLOUD901 4 MASON AND JI Render of our virtual production set. This is what our scene looks like in Unreal Engine from a view outside the camera. The image of our actor appears distorted from this angle, because it is projected from the camera to a mesh matching our green screen’s dimensions. This is the view of our virtual scene from the camera. Source: https://www.blendswap.com/blend/30009 https://www.blendswap.com/blend/30009 INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2023 VIRTUAL PRODUCTION AT CLOUD901 5 MASON AND JI WHY DOES THIS PROJECT MATTER? The Philosophy I had been pursuing photography, music, and 3D graphics since long before I joined Cloud901 professionally. This interest actually began in my own teenage years while I was a volunteer at Cloud901. At the time I was producing music, learning more about cameras through event photography, and just beginning to create 3D animations in Blender. When I left for college to study computer science, I still had these interests, but they sometimes took a back seat to my studies. As time went on, though, I began to see how these interests and art in general are not incompatible with computer science. There are many ways to exemplify this compatibility, but there is something special about virtual production in particular. Many students new to computer science get the impression that the field is all about memorizing rigid rules so that you can arrive at a “correct” solution. It is natural to feel that way even after months of classes. While it is true that creating in this medium can feel restricting, it offers many opportunities to make creative contributions. Computer programming—especially in the context of virtual production—is like lute-making. A luthier’s first lutes are likely very ordinary, much of their effort spent attaining the “correct” sound in their instruments. Yet, if they continue to develop their technical ability, to learn the mechanics of producing sound, and to become more comfortable with experimentation, they can influence the future of music in an aesthetic way without necessarily conforming to rigid notions of “correctness.” Such an opportunity presented itself during the rise of opera. Luthiers of the late sixteenth century used their technical knowledge to create the theorbo and other adaptations of the lute that would not be drowned out by an opera singer. There is nothing “correct” or “incorrect” about the theorbo or about virtual production methods, but rather they are subjective visions of how film production and music can adapt to create new experiences. Virtual production is a fascinating use of current technology; it leverages current computing power to innovate in a field not typically associated with computer science. It makes us wonder what other industries could benefit from the application of programming. Using lessons I taught students about Unreal Engine, I encouraged them to embrace topics such as programming, linear algebra, and photography as knowledge essential to crafting their own “instruments.” This began with smaller projects such as a creating a cube that moves up and down according to a sine wave and progressed to building extensions for the system we use to calibrate our camera tracking. Co-author Alan Ji first got into 3D graphics through a programming camp at the Memphis Public Library over the summer. It was there that he met David, and there that he was introduced to Unreal Engine. It’s crazy to think that I went from not knowing anything about Unreal Engine to helping create a Virtual Production set in a few months. Connecting to the Outside The novelty of this project also forged connections with people outside of the library. Spencer Burnham, a project manager critical to some of the biggest XR experiences to date, such as BAFTA award-winning app “Wonderscope,” visited Cloud901 to discuss his career and the potential he sees in learning virtual production to the cohort of youth interested in contributing to our project. While demoing the project at the “CS For All” conference last fall, I spoke to a computer science INFORMATION TECHNOLOGY AND LIBRARIES MARCH 2023 VIRTUAL PRODUCTION AT CLOUD901 6 MASON AND JI teacher at a local high school and later spoke to the students in his after-school program. The debut of Cloud901’s virtual production studio features a performance from Alfred Banks, a regularly touring vocal performer from New Orleans. Next Steps In the short term, we would like to make our system easier to use and share it via an open source license. The current version of our system is hosted and documented on GitHub. In the long term, we plan to take a more formal approach to onboarding youth contributors during the summer and record regular performances. Currently we have onboarded people to the project on an ad-hoc basis, but we plan to organize regular summer programming teaching students interested in film and programming how to use and extend this program. We plan to use this set to host a series of performances featuring established musicians from the area. This will provide our youth contributors with practical experience working on an innovative form of production and opportunities to network with established creatives. EXTERNAL LINKS • GitHub repository for our project • Our Proof of Concept Video • Cloud901 Home page https://github.com/dp-mason/Budget-Virtual-Production https://youtu.be/EvbBT_UzHdE https://youtu.be/EvbBT_UzHdE https://www.memphislibrary.org/cloud901/ Background Cloud901 Virtual Production How Can We Do This? Increasingly Accessible Tools Our Approach Why Does This Project Matter? The Philosophy Connecting to the Outside Next Steps External Links