All posts by Xuran Ni

Week 5|CG Compositing Fundamentals

NUKEⅡ

Create camera

This week we learn to create camera and import into maya. First thing is to denoise the plate. I use the lake plate here.

With the information from the original plate we can change the settings for the project and the camera.

Before exporting the camera, we need to undistort the lens, because virtual camera has no distortion. We can also keep using this node later to redistort the CG asset before compositing.

However, this week, we do not need to worry too much about the LensDistortion node, just use the node to deal with the Lens Grid card to see the difference.

If using LensDistortion, we may need transform node to cut out the black regions.

Then we just track the camera as normal, put it in the scene and scanlineRender it.

If we know the position where the asset should be put, we can create an axis on the tracking point. (Reference for the asset in the node graph)

We then use WriteGeo node to export .fbx file including camera. We can use ReadGeo node to inport the .fbx file as well for CG compositing in nuke.

node graph

Compositing the Beauty AOV

The RGBA beauty AOV can be split into smaller AOVs where each contains part of the lighting. In compositing, these AOVs can then be individually modified and added together to get the full beauty AOV.

More AOVs give more control in compositing, but also extra work to handle, and they take up more memory and disk space, especially combined with light groups.

View from LayerContactSheet node

We can create material ID pass and turn it into alpha when necessary.

We use shuffle node to swap between RGBA channels and grade node to manipulate each pass.

In this node graph, Diffuse is used as the base, highlight stuffs on the right, others on the left.

We are recommended to start with the diffuse direct and then indirect; when we merge shadows or ambient occlution, we use multiply operation; when merging hightlights, we use screen operation.

Then we merge them and write out.

We will talk about Distortion and z-depth next week. Finally, apply the noise to match the original plate and merge on it.

Gear animation and Clock design

3D CGⅡ

We can use expression editor (under windows>animation editors) or node editor to make linked animation between gears. And we need to freeze transformation of the mesh before we create the expression.

Also, in order to make sure different sizes of gears match with each other, we need to make sure the ratio of the number of gear teeth to the radius of the gear is equivalent.

Therefore, I tried to design and make an animated steampunk style gear clock, I found some online pictures as inspiration.

In this case, I just combine diffrent types of gears together, and made a antique style pendulum and clock face. I also linked four gears with the same modulus to do the rotation.

Clock design

Hierarchy and Solar System

3D CGⅡ

This time we learnt hierarchy rules and then made a simple solar system. In order to make individual movement on different spheres, I set the pivot to the centre of the object itself to do rotation, and set it parent to the locater in the center to do the revolution.

Then I downloaded a milky way image for the background, with the SkyDome light:

https://www.eso.org/public/images/eso0932a/
Solar system rendering

Week 4|3D Tracking_Crypt A01

NUKEⅡ

First we change the settings for the project based on information of the original plate.

Crypt A01 3D Tracking
Point Cloud

It is a more complicated footage, so it is better to divide into two parts(inside room and outside room) to track.

node graph

The mask created here is used in many cases, it will be set to inverted for tracking in different zones when necessary.

node graph zoom in

Week 4|3D Tracking

NUKEⅡ

We need to recreate the camera and match the camera movement for 3D tracking.

Room Tracking

We denoise the plate firstly and use grade node to make it easier for nuke to track every point in the plate. We may need to roto out some elements which are affected by other factors apart from camera movement.

node graph1
node graph2
cameraTracker

Use camera tracker to track the plate. We need to track first and solve next. Adjust some parameters can reject the points with larger errors.

Remember to set ground points beforehand in CameraTracker node.

Use point cloud to accurately position the asset in the 3D scene, for example, the cards.

Point Cloud

Position the cones and cards in the plate. Select particular tracked point in the plate and left click to choose creating the axis, we can link these axis with the suitable geometries such as cones.

Node graph3

Week 4 |THE PAST, PRESENT AND FUTURE OF MOTION CAPTURE

Design for Animation, Narrative Stuctures & Film LanguageⅠ

The release of Cyberpunk 2077 was undoubtedly one of the most eagerly anticipated events in the game industry over the past year. The production process of such an immersive simulation game is inseparable from an advanced technology, motion capture.

Motion capture, also known as mocap, is the procedure of transferring motion from the real world into three dimensional digital systems.[1]Nowadays, mocap has been widespread used in the film and game industry.

In fact, as a technology, mocap has only developed for less than forty years. Nevertheless, the concept of capturing the movement, known as rotoscoping, was proposed as early as the last century, before the birth of 3D computer graphics. Adopted by Disney in the 1930s, many of their celebrated early titles like Snow White and the Seven Dwarves, Cinderella and Alice in Wonderland, were all created via this method.[2] By filming required scenes in live action, the actors’ performance would be recorded. The film then would be projected onto transparent easels for an animator to trace the action frame-by-frame as a reference.

The success of Snow White led to Disney moving ahead with more feature-film productions. However, the rotoscoping, indeed came from its chief competitor, Fleischer Studios. In 1914, Max Fleischer demonstrated his new device, the rotoscope. The original version of the technique was shown through his animated series Out of the Inkwell [3], in which Max played a painter, interacting with his cartoon character, Koko the Clown. In order to make the clown’s actions come alive. Max’s brother Dave filmed a set of actions, becoming the first person to be captured in history.

Constrained to trace every frame accurately in 2-dimensional planes, traditional rotoscoping was actually painstaking and also painful. The prototype of today’s motion capture in 3-dimensional space appeared in the early 1960s, during which an engineer called Lee Harrison III introduced his animated stick figures. The figure composed of bones, joints, eyes, moving lips would be displayed on a Cathode Ray Tube (CRT). This idea evolved into a giant hardware contraption called ANIMAC which could generate real-time animation controlled by an actor wearing body mounted sensors.[4] Although the design behind the scene was fabulous and ahead of time, these glowing stick figures seemed not to be taken notice of.

Unfortunately, lack of funds, limits of capabilities and computing power all led ANIMAC to a business failure. After Harrison, the development of motion capture entered a long period of stagnation. Until 1980, Tom Calvert of Simon Fraser University made a major breakthrough in physical mechanical capture clothing using attached potentiometers. Almost the same time, Ginsberg and Maxwell at MIT, invented the Graphical Marionette. This system was designed to generate animation scripts by enacting the motions, using an early optical motion capture system called Op-Eye which relied on sequenced LEDs. These LEDs located the key joints of the skeleton, playing the role of active markers. Two detectors then sent the 2-D position of each marker to the computer and a 3-D coordinate would be obtained. As an advantage, the location information could be stored for later rendering of a more detailed character.[5]

The emergence of these technologies quickly laid the foundation for the rapid development of motion capture in the future, including today’s performance capture, which refers to recording an actor’s face and body motion simultaneously.[1]

Around 1992, SimGraphics developed a facial tracking system called “face waldo”. One of the most successful applications was Mario’s real-time performance in Nintendo’s trade fairs.[6] Driven by an actor wearing the face waldo, Mario could converse and joke with audiences, which attracted a great deal of media and public attention at that time. However, body and facial animation were still produced separately in a traditional workflow, such a disconnection lasted for a period of time.

At the same time, mocap began to be applied in the game industry. The earliest use went back to 1994, after the huge success of the world’s first 3D fighting game Virtua Fighter in Japan, Sega launched its sequel the next year.[7] In order to produce a realistic visual experience, Sega recruited Japanese martial artists to perform various professional techniques in mocap equipment. This put Virtua Fighter 2 in a leading position of games throughout the 1990s. Even director Mamoru Oshii intended to include an iconic move from Virtua Fighter 2 in his anime film Ghost in the Shell.[8]

On the other hand, the development of the film industry was requiring CGI to be involved, and the key to making CGI characters look lifelike was mocap. In 1999, the movie Star Wars: Episode I – The Phantom Menace made a historical leap and became a milestone in a journey of motion capture towards Hollywood. With the help of mocap, the comic relief character Jar Jar Binks [9] was portrayed as a bumbling and foolish image, thereby also he was recognized as one of the most hated characters in film history.

Now that humanoids can be made with CGI, why not take a fully CGI cast instead of real actors? In 2001, Final Fantasy: The Spirits Within implemented this idea. All the characters in this photorealistic movie are entirely computer-generated with mocap technology.[10] It was noticeable that the heroine, Dr. Aki Ross, was rendered in incredible detail, down to the 60,000 hairs on her head, in a production process which took four years and cost nearly $150m.[11] However, the end product seemed not to be accepted by the audiences. The ambition went flop and sent its studio, Square Pictures, to the wall.

Different from the reason why people hate Jar Jar Binks, the digital humans in Final Fantasy always gave a feeling of creepiness. “Solemnly realist human faces looked strikingly phoney precisely” stated by critic Peter Bradshaw[12], the imperfect visual depictions of human characters were exaggerated on the big screen, exposing the immaturity of mocap technology at that time.

What was the situation of motion capture used on cartoon characters? Coincidentally, the 2004 animated film The Polar Express received a lot of criticism for misuse of mocap.[12] From children to Santa, all characters looked creepy and had a feeling of deadness in their faces, mismatching the film atmosphere.

While the debate over motion capture rumbled on, performance capture had gradually taken the place of traditional mocap. The perfect visual depictions of some hideous creatures such as Gollum in the Lord of the Rings proved that performance capture can be embraced by the mainstream. Because such creepiness became a required element of performance in this case, concealing the technical flaws of the time.

The problems caused by technology must be solved by technology. The filming of blockbuster Avatar (2009) utilized a real-time performance capture system which allowed the director and the actor to preview the final shot[13], significantly increasing the efficiency of filmmaking. The saved time then could be spent on fine-tuning details for final rendering.

In the past few years, cognitive automation has come of age. While the rapidly updating technology results in an explosive growth in computing powers, researchers are more concentrating on machine learning innovations, aiming to bring mocap from large studios to everyday life. The future of mocap is markless.

During the 2018 Game Developers Conference, Cubic Motion presented a digital character “Siren” who was driven by a mocap actor through Unreal Engine in real time.[14] One of the most impressive progress was that her subtle-detailed face performance was totally recreated in an image-based camera tracking system without the use of markers. Such a high-fidelity immediate feedback system demonstration showed potential when putting into practice for emerging cutting-edge industries such as Augmented Reality (AR) and Virtual Reality (VR).

Today we can remove the markers to simulate a digital character, so is it possible to remove the actors one day in the future? KTH Royal Institute of Technology recently developed a speech-driven motion generator by adapting a deep learning-based motion synthesis method called MoGlow.[15] Given only high-level input speech, a rich natural variation of body motion will be obtained on a virtual speaker who may convey a desired character personality or mood. Such control just relies on a large motion capture training dataset. Even though more performance details have not been touched by this method, it should be the holy grail of motion capture.

The pursuit of reality is both the goal and driving force behind motion capture. Motion capture, or performance capture, essentially, is a tool to help match the level of realism of all aspects of the characters. We will definitely walk out of “creepiness”, and perhaps in the near future, such creepiness will develop a new aesthetic that symbolizes the era of innovation.

Bibliography

[1] En.wikipedia.org. 2021. Motion capture. [online] Available at: <https://en.wikipedia.org/wiki/Motion_capture>.

[2] Film, I., 2021. Animation: Rotoscoping. [online] Intofilm.org. Available at: <https://www.intofilm.org/films/filmlist/87>.

[3] En.wikipedia.org. 2021. Out of the Inkwell. [online] Available at: <https://en.wikipedia.org/wiki/Out_of_the_Inkwell> .

[4] Vasulka.org. 2021. [online] Available at: <http://www.vasulka.org/Kitchen/PDF_Eigenwelt/pdf/092-095.pdf> .

[5] Www6.uniovi.es. 2021. A Brief History of Motion Capture for Computer CharacterAnimation. [online] Available at: <http://www6.uniovi.es/hypgraph/animation/character_animation/motion_capture/history1.htm> .

[6] Super Mario Wiki. 2021. Mario in Real Time. [online] Available at: <https://www.mariowiki.com/Mario_in_Real_Time> .

[7] En.wikipedia.org. 2021. Virtua Fighter 2. [online] Available at: <https://en.wikipedia.org/wiki/Virtua_Fighter_2>.

[8] Virtua Fighter dot com. 2021. Virtua Ghost in the Shell. [online] Available at: <https://virtuafighter.com/threads/virtua-ghost-in-the-shell.21086/>.

[9] En.wikipedia.org. 2021. Jar Jar Binks. [online] Available at: <https://en.wikipedia.org/wiki/Jar_Jar_Binks>.

[10] En.wikipedia.org. 2021. Final Fantasy: The Spirits Within. [online] Available at: <https://en.wikipedia.org/wiki/Final_Fantasy:_The_Spirits_Within>.

[11] Empire. 2021. A History Of CGI In The Movies. [online] Available at: <https://www.empireonline.com/movies/features/history-cgi/>.

[12] En.wikipedia.org. 2021. Uncanny valley. [online] Available at: <https://en.wikipedia.org/wiki/Uncanny_valley>.

[13] En.wikipedia.org. 2021. Avatar (2009 film). [online] Available at: <https://en.wikipedia.org/wiki/Avatar_(2009_film)>.

[14] Cdn2.unrealengine.com. 2021. [online] Available at: <https://cdn2.unrealengine.com/Unreal+Engine%2Fperformance-capture-whitepaper%2FLPC_Whitepaper_final-7f4163190d9926a15142eafcca15e8da5f4d0701.pdf>.

[15] Alexanderson, S., Henter, G., Kucherenko, T. and Beskow, J., 2021. Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows. [online] Diglib.eg.org. Available at: <https://diglib.eg.org/handle/10.1111/cgf13946>.