ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation

We study image-conditioned video generation with joint control over the character’s motion and the camera trajectory, using a pretrained conditioned video diffusion backbone. We assume a sequence length $T$ and three inputs: a reference image $I_{\mathrm{ref}}\in\mathbb{R}^{H\times W\times 3}$ defining the target character identity and the environment appearance, an acting video $V_{\mathrm{act}}=\{I^{\mathrm{act}}_{\tau}\}_{\tau=1}^{T}$ providing the target performance, and a target per-frame camera trajectory $\mathcal{C}=\{(K_{\tau},R_{\tau},\mathbf{t}_{\tau})\}_{\tau=1}^{T}$, with intrinsics $K_{\tau}\in\mathbb{R}^{3\times 3}$ and extrinsics $(R_{\tau},\mathbf{t}_{\tau})\in SO(3)\times\mathbb{R}^{3}$. Let $V=\{I_{\tau}\}_{\tau=1}^{T}$ be the generated video. Our goal is to generate $V$ such that the identity and appearance match $I_{\mathrm{ref}}$, the motion in the output video matches the performance in $V_{\mathrm{act}}$, and finally, such that each frame’s viewpoint is in accordance with the desired camera parameters $\mathcal{C}$.

We formalize the pretrained VACE (Jiang et al., 2025) model as a mapping $f:\mathfrak{C}\rightarrow\mathcal{V}$ that transforms a set of dense video-aligned signals $C$ into a video sequence $V$. VACE is trained on a video dataset combining various video-aligned conditions such as: depth, optical flow, character poses, and activation masks. We restrict the conditioning input to the tuple $C=(I_{\mathrm{ref}},c_{\mathrm{pose+depth}},c_{\mathrm{pose}})$. Our objective is to synthesize novel, geometrically consistent dense conditions $c_{\mathrm{pose}}$ and $c_{\mathrm{depth}}$ derived jointly from the acting video $V_{\mathrm{act}}$ and the target camera $\mathcal{C}$, such that the generated video $V=f(C)$ satisfies the desired motion and trajectory constraints. ActCam is a pure inference-time method: we keep the backbone fixed and design conditioning signals that jointly encode motion and camera.

Let $z_{t}$ denote the latent video state at continuous time $t\in[0,1]$ and let $C$ denote the conditioning input. We model the generative dynamics as a flow governed by an ordinary differential equation (ODE). The backbone network $v_{\theta}$ approximates the instantaneous velocity field, defining the temporal evolution of the latent state as

The target latent $z_{1}$ is obtained by integrating this flow over time from $z_{0}\sim\mathcal{N}(0,1)$ and is subsequently decoded into the video $V$.

To jointly ensure camera control and pose reproduction, the conditioning must provide spatially grounded, per-frame cues in the target camera view: it must both enforce the intended motion and stabilize the scene layout under viewpoint changes. To enforce camera control, we preferred a dense per frame aligned depth signal making sure the model understands the 3D geometry and camera motion rather than numerical camera control using Plücker rays since dense superiority has been proven in (Ren et al., 2025). In practice, simply combining off-the-shelf controls is brittle: view-locked motion cues (e.g., 2D keypoints) become ambiguous under camera motion, and conditioning on depth estimated from $I_{\mathrm{ref}}$ entangles the static reference character with the dynamic character we want to animate, which can cause duplicated characters, freezing, or violations of motion/camera constraints. ActCam addresses this by constructing camera-aligned pose/depth conditions and by using a two-phase conditioning schedule (Sections 3.2–3.3).

Our goal is to provide the diffusion model with a target-view-aligned condition that jointly encodes character motion and camera motion. To this end, we leverage depth as a geometric prior: once rasterized under the target camera trajectory, the depth signal implicitly conveys viewpoint changes as apparent background motion. However, naively using the reference image depth introduces a static character that conflicts with the dynamic pose signal (see Figure 2). We address this by inpainting the character out of the reference image and estimating a background-only depth map $\mathcal{D}_{\text{bg}}$. ActCam then constructs a unified 3D scene from $\mathcal{D}_{\text{bg}}$ and the recovered poses, from which we rasterize two control signals under the target camera: a depth+pose condition and a pose-only condition obtained by omitting the depth.

In order to provide a 3D anchor to VACE, we construct a 3D background environment which we argue, once rasterized, will provide enough information to the model to effectively encode the camera motion as an equivalent background motion, and disentangle the character motion from camera motion, reducing conflicts between the two signals. We estimate a depth map $D_{\mathrm{ref}}$ from $I_{\mathrm{ref}}$ using a monocular depth estimator (MoGe (Wang et al., 2025b)). This leads to the creation of a 3D mesh that can be rendered from different viewpoints. Mesh-based rendering provides stronger geometric consistency than point clouds, which tend to exhibit sparsity and visual artifacts under camera motion.

Nonetheless, solely relying on the reference image depth proves insufficient. Indeed, the presence of a static character in the 3D mesh provides a conflicting signal when rendered with a dynamic pose control signal, as in Figure 2. To tackle this, we propose to inpaint the character out of the reference image $I_{\mathrm{ref}}$ to extract a background-only depth map $\mathcal{D}_{\text{bg}}$, yielding a background 3D mesh, suppressing depth-static character related issues, such as character duplication in the output video.

Also, since VACE relies on dense conditioning, enforcing strict pixel correspondence between the control signals and the generated video is crucial. A key challenge is the alignment of the dynamic actor pose with the background 3D scene geometry. In practice, the reference depth map $\mathcal{D}_{\mathrm{ref}}$ and the background depth map $\mathcal{D}_{\mathrm{bg}}$ are estimated in two independent passes, leading to inconsistencies in 3D space. To resolve this discrepancy, we estimate a geometric alignment allowing the character to be correctly registered within the background 3D mesh. We refer to this stage as the scene transfer, described in the next paragraph, and visible in Figure 2.

Let $\mathcal{M}\subset\llbracket 1,H\rrbracket\times\llbracket 1,W\rrbracket$ denote the binary character segmentation mask in $\mathcal{I}_{\mathrm{ref}}$, obtained using (Liu et al., 2024). Our goal is to find the new position of the reference character when transferred into the 3D background mesh, while preserving pixel reprojection with $\mathcal{I}_{\mathrm{ref}}$; therefore, only the depth of the character points is adjusted, since its scale and image plane coordinates will be imposed. We exploit the set of non-inpainted pixels $(u,v)\notin\mathcal{M}$, for which reliable correspondences exist between the two depth maps. Let $\mathbf{x}^{\mathrm{ref}}_{u,v}\in\mathbb{R}^{3}$ and $\mathbf{x}^{\mathrm{bg}}_{u,v}\in\mathbb{R}^{3}$ denote the 3D points reconstructed from $\mathcal{D}_{\mathrm{ref}}$ and $\mathcal{D}_{\mathrm{bg}}$, respectively. To emphasize constraints near the character boundary, we assign each an importance weight: not all environment points contribute equally to the scene transfer, as points closer to the character are more informative for rendering correct scene interactions. We assign higher weight to closer points:

where $\mathrm{dist}(\cdot,\mathcal{M})$ denotes the Euclidean distance in the image plane to the closest pixel in the mask $\mathcal{M}$. We then compute the weighted centroids of the character relative to each set of points in $\mathcal{D}_{\mathrm{ref}}$ and $\mathcal{D}_{\mathrm{bg}}$ respectively:

Assuming that the relative position of the character with respect to these centroids is preserved up to a global depth scaling, we align the character by applying an affine transformation along the depth axis. For any character point with reference depth $z^{\mathrm{char}}_{\mathrm{ref}}$, the aligned depth in the background coordinate system is given by

where $p^{z}_{\mathrm{ref}}$ and $p^{z}_{\mathrm{bg}}$ denote the $z$-coordinates of $\mathbf{p}_{\mathrm{ref}}$ and $\mathbf{p}_{\mathrm{bg}}$, respectively. This transformation jointly accounts for scale and translation mismatches, enabling consistent addition of the character points to the background 3D scene, while effectively taking character-environment proximities (e.g. contacts) into account via the importance weighting.

We recover a motion sequence of 3D humans from $V_{\mathrm{act}}$ using a monocular 3D human motion estimator (GVHMR (Chu and others, 2024)). We denote the recovered articulated state at frame $\tau$ as $\mathcal{S}_{\tau}$ (e.g., SMPL parameters and root pose). Unlike 2D keypoints, $\{\mathcal{S}_{\tau}\}$ reduces depth ambiguity and provides a stable motion signal under viewpoint changes.

As in (Cao et al., 2025), we align the dynamic poses $\mathcal{S}$ with the actual replaced character in the assembled background 3D scene (see Figure 2). To that extent, we adopt the same approach as in (Cao et al., 2025) using a least squares estimation based on rigid transformation at time $\tau=0$ between the $\mathcal{S}_{0}$ and the extracted keypoints from $I_{\mathrm{ref}}$. This method (Umeyama, 1991) provides a rotation matrix $R\in SO(3)$, a translation $\hat{t}\in\mathbb{R}^{3}$ and a scale $s$ that we will apply to the pose sequence as follows:

This new sequence $\hat{\mathcal{S}}_{\tau}$ is the one being rendered.

VACE control signal is a dense, image-like representation that can be directly processed by standard video encoders. We thus directly rasterize both the pose and the depth+pose control signals as videos. We denote $\mathcal{R}^{\mathcal{C}}$ the rasterization operator under the camera $\mathcal{C}$. For the pose, we rasterize the animated skeleton $\hat{\mathcal{S}}_{\tau}$ following the standard openpose representation in other works (see (Jiang et al., 2025; Zhang et al., 2025a)) encoding 2D joint locations and limb connectivity. This yields a pose control video $c_{\mathrm{pose}}=\mathcal{R}^{\mathcal{C}}(\hat{\mathcal{S}}_{\tau})\in\mathbb{R}^{T\times H\times W\times 3}$, rendered on a black background, following the target camera viewpoint (see Figure 2). For the depth+pose control signal, we simply render the background 3D mesh built from $\mathcal{D}_{\mathrm{bg}}$ using min-max depth normalized grayscale color values, following the format that VACE uses at train time. To obtain actual depth+pose joint signal, we finally superimpose the previously rendered pose signal with the background depth rasterization to obtain $c_{\mathrm{pose+depth}}=\mathcal{R}^{\mathcal{C}}(\hat{\mathcal{S}}_{\tau},\mathcal{D}_{\mathrm{bg}})\in\mathbb{R}^{T\times H\times W\times 3}$.

For moving camera, we design an evaluation inspired by Uni3C (Cao et al., 2025). We select 4 camera presets with common cinematic motions and evaluate on 100 reference clips from RealisDance-Val (Zhou et al., 2025b) per preset (total 4$\times$100 tests). For each clip, we sample a reference $I_{\mathrm{ref}}$, extract an acting-motion signal from the original clip, and generate a new video under one of the 4 camera presets. For baselines, we input the same setup, meaning same $I_{\mathrm{ref}}$, extracted motion, camera preset. For static cameras, we use RealisDance-Val (Zhou et al., 2025b) under fixed viewpoint.

For moving camera, we compare with Uni3C (Cao et al., 2025), the most similar method in literature. Both ActCam and Uni3C are based on the same Wan 2.1 14B backbone, ensuring a fair comparison. We also test ActCam in a static camera setup by comparing against strong motion control methods without explicit camera control: Moore-AnimateAnyone (Hu et al., 2024), HumanVid (Wang et al., 2024), MimicMotion (Zhang et al., 2024), Animate-X (Tan et al., 2024), Hyper-Motion (Xu et al., 2025b), UniAnimate-DiT (Wang et al., 2025c), VACE (Jiang et al., 2025), Wan-Animate (Cheng et al., 2025), and SteadyDancer (Zhang et al., 2025a). We intentionally do not compare to camera-control-only methods, as our goal is joint control with motion.

We follow Uni3C (Cao et al., 2025) for the evaluation protocol. First, we use VBench (Huang et al., 2024a) for evaluating the visual quality of the generated videos. We evaluate Subject Consistency (SC), Background Consistency (BC), Appearance Fidelity (AF), Imaging Quality (IQ), Temporal Consistency (TC) and Motion Smoothness (MS), all higher is better. This quantifies quality-oriented generation capabilities. We also calculate the Mean Per Joint Error (MPJPE) between the estimated 3D humans in the acting video and in the generated one, to assess the quality of the joint camera and motion control. We evaluate the Sampson Error (SE) (Sampson, 1982) for geometric consistency in presence of moving camera. We also report the 3D Consistency (3D-C) and Object Control (OC) scores from WorldScore (Chen et al., 2025): 3D-C measures multi-view coherence of the generated scene, while OC quantifies the fidelity of object appearance across frames. In the comparison against methods with no camera control, we evaluate only motion quality with VBench metrics.

We vary the number of initial diffusion steps conditioned on both pose and depth ($N_{D}$) and observe a trade-off. This is reflected in VBench metrics reported in Figure 4. We select a canonical $N_{D}$ that balances these effects; unless stated otherwise, we use $N_{D}=0.2N$ (e.g., $N_{D}{=}2$ when $N{=}10$). From our evaluation, introducing $N_{D}$ denoising iterations with depth conditioning improves environment/camera stability, improving VBench-based evaluations. However, setting $N_{D}$ too high can over-constrain late-stage refinement, as we show in Figure 5. In there we set $N_{D}=1$, resulting in guidance on depth for all diffusion steps. As visible, in presence of dynamic elements or interacting objects, the rigid depth conditioning prevents motion, limiting the realism of the output scene. On the contrary, not benefiting from depth conditioning ($N_{D}=0$) results in ambiguities between character and camera motion, as we demonstrate in Figure 6. There, providing only pose information results in a moving character on a fixed background, failing to capture adequately the camera motion.

As described in Sec. 3.2, we remove the static reference character from the reconstructed scene depth before inserting the animated character. Without this step, the static geometry of the reference character interferes with the dynamic motion conditioning in the depth map, often leading to duplicated characters. Figure 7 shows a representative example: the static character imprint in the depth signal interferes with the animated actor, yielding duplication and inconsistent compositing. This proves the importance of our design decision based on character inpainting.

In Section 3.2, we describe how we align the composed character depth with the rendered environment depth to stabilize occlusions and prevent depth-inconsistent decomposition under difficult motions or camera changes. Without alignment, reusing the first-frame depth map can produce implausible occlusions and unstable layout when the scene contains strong depth variation or large viewpoint change. Figure 8 illustrates that depth alignment improves occlusion ordering and reduces layout/tearing artifacts around the actor during strong viewpoint changes. Our importance weighting produces depth-faithful placements, as shown in Figure 8.