Image / video

Local video generation: from text or an image to a clip

I made a photo that I liked, but as it starts to move, the shape of the face or product changes. Local video creation isn't just about creating a lot of frames. Since you'll need to maintain the same scene across multiple moments, it's a good idea to start by watching one short cut through to the end.

Decide on a cut of a few seconds

For example, let's say you want to create a scene where the camera moves a little closer to a product photo. At first, just check that one action. If you require product rotation, background changes, and large camera movements all at once, it's difficult to know under what conditions the form will break down.

The path to create a new scene with only text is different from the path to using an image as a starting condition. If you already have a well-selected photo, you can start by looking at models that allow image input. The presence of input photos does not guarantee the shape of all frames.

Storyboard illustration of a still photo leading to multiple video frames with different camera movements and subject movements
When creating a video from an image, failures can be reduced by fixing the starting frame and determining the movement, camera, and video length in order.

Separate the clip's running time from its generation time

This means that the 5-second resulting video will be played for 5 seconds after completion. It is different from the time it took to create it. As the number of frames, resolution, model and steps change, the creation time also changes.

For example, if you play 5 seconds at 24fps, you will see 120 frames. However, the time axis that the model processes internally depends on the workflow, such as compression or interpolation. The cost of video creation cannot be simply calculated based on the time it takes to create 120 independent images.

An illustration showing how the number of frames to be processed and the waiting time increase as the resolution and length of the video increase.
When creating a video, not only the resolution but also the number of frames increases the cost, so it is realistic to check the movement in 5 seconds and then increase the length and quality.

Movement first, large resolution second.

See if your subject holds up to the end at the short lengths and small resolutions supported. If the first scene is good and the shape changes midway through, re-creating it at larger resolution may not be the solution. This gives us a reason to reduce motion or test different input conditions.

Once you find the movement you want, change the resolution and length. Even then, there is no guarantee that the composition and movement will be the same, so play it again to check. By saving the seed, prompt, model, and settings together, you can look back and see which modifications were helpful.

A diagram comparing the range of video workloads from a single 16GB graphics card to 24GB, 48GB and multi-GPU workstations.
The possible tasks vary, such as lightweight models and offloading at 16GB, 720p routes at 24GB, and audio and large video models at larger memory.

Sound and editing are also in the process of completion.

Not all video models produce audio together. If the route is supported, check separately whether the sound matches the movement and whether unwanted voices are included. There is also a way to add the necessary sounds to the silent result in post-production.

If you made several short cuts, you can stitch them together in an editing program and match the colors and sounds. It is easier to divide the problem into the goal of creating a few usable seconds rather than the goal of obtaining a long finished video with one click of the create button.

Enough GPU memory does not make every video model runnable

Even in the same VRAM, the execution range varies depending on the model file, precision, resolution, number of frames, and offload. If you use two GPUs, your workflow must support that split. Don't calculate execution time based on memory sum alone.

You need to think not only about the time it takes to make a piece, but also the number of times you make it again. The amount of time you can wait will vary if you make multiple cuts every day and if you occasionally move personal photos. This difference creates a reason to spend extra money on equipment.

Distinguish between site experience and actual creation

The site plays the prepared video sample after an estimated amount of time. The equipment you choose does not currently produce the video or interpret the prompts in real time. This is an experience comparing the atmosphere between devices.

Verify results of the same length and resolution with the actual workflow of your target model before making a purchase. If even a short cut maintains its shape until the end and is worth the wait to make again, that composition can be the starting point for the work you are trying to do.