ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024)

2.1 Data Collection

Selection of Data Sources. To serve both video understanding and video generation tasks, we consider the aesthetic quality and content complexity of videos during our collection process. We first consider Panda-70M [12], a high-resolution video dataset sourced from YouTube, featuring clips ranging in one minute. This open-domain source covers diverse areas such as wildlife, cooking, sports, news & TV shows, gaming & 3D rendering. It typically includes complex content and transitions, providing a solid foundation for understanding various real-world scenarios. However, the complexity of these contents and transitions presents a significant challenge for the video generation field. To address this, we also source a large volume of aesthetically appealing videos from some user-uploaded video websites [48, 49, 43]. These videos predominantly consist of scenic views and aesthetically pleasing human activities, involving fewer transitions and simpler events.Finally, we supplement our collection with selected videos from Ego4D [18] and BDD100K [66] to fill the gaps in ego-centric human activities and auto-driving scenarios, ensuring our video sources encompass as many real-world scenes as possible.

Semantic-Based Data Filtering. Although our captioning method can support videos of extended lengths, our collection primarily focuses on videos shorter than two minutes due to the trade-off of the duration and amount of videos. We initially filter out videos from our selected data sources longer than two minutes, leaving videos in two minutes as the candidates. We then introduce a semantic-based data filtering strategy to mitigate content hom*ogeneity among these candidates and maintain diversity in the final video dataset. This approach aims to select videos with significant thematic differences from the pool of candidates to compose our final video collection.Specifically, we first use the Panda-Student [12] model to generate a short caption with one sentence for each candidate video, and then maintain a final pool of video candidates. Whenever a new video $V$ is processed, we encode its corresponding short caption $S$ using the Bert-Base-Uncased [15] language model to obtain the CLS token $P_{n+1}\in {ℝ}^{1\times D}$, which captures high-level semantic expressions. We then calculate the similarity between this CLS token $P_{n+1}$ and the CLS tokens $\{P_1,P_2,\mathrm{\dots },P_n\}$ of videos already in the final candidate pool. A new video will only be added to the pool if its maximum similarity is below a predefined threshold. We provide the pseudo-code in Figure 14.

2.2 Video Processing

Videos are commonly redundant on the temporal dimension, and keyframe sampling is a general idea to represent a video compactly.However, traditional key-frame extraction methods [74, 5] often struggle to ensure semantic coherence, leading to missing key-frames covering crucial changes and transitions. Consequently, we develop a semantic-aware key-frame extraction method that strikes a balance between reducing temporal redundancy and maintaining semantic coherence.

Semantic-aware Key-frame Extraction.We denote $V\in {ℝ}^{T\times H\times W\times 3}$ as a $T$ frame set sampled from a video with fixed 2-second intervals.We calculate the keyframe set $V^{\prime }\in {ℝ}^{T^{\prime }\times H\times W\times 3}$ that are sufficiently sparse yet comprehensively cover the evolution of events within the video that . We view the output CLS token of the CLIP-Large image encoder [50] as the global semantics of each frame and remove the adjacent frames that have a high semantic similarity. In practice, we initialize the keyframe set $V^{\prime }$ with the first frame of $V$. For each frame in $V$, we calculate its semantic similarity $d$ with the latest keyframe in $V^{\prime }$. If $d$ is lower than the pre-defined threshold, we view the frame as a keyframe and add it to the $V^{\prime }$. If not, the frame is skipped as redundant. For completeness, the last frame of $V$ is always added in $V^{\prime }$. We provide the pseudo-code in Figure 15.

2.3 Captioning Pipeline

As we mentioned in Section 1, we find if we feed all the frames to the GPT4V directly, the GPT4V struggles to stably generate captions with the correct temporal relation between frames, and its performance further worsens with the frame number increasing. On the other hand, if we concatenate all the frames into a large image, the GPT4V loses more details with the increasing frame number, as shown in Figure 11-12.

Differential Sliding-window Captioning.To this end, we develop a differential sliding-window captioning pipeline to generate high-quality captions with detailed temporal descriptions for various videos. Specifically, the input fed to the image multi-modal model each time includes the current key-frame and the previous key-frame along with its differential caption. Then, we introduce the Differential Prompt to guide GPT4V in focusing on the changes between the current and previous frames, such as posture, position, camera angle, etc. Additionally, incorporating the differential caption of the previous frame as supplementary context enhances the response quality and reduces hallucinations. This is because the image embedding and textual caption provide explicit and implicit representations of the image, respectively. The differential caption not only adds extra context but also integrates temporal information from two frames ago, further improving the model’s temporal understanding. It’s important to note that for the first key-frame, which lacks a preceding frame, its differential caption is replaced directly with the standard caption. Finally, we input all differential captions along with their corresponding timestamps into GPT4. A specific Summary Prompt is designed to instruct the LLM to generate high-quality video captions with precise temporal dynamics and detailed spatial information. In practice, we use GPT-4-Turbo-04-09 for all the annotations.

In the design of the prompts, we discovered that an explicit Hierarchical Prompt Design significantly aids the GPT4V in comprehending its role, its expected format, and its operational boundaries. This approach contributes to the stabilization of the output’s format and enhances the overall quality of the results. For more details, please refer to Section A.2

3.1 Model design

We fine-tune the IXC2-4KHD [17] using the collected video caption data, resulting in our ShareCaptioner-Video. For flexible usage, we re-organize the data for the following capabilities:

1. Fast Captioning The model employs an image-grid format for direct video captioning, providing rapid generation speeds that are ideal for short videos. In practice, we concatenate all the keyframes of a video into a vertically elongated image and train the model on a caption task.

2. Sliding Captioning The model supports streaming captioning in a differential sliding-window format, yielding high-quality captions that are suitable for long videos. Similar to the captioning pipeline used in Section 2.3, we take the two adjacent keyframes alongside the previous differential caption as input, and train the model to describe the events occurring between them.

3. Clip Summarizing The model can swiftly summarize any clip from ShareGPT4Video or videos that have undergone the differential sliding-window captioning process, eliminating the need to re-process frames. We use all the differential descriptions as input, and the output is the video caption.

4. Prompt Re-Captioning: The model can rephrase prompts input by users who prefer specific video generation areas, ensuring that T2VMs trained on high-quality video-caption data maintain format alignment during inference with their training. In practice, we use GPT-4 to generate Sora-style prompts for our dense captions, and we train the re-captioning task in reverse, i.e., by using the generated prompt as input and the dense caption as the training target.

In practice, we fine-tune the model end-to-end over one epoch. We follow the default high-resolution strategy, using ‘HD-55’ for fast captioning and ‘HD-25’ for the others. The learning rate is uniform across all model components and warms up from 0 to $2.5\times {10}^{-5}$ within the first 1% of steps. The batch size is set to $1024$, and we sample the data uniformly.

3.2 Scaling-up Captions

To validate the effectiveness of our ShareCaptioner-Video in the video captioning task and further support the development of the video generation domain, we utilized it to annotate a large volume of aesthetically appealing videos. Specifically, we meticulously collect and process 4.8 million video clips, totaling approximately 3000 hours, from three sources: MixKit [43], Pexels [48], and Pixabay [49]. Subsequently, we employ the sliding captioning mode of ShareCaptioner-Video to generate high-quality captions for these videos. The total captioning process requires approximately 4000 H100 GPU hours. We provide some statistics on generated captions in Figure 8.

4.1 Video Understanding

Datasets and Benchmarks.To thoroughly explore the benefits that our high-quality video-caption data bring to LVLMs, we conduct comprehensive evaluations of the model across three multi-modal video benchmarks. VideoBench [44] curates approximately 15,000 QA pairs spanning 10 evaluation dimensions from 13 existing data sources, such as MSVD-QA [62], MSRVTT-QA [62], Activitynet-QA [67], etc. MVBench [30] is designed to challenge LVLMs with video tasks that cannot be effectively resolved by single-frame reliance, featuring 4,000 QA pairs derived from 11 public video benchmarks. TempCompass [38] specifically assesses the nuanced performance of LVLMs across various temporal aspects, such as speed, direction, and attribute changes. It includes 410 videos and 7,540 meticulously collected instructions, emphasizing temporal comprehension and interaction.

Improving current LVLMs with ShareGPT4Video.We validate the effectiveness of the high-quality video-caption data collected in ShareGPT4Video to improve the performance of current LVLMs. For fairness and simplicity, we integrate 28K high-quality video-caption data related to complex scenes (Panda-70M [12], Ego4D [18], and BDD100K [66]) of ShareGPT4Video to replace the captions data in the VideoChatGPT-100K [42] conversation data with an equivalent number.Then we train the VideoLLaVA [34] and LLaMA-VID [32] with their default training settings and hyperparameters. As shown in Table 1, ShareGPT4Video consistently improves the alignment between video and language modalities in different LVLM architectures and scales. Specifically, VideoLLaVA-7B [34] achieves an average performance gain of 1.1 across three comprehensive multi-modal video benchmarks after integrating high-quality captions, while LLaMA-VID-7B and LLaMA-VID-13B achieve an average gain of 2.0 and 2.3, separately. Our high-quality video-caption data is particularly effective in helping LVLMs achieve significant performance improvements on benchmarks that require complex temporal understanding, such as TempCompass [38].

ShareGPT4Video-8B.To obtain our final ShareGPT4Video-8B model, we start with the LLaVA-Next-8B [26] image multi-modal model. Consistent with previous LVLM approaches [34, 42], we uniformly sample 16 frames from each video and arrange these frames into a 4x4 image grid to form the input for both training and inference, following the IG-VLM [24] strategy. For training data, we first collect 153K VQA data from various instructional video-to-text datasets to build our baseline. This collection includes 13K conversational data from VideoChatGPT [42] and 140K question-answer pairs, with 45K data points from CLEVRER [64], 8K from EGO-QA [18], 34K from NextQA [61], and 53K from TGIF-Transition [31]. Then, these VQA data are combined with 28K video-caption data, forming a consolidated training dataset of 181K samples. For more training details, please refer to Section A.1.

As illustrated in Table 3, 4, 5, we present a quantitative comparison between our ShareGPT4Video-8B model supercharged by our ShareGPT4Video dataset with existing state-of-the-art LVLMs. Notably, compared with previous LVLMs, our ShareGPT4Video-8B attains the most superior performance in all three comprehensive benchmarks.Specifically, thanks to the rich temporal information provided by ShareGPT4Video, our ShareGPT4Video-8B model achieves an impressive average accuracy of 61.5% on the TempCompass benchmark. This is an 11.6% increase over the previous best-performing LVLM, VideoLLaVA-7B.Additionally, despite the VideoBench and MVBench benchmarks collecting a diverse range of QA data from various existing video datasets, we achieve solid performance on these benchmarks, surpassing the previous state-of-the-art by an average accuracy of 2.7% and 8.2%.

Abaltion on caption quality and ViT.Based on ShareGPT4Video-8B, we study how the modality alignment is influenced by caption quality and learnable vision encoder. As indicated in Table 2, introducing short captions on top of VQA data may not yield substantial performance gains. It could even degrade performance on some benchmarks due to sub-optimal modality alignment. Comparing the first, second, and fourth rows of Table 2, the significant performance gains of comprehending temporal sequences benefited from our high-quality caption data are evident. Moreover, unlocking the vision encoder when training with detailed captions facilitates better LVLMs modality alignment.

4.2 Video Captioning

To verify the capability of ShareCapitoner-Video, we quantitatively compare the video captioning quality between ShareCapitoner-Video and GPT4V with human preference voting. As shown in Table 7, it performs on par with GPT4V. We also shows the qualitative results in Figure 9. For more details, please refer to Section A.4

4.3 Video Generation

Model setup.To validate the effectiveness of high-quality captions in the T2VMs area, we utilize ShareCaptioner-Video and Panda-Student [12] to generate high-quality and short video captions for 4.5M videos with 65 frames and 0.3M videos with 221 frames, separately. Following the process outlined in the Open-Sora-Plan [25], we fine-tuned the pretrained T2VM to enable the generation of high-fidelity 10-second videos. For comparison, we fine-tuned a baseline model with the same quantity of video-short-captions pairs. For more training details, please refer to Section A.1.

Qualitative analysis.As illustrated in Figure 5, the T2VM can accurately follow detailed prompts and demonstrate remarkable control over semantic content and camera movement when aided by high-quality, detailed captions generated by ShareCaptioner-Video. The resulting video showcases intricate and lively content. In contrast, when provided with brief captions, the T2VM struggles to adhere to complex generation prompts, leading to subpar results.

A.1 Experimental Details

Training details of ShareGPT4Video-8B.During training, we employ a batch size of 128 and the AdamW optimizer. We opt to fine-tune the entire model, setting the learning rate for the vision encoder at 2e-6, for the MLP projector at 2e-5, and for the LLM using LoRA [21], the learning rate is set at 2e-4. Such training strategy enables us to obtain an exceptional LVLM, ShareGPT4Video-8B, with 8 A100 GPUs in about 5 hours.

Implementation details for T2VMs. We utilize the Latte-XL [41] model with pre-trained weights and a text encoder from T5-XXL [51]. In the first stage, we perform pretraining on a lower number of frames and enable joint image-video training, with the image batch size being four times that of the video. This stage involved 50k training steps. In the second stage, both the video and image batch sizes were reduced to 2. For all training stages, we used AdamW optimizer with a constant learning rate of 2e-5, and the resolution for both images and videos was set to 512×512. For training precision, we used Bf16 since fp16 led to loss becoming NaN. The first stage requires around 6K H100 GPU hours, and the second stage requires around 36K Ascend GPU hours.

A.2 Hierarchical Prompt Design.

We introduce this design to help the multi-modal and language models effectively perform their roles during the captioning process. We separately illustrate the differential-caption prompt and summary prompt in Figure 7 and Figure 8.Hierarchical prompts primarily consist of four components. The Character part provides the model with an overall perception of the role it is to play and the work environment it faces. The Skills section specifies the skills the model needs to possess, ensuring precise compliance with multiple requirements without omissions or confusion. The Constraints section clarifies behaviors that users do not desire and the rules to be followed when constructing outputs. The Structured Input section requires users to set up according to their specific scenarios. For example, when guiding the model to generate differential captions, the Character part informs the model that it is an expert in analyzing video frames. The Skills section requires the model to describe inter-frame target actions and behaviors, changes in environment and background, alterations in target appearance attributes, and camera movements reflecting temporal changes. The Constraints section demands precise descriptions without listing them item by item. In the Structured Input section, the input consists of frame indexes, timestamps, the previous frame, and its differential caption, among others.

A.3 Prompt Template

A.4 Comparison of Caption Quality

Quantitative captioning capability comparison between ShareCaptioner-Video and GPT4V. We first analyze the linguistic composition of the captions produced by GPT4V and ShareCaptioner-Video, and the results are presented in Table 7. The analysis reveals that the captions generated by our ShareCaptioner-Video contain a comparable information level to those generated by GPT4V. Furthermore, as shown in Table 7, we first generate 100 captions with GPT4V and ShareCaptioner-Video and then invite 10 volunteers to evaluate the captions based on three aspects. These aspects include (1) Omission and Fabrication - checking for no key elements missing in the caption and identifying imaged elements not present in the video; (2) Distortion - assessing the accuracy of element attributes such as color and size; and (3) Temporal Mismatch - evaluating whether the description accurately reflects the evolution of temporal events in the video. Each pair can earn a maximum of 3 points, one for each criterion met. As anticipated, our ShareCaptioner-Video performs on par with the GPT4V.

Qualitative comparison of caption quality from various sources. In Figure 9, we compare the quality of captions from different sources. The result which uses the Panda-Student [12] model learned from multiple LVLMs, generates one-sentence captions for videos. While this approach minimizes errors, it provides insufficient textual information for information-dense videos, hindering the alignment of video and language modalities. The Video ChatCaptioner [9], on the other hand, uses ChatGPT to construct questions, employs BLIP2 [28] for VQA, and then uses ChatGPT again to summarize a complete caption based on the QA. Although this method produces longer captions than Panda-Student, the extensive involvement of ChatGPT often leads to over-imagination and hallucinations, negatively impacting the alignment between video and language modalities. Thanks to our carefully designed pipeline for generating high-quality video captions, GPT4V demonstrated impressive captioning capabilities. Additionally, after fine-tuning on the ShareGPT4Video dataset, our ShareCaptioner-Video also exhibits captioning capabilities comparable to GPT-4V.

A.5 Data Statistics

A.6 Pseudo Code

A.7 Related Work

Large video-language models. The vision-language community has recently made significant advancements in the field of large image-language models [36, 35, 37, 10, 13, 11, 58, 59, 2, 72, 33]. These models typically employ three core components: a visual encoder for extracting visual features, a connector to bridge the visual and language modalities, and a large language model (LLM) to decode the multimodal context. A milestone in this field is LLaVA-1.5 [35], which uses CLIP-Large [50] as the visual encoder, a two-layer MLP as the connector, and Vicuna-v1.5 [14] as the LLM backbone. This model achieves exceptional performance with only affordable data for training. Following this paradigm, the large video-language model field has also seen emerging efforts to encode video into vision tokens suitable for LLMs, enabling understanding and reasoning about video content. For example, VideoChat2 [30] employs a video transformer and a Q-Former [28] to extract and compress visual tokens. VideoLLaVA [34] uses a pre-aligned video encoder with image and language, along with an MLP, to extract and transform visual tokens. LLaMA-VID [32] introduces the LLaVA-like three-stage training strategy and compresses each frame’s vision token into one to support long video understanding. Despite the continuous exploration of training strategies and model architectures, none of these methods have investigated the role of high-quality captions in aligning video and language modalities, as has been done in the large image-language model field [10, 7].

Text-to-video models. Recent advancements in the text-to-image field, such as DALL·E [3], and Stable Diffusion [52], have significantly propelled the T2I area. Inspired by these achievements, researchers have increasingly begun to explore the potential of generating high-fidelity videos from textual descriptions. For instance, MagicVideo [73] introduces a 3D U-Net-based architecture and directs temporal attention to ensure efficient, high-fidelity video generation while maintaining temporal consistency and authenticity. PixelDance [68] and SVD [4] leverage pixel-based and latent-based techniques to produce high-resolution videos, while Make-A-Video [54] and Imagen Video [19] extend text-to-image models and techniques to the text-to-video domain. A milestone in this field is Sora [39], which ambitiously employs DiT to train a text-to-video model from scratch. Sora is the first model capable of generating minute-long videos based on user instructions. The success of this remarkable feat relies on extensive high-quality video-caption data, meticulously designed architectural optimizations, and substantial computational power. Despite the pivotal role of high-quality video-caption data in T2VMs, this aspect has not received sufficient attention. Therefore, in this work, we aim to construct a high-quality video-caption dataset to advance the development of T2VMs.