Grand Challenges:

Challenge Introduction:

With the rise of live broadcasting services, users have a higher expectation of video quality. The great variations of videographic skills in shot environment, photographic apparatus, compression and processing protocols give rise to very complicated impairments in the live broadcasting videos, which can adversely impact the quality of experience (QoE) of end users.The complexity of distortion in live broadcasting videos and the fact that user experience is subjective and hard to quantify and measure pose challenges to QoE-based live video quality assessment (VQA). To address this challenge, we have built a large VQA database for live broadcasting with the associated QoE scores. It consists of 1013 videos with a carefully selected range of distortion type and intensity, focusing on the complicated impairments induced in the live broadcasting services. The paper can be found at here.

Our dataset fills the gap in publicly available datasets for studying the comprehensive effects of distortions in live video. We also conducted a subjective experiment, and with the analysis of the collected subjective data, it was found that the test results in the proposed database were not satisfactory, indicating that further work has to be done to deeply measure the characteristics of those specific distortions in live broadcasting videos. This proposed competition invites participants to benchmark their models on our publicly available database.

Challenge Significance:

The competition aims to foster innovation in both subjective and objective VQA techniques tailored to live broadcasting videos, addressing the unique challenges posed by live streaming impairments while emphasizing the evaluation of QoE. It also encourages the exploration of novel perceptual metrics that integrate QoE evaluation with learning-based approaches, extending beyond traditional distortion-focused methods. The competition is highly relevant to the fields of visual communication and video processing, as it supports the development of practical algorithms that can be applied in real-world scenarios such as live video streaming, content recommendation, mobile video capturing, and video enhancement. By advancing the state-of-the-art in live video quality assessment, this challenge will contribute to optimizing the visual quality and Quality of Experience (QoE) for live streaming platform users.

Submission Requirements:

The participants are required to follow the steps to submit their results:

A.Results

• (1) Process all video files and store predicted scores in a structured JSON output file named result.json. The output structure should preserve the original video processing order through sequential array positioning, with each entry containing:
• video_name: Original filename (string)
• scores: Corresponding predicted scores (float)

• (2) To ensure order consistency:
• Maintain native file sequence from the processing pipeline
• Implement explicit sorting with Python's sorted() before JSON serialization

[
  {"video_name": "video_001.mp4", "scores": 0.92},
  {"video_name": "video_002.mp4", "scores": 0.88},
  {"video_name": "video_003.mp4", "scores": 0.95}
]

Technical guidance:

• Use json.dump() with indent=4 parameter for human-readable formatting
• Validate JSON structure with jsonlint before deployment

B. Computational Complexity Analysis

(1) Execution Time Measurement:

• Per-video processing time (ms)
• Total pipeline duration (seconds)
• Time complexity trend analysis (linear/polynomial/exponential)

(2) Temporal Statistics Example:

Time Metrics:
- Average processing time per frame: 34.2 ± 2.8 ms
- Total execution duration: 12m 45s
- Complexity scaling factor: O(n^1.1) observed

C.Environment

Hardware Configuration:
- CPU: Intel Xeon Platinum 8480CL @ 2.4GHz (64 cores)
- GPU: NVIDIA A100 80GB PCIe
- Memory: 512GB DDR5 ECC
- Storage: NVMe SSD RAID-0 Array

Software Environment:
- OS: Ubuntu 22.04 LTS
- Python: 3.10.12
- CUDA: 12.2
- Dependencies:
  • OpenCV 4.8.0
  • PyTorch 2.1.1+cu121
  • NumPy 1.26.2

D. Benchmarking Information

[Benchmark Parameters]
Batch Size: 16  
Precision: FP16  
Warmup Iterations: 100  
Measurement Iterations: 1000  

Implementation Guidance:

• Use Python's time.perf_counter() for microsecond-level measurements
• Record environment details via pip freeze > requirements.txt
• Include hardware fingerprints using nvidia-smi and lscpu outputs

E. Summarize

• (1)Store the predicted scores in a structured JSON file named result.json according to requirement A
• (2)Store computational metrics and system configurations in a standardized text file benchmark_report.txt using human-readable key-value pairs according to requirement B,C,D
• (3)(Optional) Create a paper in the VCIP paper template to describe the designed VQA method for this challenge. It should give a detailed description of the proposed evaluation scheme and the computational complexity analysis.
• (4)Create and send a ZIP package named submission.zip, which contains the result.json, benchmark_report.txt files and the research paper (if any).
• (5)DATASET can be download at Baidu Netdisk (code:p63p) (All data is stored in videos and distinguished by "id" in train/test.xlsx)

Important Dates:

• 2025.04.20: Release of train and validation data
• 2025.05.25: Release of final test data
• 2025.06.04: Test output results submission deadline
• 2025.06.05: Code submission deadline
• 2025.06.15: Winner announcement
• 2025.06.30: Challenge paper submission deadline

Evaluation:

The evaluation consists of the comparison of the predictions with the reference ground truth, Mean Opinion Scores (MOS). We will assess your solutions based on Pearson Linear Correlation Coefficient (PLCC) and Spearman’s Rank-order Correlation Coefficient (SRCC).