VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
This research explores the integration of CLIP, a pretrained model, into video content analysis. In a landscape inundated with multimedia data, pinpointing specific moments within videos is a persistent challenge. By leveraging CLIP's semantic and visual search capabilities, this study endeavors to...
| Author: | |
|---|---|
| Format: | Master's thesis |
| Publication date: | 2024 |
| Country: | España |
| Institution: | Universitat Oberta de Catalunya (UOC) |
| Repository: | O2, repositorio institucional de la UOC |
| OAI Identifier: | oai:openaccess.uoc.edu:10609/149808 |
| Online Access: | http://hdl.handle.net/10609/149808 |
| Access Level: | Open access |
| Keyword: | video analysis moment retrieval CLIP Computer vision -- TFM Visió per ordinador -- TFM |
| _version_ | 1878735565860372480 |
|---|---|
| author | Caravaca Müller, Oriol |
| author_browse | Caravaca Müller, Oriol |
| author_facet | Caravaca Müller, Oriol |
| author_role | author |
| collection | O2, repositorio institucional de la UOC |
| description | This research explores the integration of CLIP, a pretrained model, into video content analysis. In a landscape inundated with multimedia data, pinpointing specific moments within videos is a persistent challenge. By leveraging CLIP's semantic and visual search capabilities, this study endeavors to refine content retrieval methods. Emphasizing efficiency and applicability, this study aims to make this process more precise and practical. With this research we also reviewed the state-of-the-art methods and produced empirical analysis on the effects of postprocessing on the similarity vectors obtained from CLIP encoders. Finally, we developed two distinct methods aimed at moment retrieval tasks in audiovisual data, obtaining a model that is able to outperform previous works in Zero-shot moment revival, reaching 57.3 at R@1 IoU=0.5 and 51.6 at mAP@0.5. |
| eu_rights_str_mv | openAccess |
| format | masterThesis |
| id | ES_7f3f0dfceefe7daec6162019fb5557ea |
| institution | Universitat Oberta de Catalunya (UOC) |
| instname_str | Universitat Oberta de Catalunya (UOC) |
| network_acronym_str | ES |
| network_name_str | España |
| oai_identifier_str | oai:openaccess.uoc.edu:10609/149808 |
| publishDate | 2024 |
| publishDateSort | 2024 |
| publisher | Universitat Oberta de Catalunya (UOC) |
| publisherStr | Universitat Oberta de Catalunya (UOC) |
| reponame_str | O2, repositorio institucional de la UOC |
| spelling | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings Caravaca Müller, Oriol video analysis moment retrieval CLIP Computer vision -- TFM Visió per ordinador -- TFM This research explores the integration of CLIP, a pretrained model, into video content analysis. In a landscape inundated with multimedia data, pinpointing specific moments within videos is a persistent challenge. By leveraging CLIP's semantic and visual search capabilities, this study endeavors to refine content retrieval methods. Emphasizing efficiency and applicability, this study aims to make this process more precise and practical. With this research we also reviewed the state-of-the-art methods and produced empirical analysis on the effects of postprocessing on the similarity vectors obtained from CLIP encoders. Finally, we developed two distinct methods aimed at moment retrieval tasks in audiovisual data, obtaining a model that is able to outperform previous works in Zero-shot moment revival, reaching 57.3 at R@1 IoU=0.5 and 51.6 at mAP@0.5. Aquesta investigació explora la integració de CLIP, un model preentrenat, en l'anàlisi de contingut de vídeo. En un paisatge inundat de dades multimèdia, identificar moments específics dels vídeos és un repte persistent. Aprofitant les capacitats de cerca semàntica i visual de CLIP, aquest estudi intenta perfeccionar els mètodes de recuperació de contingut. Subratllant l'eficiència i l'aplicabilitat, fent aquest procés més precís i pràctic. En aquesta investigació també s’ha revisat l’estat de l’art i s’ha produit un anàlisis empíric sobre els efectes del postprocessament sobre els vectors de semblança obtinguts a partir dels codificadors de CLIP. Finalment s’han desenvolupat dos mètodes diferents dirigits a tasques de recuperació de moments en dades audiovisuals, obtenint un model que és capaç de superar els treballs anteriors en Zero-shot moment revival, arribant a 57,3 a R@1 IoU=0,5 i 51,6 a mAP@0,5. Universitat Oberta de Catalunya (UOC) http://hdl.handle.net/10609/149808 |
| spellingShingle | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings Caravaca Müller, Oriol video analysis moment retrieval CLIP Computer vision -- TFM Visió per ordinador -- TFM |
| title | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings |
| title_full | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings |
| title_fullStr | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings |
| title_full_unstemmed | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings |
| title_short | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings |
| title_sort | VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings |
| topic | video analysis moment retrieval CLIP Computer vision -- TFM Visió per ordinador -- TFM |
| topic_facet | video analysis moment retrieval CLIP Computer vision -- TFM Visió per ordinador -- TFM |
| url | http://hdl.handle.net/10609/149808 |