DeVIS: Making Deformable Transformers Work for Video Instance Segmentation
Video Instance Segmentation (VIS) jointly tackles multi-object detection, tracking, and segmentation in video sequences. In the past, VIS methods mirrored the fragmentation of these subtasks in their architectural design, hence missing out on a joint solution. Transformers recently allowed to cast t...
| Autor: | |
|---|---|
| Tipo de documento: | Dissertação de mestrado |
| Data de publicação: | 2022 |
| País: | España |
| Recursos: | Universitat Politècnica de Catalunya (UPC) |
| Repositório: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | Inglês |
| OAI Identifier: | oai:upcommons.upc.edu:2117/382651 |
| Acesso em linha: | https://hdl.handle.net/2117/382651 |
| Nível de acesso: | Acesso aberto |
| Palavra-chave: | Video recording Image segmentation Deformable Transformers Video Instance Segmentation Vídeo Imatges--Segmentació Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la imatge i del senyal vídeo |
| _version_ | 1878731745968259073 |
|---|---|
| author | Caelles Prat, Adrià |
| author_browse | Caelles Prat, Adrià |
| author_facet | Caelles Prat, Adrià |
| author_role | author |
| collection | UPCommons. Portal del coneixement obert de la UPC |
| description | Video Instance Segmentation (VIS) jointly tackles multi-object detection, tracking, and segmentation in video sequences. In the past, VIS methods mirrored the fragmentation of these subtasks in their architectural design, hence missing out on a joint solution. Transformers recently allowed to cast the entire VIS task as a single set-prediction problem. Nevertheless, the quadratic complexity of existing Transformer-based VIS methods requires long training times, high memory requirements, and processing of low-single-scale feature maps.Deformable attention provides a more efficient alternative but its application to the temporal domain or the segmentation task have not yet been explored. In this work, we present Deformable VIS (DeVIS), a VIS method which capitalizes on the efficiency and performance of deformable Transformers. To reason about all VIS subtasks jointly over multiple frames, we present temporal multi-scale deformable attention with instance-aware object queries. We further introduce a new image and video instance mask head which exploits multi-scale features, and perform near-online video processing with multi-cue clip tracking. DeVIS benefits from comparatively small memory as well as training time requirements, and achieves state-of-the-art results on the YouTube-VIS 2019 and 2021, as well as the challenging OVIS dataset. |
| eu_rights_str_mv | openAccess |
| format | masterThesis |
| id | ES_35b2b39cc68d15102d6797d5106de48b |
| institution | Universitat Politècnica de Catalunya (UPC) |
| instname_str | Universitat Politècnica de Catalunya (UPC) |
| language | eng |
| network_acronym_str | ES |
| network_name_str | España |
| oai_identifier_str | oai:upcommons.upc.edu:2117/382651 |
| publishDate | 2022 |
| publishDateSort | 2022 |
| publisher | Universitat Politècnica de Catalunya |
| publisherStr | Universitat Politècnica de Catalunya |
| reponame_str | UPCommons. Portal del coneixement obert de la UPC |
| spelling | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation Caelles Prat, Adrià Video recording Image segmentation Deformable Transformers Video Instance Segmentation Vídeo Imatges--Segmentació Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la imatge i del senyal vídeo Video Instance Segmentation (VIS) jointly tackles multi-object detection, tracking, and segmentation in video sequences. In the past, VIS methods mirrored the fragmentation of these subtasks in their architectural design, hence missing out on a joint solution. Transformers recently allowed to cast the entire VIS task as a single set-prediction problem. Nevertheless, the quadratic complexity of existing Transformer-based VIS methods requires long training times, high memory requirements, and processing of low-single-scale feature maps.Deformable attention provides a more efficient alternative but its application to the temporal domain or the segmentation task have not yet been explored. In this work, we present Deformable VIS (DeVIS), a VIS method which capitalizes on the efficiency and performance of deformable Transformers. To reason about all VIS subtasks jointly over multiple frames, we present temporal multi-scale deformable attention with instance-aware object queries. We further introduce a new image and video instance mask head which exploits multi-scale features, and perform near-online video processing with multi-cue clip tracking. DeVIS benefits from comparatively small memory as well as training time requirements, and achieves state-of-the-art results on the YouTube-VIS 2019 and 2021, as well as the challenging OVIS dataset. Universitat Politècnica de Catalunya https://hdl.handle.net/2117/382651 |
| spellingShingle | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation Caelles Prat, Adrià Video recording Image segmentation Deformable Transformers Video Instance Segmentation Vídeo Imatges--Segmentació Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la imatge i del senyal vídeo |
| title | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation |
| title_full | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation |
| title_fullStr | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation |
| title_full_unstemmed | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation |
| title_short | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation |
| title_sort | DeVIS: Making Deformable Transformers Work for Video Instance Segmentation |
| topic | Video recording Image segmentation Deformable Transformers Video Instance Segmentation Vídeo Imatges--Segmentació Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la imatge i del senyal vídeo |
| topic_facet | Video recording Image segmentation Deformable Transformers Video Instance Segmentation Vídeo Imatges--Segmentació Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la imatge i del senyal vídeo |
| url | https://hdl.handle.net/2117/382651 |