Hello,
First thanks for sharing the code. Actually, I tried to estimate the segmentation masks with other queries than those you provided for each video (for this I used your demonstration on Google Colab). However, the result was super disappointing. Whatever query I use for the videos, the network still returns the segmentation masks related to the original queries. Here is just an example of many I tries with your network:
A random frame of the original video:
The original queries provided for this video:
'man in red shirt playing tennis', 'white tennis racket held by a man in a red shirt
The output mask generated by the network for the first query 'man in red shirt playing tennis':
Now if we change the queries to:
'a man wearing a black shirt sitting on the side', 'a man standing at the back of tennis court'
the result for the first query 'a man wearing a black shirt sitting on the side'
Now, we change the queries to:
'referee of a tennis match', 'the wall at the end of a tennis court'
the output mask from the network for the first query 'referee of a tennis match'
In fact, whatever queries you provide to the network, the output mask is related to one of those original queries originally assigned to the video. If we say "a kid in Disneyland" it returns one of these two players, "a lady reports for a tv show" it still returns one of these two players.
Hello,
First thanks for sharing the code. Actually, I tried to estimate the segmentation masks with other queries than those you provided for each video (for this I used your demonstration on Google Colab). However, the result was super disappointing. Whatever query I use for the videos, the network still returns the segmentation masks related to the original queries. Here is just an example of many I tries with your network:
A random frame of the original video:
The original queries provided for this video:
'man in red shirt playing tennis', 'white tennis racket held by a man in a red shirt
The output mask generated by the network for the first query 'man in red shirt playing tennis':
Now if we change the queries to:
'a man wearing a black shirt sitting on the side', 'a man standing at the back of tennis court'
the result for the first query 'a man wearing a black shirt sitting on the side'
Now, we change the queries to:
'referee of a tennis match', 'the wall at the end of a tennis court'
the output mask from the network for the first query 'referee of a tennis match'
In fact, whatever queries you provide to the network, the output mask is related to one of those original queries originally assigned to the video. If we say "a kid in Disneyland" it returns one of these two players, "a lady reports for a tv show" it still returns one of these two players.