Algorithm to detect duplicate videos ?

[Coolman]

hello,

is what someone has created a program to detect the videos in duplicate. or has developed an efficient algorithm in this direction ...

ps: I already test programs in that sense. I looking for a free solution.

[sean_vn]

http://en.wikipedia.org/wiki/Locality-sensitive_hashing

[Coolman]

Interesting but very complex. difficult to understand. I am not a mathematician. I looking for a simpler algorithm (FreeBasic) based on the research of the same color matching point ...

it seems that the subject was not studied here, I'll see in other forums ...

but thank you for the link ...

[sean_vn]

Well, ok. It depends on what you want to do. What is the problem? You have 2 video streams the same resolution, frame rate etc? Are they the same length or are you looking to locate where they overlap? If the streams have the same resolution etc then pick say 10 random pixel positions eg. (163,573), (437,294)... then find the colours in each frame at those pixel locations. When you are compairing the 2 streams you only have to check 10 colours per frame instead of 10's or 100's of thousands per frame. Sure it's approximate, you can do better if you do some more math.
It is extremely unlikely you are going to find, off the bat, code that does identical to what you want for free just out there on the internet somewhere. And even if you did you would likely end up having to learn an extensive API just to use the thing.
It generally ends up far quicker and easier to write your own code, even if it means reinventing the wheel.

[vdecampo]

Just randomly sample 100+ bytes in each file and build a table of values. Then compare all of the values for matches. Also compare based on file date/time, size, and name.

IMO
-Vince

[Coolman]

sean_vn

in general. before creating a feature I do a search to avoid wasting time reinventing the wheel. I give an example. Recently I was looking for a way to clean html code, leaving only the text. I found a function pascal language in three minutes on the net. then the creation of the program in lazarus took a few minutes. if I created it myself the function. it would probably take a lot longer ...

vdecampo

good idea. I had thought but it is a video file that can be of different resolutions. so this is suitable ...

though. thank you for your answers.

[TESLACOIL]

one method i would be tempted to use is too simply look at 'the rate' that the data changes over time

think of it as a kind of 'beat detection' , almost every video will have a unique 'data beat' even when examined roughly


There are 101 variables which will defeat any narrow thru the keyhole type sampling * assuming you are comparing videos from different sources, then you have resolution, frame rate, display ratio, compression method etc etc

By using a high level snooper, ie standing back from the technical details you will be able to apply a 'common sense kind of algorithm' to get a common sense kind of answer which will give you a good idea of 'how similar' any two videos are to a person viewing them

[TESLACOIL]

Must add, Google Optical Flow

with this method you could pick out two similar videos even if one was in black and white, played backwards, mirrored and at a jaunty angle and quite possibly via a fish eye lens too !...our brains use multiple algorithms which are dynamically assigned priorities according to experience...we can be fooled but the task of fooling an experienced observer is non trivial

[Coolman]

http://duplicatevideosearch.com/

I found and tests Duplicate Video Search, efficient, the test version was actually find many videos in double ...

but will not pay for that, I should prefer an open source solution, I continue my research ...

[Coolman]

Resolved. the solution is a basic ffmpeg to create thumbnails in jpg SELECTED videos in resolution of 80x60. Then I tested some duplicate image detection programs. The most effective was a free program called ImgSearch. the final isolation of double image files and video files association with visual selection and deleting double ...

DOS is still alive and effective for this kind of treatment in batch ...

Note: I have recovered many gigas occupied by duplicate files videos ...

I would have to find the time to automate all this in a program ...

[St_W]

DOS is still alive and effective for this kind of treatment in batch ...

You mean Windows NT's command line interface (cmd.exe), do you? Plain DOS is only of very limited usefulness nowadays; e.g. for flashing firmware, hardware diagnostics and other low level/realtime stuff.[/quote]

I would have to find the time to automate all this in a program ...

You can also try this program instead: https://tn123.org/simimages/ (does also compare videos; not ony images as suggested by its name)
It's not open source though, but it's freeware.

[Coolman]

DOS is still alive and effective for this kind of treatment in batch ...

You mean Windows NT's command line interface (cmd.exe), do you? Plain DOS is only of very limited usefulness nowadays; e.g. for flashing firmware, hardware diagnostics and other low level/realtime stuff.

I would have to find the time to automate all this in a program ...

You can also try this program instead: https://tn123.org/simimages/ (does also compare videos; not ony images as suggested by its name)
It's not open source though, but it's freeware.[/quote]

I use a lot back to launch batch script which perform many tasks such as backups, file copies displacement ... for ffmpeg (download here http://ffmpeg.zeranoe.com/builds/ , the 64 bits is faster) I created the following script:

Code: Select all

cls
echo off
set crep=%cd%
set thumb="%crep%"\_Thumbnails_\
echo.
echo Creation Thumbnails en cours. Patientez svp...
if not exist "%thumb%" mkdir "%thumb%"
for %%A IN (*.mp4) do (	
	if not exist "%thumb%%%A.jpg" ffmpeg -ss 00:02:00 -i "%%A" -s 80x60 -vframes 1 "%thumb%%%A.jpg" -an -loglevel quiet
)
echo.
echo.
echo Fini
pause
echo on

save this script for example in ffthumb.bat

must be positioned at the root of the directory containing the videos, then run the script, a Thumbnails directory will be created at this level with the preservation of the image of the video to the necessary 0:02:00 position to avoid the generic and increase the detection of duplicates. Then it's simple enough to be used to isolate ImgSearch duplicate images that correspond with the name of the videos ...

ps: the script is configured to take into account that the mp4 format, if you want to add others eg avi, replace (* .mp4) by (* .mp4 * .avi) ...

there there's another script that level to move the videos whose images correspond to a different directory ...

if you want more explanation, just ask but I think you have grasped the concept ...

ImgSearch a small program of about 135 kb programmed with Delphi is surprising efficiency. pity that the author does not provide the source code if I could adapt to automate the entire process...

I tested SimilarImages but it does not take into account the videos in mp4 format, so I drop ...

[peterfan]

You can do some research on this software:
http://couponcode0.com/duplicate-video-search.html