Speed Comparison Test

Here you will get information about a speed comparison between the comparison method Byte by Byte and SHA-1. The additional options "Ignore Exif metadata of JPEG files" and "Log every x minute the search statistic of the scan progress info" where activated for each scan.

The files of the scanned folder were stored on an external hard disk connected via USB2 and encrypted with TrueCrypt. AllDup was installed at a computer running Windows 7 x64 with 8 GB RAM and an i3 CPU.

The log of the first scan using the comparison method Byte by Byte:

13:29:36 - Search: File content Byte by Byte
13:29:37 - File Count: 45442
13:29:37 - Scan: R:\!P2
13:33:37 - Duration : 0:04:01, Progress : 7 %, Current File : 3361 \ 45442, Duplicates Found : 1377 (3 %) (2,83 GB), Groups : 584, File Comparison Count : 811
13:37:39 - Duration : 0:08:03, Progress : 16 %, Current File : 7652 \ 45442, Duplicates Found : 2717 (5 %) (5,58 GB), Groups : 999, File Comparison Count : 1776
13:41:39 - Duration : 0:12:03, Progress : 45 %, Current File : 20806 \ 45442, Duplicates Found : 14426 (31 %) (9,30 GB), Groups : 3678, File Comparison Count : 11545
13:45:39 - Duration : 0:16:03, Progress : 57 %, Current File : 26252 \ 45442, Duplicates Found : 20138 (44 %) (11,94 GB), Groups : 4462, File Comparison Count : 16705
13:49:39 - Duration : 0:20:03, Progress : 65 %, Current File : 29665 \ 45442, Duplicates Found : 23521 (51 %) (14,16 GB), Groups : 4776, File Comparison Count : 19873
13:53:39 - Duration : 0:24:03, Progress : 70 %, Current File : 32144 \ 45442, Duplicates Found : 26118 (57 %) (16,41 GB), Groups : 5015, File Comparison Count : 22286
13:57:39 - Duration : 0:28:03, Progress : 76 %, Current File : 34663 \ 45442, Duplicates Found : 28575 (62 %) (18,42 GB), Groups : 5084, File Comparison Count : 24733
14:01:39 - Duration : 0:32:03, Progress : 82 %, Current File : 37573 \ 45442, Duplicates Found : 30751 (67 %) (20,45 GB), Groups : 5267, File Comparison Count : 26992
14:04:51 - Found 32215 duplicates with 22.970.559.143 Bytes in source folder 'R:\!P2'
14:04:51 - Groups: 5.442
14:04:51 - File Comparison Count: 28.370
14:04:51 - Duplicates: 32215 (70%) (21,39 GB)
14:04:51 - Elapsed time: 00:35:15

The Log of the second scan using the comparison method SHA-1:

14:10:31 - Search: File content SHA-1 (160-Bit)
14:10:32 - File Count: 45442
14:10:32 - Scan: R:\!P2
14:14:32 - Duration : 0:04:01, Progress : 7 %, Current File : 3422 \ 45442, Duplicates Found : 1487 (3 %) (3,08 GB), Groups : 639, File Comparison Count : 867
14:18:32 - Duration : 0:08:01, Progress : 26 %, Current File : 12073 \ 45442, Duplicates Found : 5778 (12 %) (6,98 GB), Groups : 2400, File Comparison Count : 3689
14:22:32 - Duration : 0:12:01, Progress : 55 %, Current File : 25020 \ 45442, Duplicates Found : 18904 (41 %) (11,16 GB), Groups : 4373, File Comparison Count : 15542
14:26:32 - Duration : 0:16:01, Progress : 66 %, Current File : 30193 \ 45442, Duplicates Found : 24082 (52 %) (15,22 GB), Groups : 4816, File Comparison Count : 20396
14:30:32 - Duration : 0:20:01, Progress : 78 %, Current File : 35462 \ 45442, Duplicates Found : 29392 (64 %) (19,56 GB), Groups : 5116, File Comparison Count : 25525
14:34:12 - Found 32215 duplicates with 22.970.559.143 Bytes in source folder 'R:\!P2'
14:34:12 - Groups: 5.442
14:34:12 - File Comparison Count: 28.370
14:34:12 - Duplicates: 32215 (70%) (21,39 GB)
14:34:12 - Elapsed time: 00:23:41

As expected, the comparison method SHA-1 wins the rally. Comparing only the checksums stored inside the RAM reduced the file read access and saved a lot of time.