dup.rst
author Oleksandr Gavenko <gavenkoa@gmail.com>
Mon, 22 Feb 2016 13:34:05 +0200
changeset 1910 123f59618e87
parent 1905 fba288d59662
child 1912 8b81a8f0f692
permissions -rw-r--r--
Fix: center TOC (after adding :local:).
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1164
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     1
.. -*- coding: utf-8; -*-
1334
9bf0d5a1f0cf Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1182
diff changeset
     2
.. include:: HEADER.rst
1164
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     3
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     4
==============
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     5
 Duplication.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     6
==============
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     7
.. contents::
1905
fba288d59662 Include only local subsections into TOC. This prevent duplication of
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1691
diff changeset
     8
   :local:
1164
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
     9
1166
3c925325d9d4 Search for duplicate lines.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1165
diff changeset
    10
Search for duplicate lines.
3c925325d9d4 Search for duplicate lines.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1165
diff changeset
    11
===========================
3c925325d9d4 Search for duplicate lines.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1165
diff changeset
    12
3c925325d9d4 Search for duplicate lines.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1165
diff changeset
    13
  http://en.wikipedia.org/wiki/Duplicate_code
3c925325d9d4 Search for duplicate lines.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1165
diff changeset
    14
                wiki page
1181
c3854c05f00d Code Clones Literature
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1166
diff changeset
    15
  http://students.cis.uab.edu/tairasr/clones/literature/
c3854c05f00d Code Clones Literature
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1166
diff changeset
    16
                Code Clones Literature
1166
3c925325d9d4 Search for duplicate lines.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1165
diff changeset
    17
1182
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    18
Open source or free licence:
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    19
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    20
 * http://duplo.sourceforge.net/
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    21
 * http://clonedigger.sourceforge.net/
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    22
 * http://www.ccfinder.net/ccfinderxos.html
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    23
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    24
Proprietary or restricted licence:
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    25
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    26
 * http://www.txl.ca/nicaddownload.html
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    27
 * http://www.harukizaemon.com/simian/index.html
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    28
 * http://getatomiq.com/
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    29
 * http://www.harukizaemon.com/simian/index.html
f0fd5e35e832 Open source or free licence:
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1181
diff changeset
    30
1164
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    31
Search for duplicate files.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    32
===========================
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    33
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    34
This utilities only search for duplicate files:
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    35
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    36
  http://duff.sourceforge.net/
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    37
                duff home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    38
  http://freedup.org/
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    39
                freedup home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    40
  http://dupedit.com/
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    41
                dupedit home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    42
  http://rdfind.pauldreik.se/
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    43
                Rdfind home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    44
  http://code.google.com/p/softenido/wiki/FindRepe
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    45
                FindRepe home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    46
1165
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    47
fdupes.
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    48
=======
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    49
::
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    50
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    51
  $ sudo apt-get install fdupes
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    52
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    53
See:
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    54
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    55
  http://code.google.com/p/fdupes/
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    56
                fdupes home page
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    57
  http://ru.wikipedia.org/wiki/Fdupes
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    58
                fdupes wiki page
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    59
  http://packages.debian.org/search?keywords=fdupes
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    60
                fdupes Debian package
b5e09fc4d751 fdupes.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents: 1164
diff changeset
    61
1164
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    62
freedups.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    63
---------
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    64
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    65
Freedups searches through the directories you specify. When it finds two
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    66
identical files, it hard links them together. Now the two or more files still
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    67
exist in their respective directories, but only one copy of the data is stored
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    68
on disk; both directory entries point to the same data blocks.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    69
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    70
  http://www.stearns.org/freedups/
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    71
                freedups home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    72
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    73
dupmerge.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    74
---------
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    75
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    76
Dupmerge reads a list of files from standard input (eg., as produced by "find .
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    77
-print") and looks for identical files. When it finds two or more identical
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    78
files, all but one are unlinked to reclaim the disk space and recreated as hard
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    79
links to the remaining copy.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    80
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    81
  https://sourceforge.net/projects/dupmerge/
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    82
                dupmerge home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    83
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    84
ssdeep.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    85
-------
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    86
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    87
ssdeep is a program for computing context triggered piecewise hashes (CTPH).
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    88
Also called fuzzy hashes, CTPH can match inputs that have homologies. Such
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    89
inputs have sequences of identical bytes in the same order, although bytes in
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    90
between these sequences may be different in both content and length.
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    91
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    92
  http://ssdeep.sourceforge.net/
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    93
                ssdeep home page
376df9f34507 Search for duplicate files.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff changeset
    94