compare folders and find duplicate files using script

fdupe: a small script written in Perl.findrepe: free Java-based command-line tool designed for an efficient search of duplicate files, it can search within zips and jars.(GNU/Linux, Mac OS X, *nix, Windows).ua: Unix/Linux command line tool, designed to work with find (and the like).rmlint: Fast finder with command line interface and many options to find other lint too (uses MD5), since 18.04 LTS has a rmlint-gui package with GUI (may be launched by rmlint -gui or from desktop launcher named Shredder Duplicate Finder).rdfind: One of the few which rank duplicates based on the order of input parameters (directories to scan) in order not to delete in 'original/well known' sources (if multiple directories are given).liten2: A rewrite of the original Liten, still a command line tool but with a faster interactive mode using SHA-1 checksums (Linux, Mac OS X, *nix).liten: Pure Python deduplication command line tool, and library, using md5 checksums and a novel byte comparison algorithm.fslint: has command line interface and GUI.Now the two or more files still exist in their respective directories, but only one copy of the data is stored on disk both directory entries point to the same data blocks. When it finds two identical files, it hard links them together. freedups: shell script, that searches through the directories you specify.Uses MD5, SHA1 and other checksum algorithms 6 Best Free Duplicate File Finder Software 2020 for Windows 10/8/7 Get All Tech. Dont use this to hack the O/S - rather use it for personal data.

But please be careful- just because something occurs more than once does not mean its unnecessary. fdf: Perl/c based and runs across most platforms (Win32, *nix and probably others). Example: WinMerge - You will see the difference.dupseek: Perl with algorithm optimized to reduce reads.dupmerge: runs on various platforms (Win32/64 with Cygwin, *nix, Linux etc.).Avoids comparing files against themselves when multiple paths point to the same file. dupedit: Compares many files at once without checksumming.List of programs/scripts/bash-solutions, that can find duplicates and run under nix:

Compare folders and find duplicate files using script