This is the official home for dovecot-deduper, a script that removes mail duplicates in your IMAP server on a by-folder basis.

Yes, you read right; it does only work on IMAP-style Maildirs where you have local access (ie. you are the serveradmin).

It worked for me, but this doesn't mean it will necessarily do the same for you.

Be cautious!

Add the "doit" flag only when you like what it suggests should be done.
It IS just a rather bad hackup, that does what I wanted it to do. Your mileage may vary. Have a backup ready! ;-)

Anyhow here is a list of releases available for download here.

To download the current release (0.0.1), click here.

Sample session output:

someuser@yourserver:/# dovecot-deduper-Maildir.sh                   

usage: dovecot-deduper-Maildir.sh  [doit]

   Scans all folders in  and deletes all (but one) dupe messages found per folder
   (using Message-Id or md5sum)

   Set second parameter to 'doit' to actually delete files.

someuser@yourserver:/# dovecot-deduper-Maildir.sh /home/mailuser/Maildir doit

. Looking for cur/ folders in users-maildir /home/mailuser/Maildir
.. Processing maildir /home/mailuser/Maildir/cur
... Getting Message-Ids from 1441 mails in directory
.... Got mfile /home/mailuser/Maildir/cur/1376448954.M488223P30455.yourserver,S=9848,W=10018:2,S (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376448955.M275461P30480.yourserver,S=9102,W=9263:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376448960.M236735P30640.yourserver,S=6245,W=6385:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449045.M432776P893.yourserver,S=116041,W=117456:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449054.M791567P1185.yourserver,S=6175,W=6309:2,S (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449095.M236323P2425.yourserver,S=80310,W=81582:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449254.M778603P5742.yourserver,S=6066,W=6175:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449255.M960489P5777.yourserver,S=5906,W=6014:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449257.M721529P5827.yourserver,S=6238,W=6350:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449267.M229360P6067.yourserver,S=6292,W=6404:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449270.M206803P6153.yourserver,S=6456,W=6571:2, (had to use md5sum)
.... Got mfile /home/mailuser/Maildir/cur/1376449279.M608205P6361.yourserver,S=6449,W=6563:2, (had to use md5sum)
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Downloads/cur
... Getting Message-Ids from 3 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.... Deleting dupe mail: /home/mailuser/Maildir/.Downloads/cur/1376441997.M719724P23543.yourserver,S=23634,W=23884:2,
.... Deleting dupe mail: /home/mailuser/Maildir/.Downloads/cur/1376443746.M314566P8661.yourserver,S=23631,W=23881:2,
.... Deleting dupe mail: /home/mailuser/Maildir/.Downloads/cur/1376445466.M182588P26224.yourserver,S=23635,W=23885:2,
.... Deleting dupe mail: /home/mailuser/Maildir/.Downloads/cur/1376447213.M591640P11330.yourserver,S=23635,W=23885:2,
.... Deleting dupe mail: /home/mailuser/Maildir/.Downloads/cur/1376440799.M713590P17174.yourserver,S=25316,W=25593:2,
.. Processing maildir /home/mailuser/Maildir/.Drafts/cur
... Nothing to do in directory
.. Processing maildir /home/mailuser/Maildir/.Newsletters/cur
... Getting Message-Ids from 63 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Paypal/cur
... Getting Message-Ids from 7 mails in directory
... Sorting ids gotten and return all but the last entry
.. Processing maildir /home/mailuser/Maildir/.Sent/cur
... Nothing to do in directory
.. Processing maildir /home/mailuser/Maildir/.Spam/cur
... Getting Message-Ids from 25 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Trash/cur
... Getting Message-Ids from 1 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Websites/cur
... Getting Message-Ids from 2 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Websites.Ebay/cur
... Getting Message-Ids from 79 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Websites.FB/cur
... Getting Message-Ids from 84 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Websites.Google/cur
... Getting Message-Ids from 21 mails in directory
... Sorting ids gotten and return all but the last entry
... Deleting duplicate mailfiles in directory
.. Processing maildir /home/mailuser/Maildir/.Work/cur
... Nothing to do in directory
. Cleaning up our current tempfiles (/tmp/dovecotdeduper-maildirs.2xQMc.tmp /tmp/dovecotdeduper-maildir-contents.dM078.tmp /tmp/dovecotdeduper-maildir-contents-sorted.v2rNd.tmp)
. Cleaning up our tempfiles older than two days
. fin!

someuser@yourserver:/#

dovecot-deduper by Frederic Krueger is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Impressum / Contact