Path: bloom-beacon.mit.edu!senator-bedfellow.mit.edu!faqserv From: Nancy McGough Newsgroups: comp.mail.misc,comp.mail.elm,comp.mail.pine,comp.answers,news.answers Subject: Filtering Mail FAQ Supersedes: Followup-To: comp.mail.misc Date: 13 Jan 1995 18:22:27 GMT Organization: Infinite Ink, Seattle, Washington, USA Lines: 858 Approved: news-answers-request@mit.edu Expires: 24 Feb 1995 18:16:00 GMT Message-ID: Reply-To: Nancy McGough NNTP-Posting-Host: bloom-picayune.mit.edu X-Last-Updated: 1994/11/18 Originator: faqserv@bloom-picayune.MIT.EDU Xref: bloom-beacon.mit.edu comp.mail.misc:11457 comp.mail.elm:10037 comp.mail.pine:5426 comp.answers:9503 news.answers:33088 Archive-name: mail/filtering-faq Posting-frequency: approximately monthly Last-modified: 17 November 1994 _____________________________________________________ _________| |_________ \ | | / \ | FILTERING MAIL FAQ | / \ | | / \ | | / / | Copyright (c) 1994 Nancy McGough | \ / | | \ / |_____________________________________________________| \ /____________) (___________\ TABLE OF CONTENTS 0.0 About this Article ... 0.1 Copyright Notice ... 0.2 Acknowledgements ... 0.3 Terminology and Notation ... 0.4 Getting the Latest Version of This Article ... 0.5 Reading the Hypertext Version of This Article ... 0.6 Reading the Plain Text Version of This Article 1.0 Naming Your Incoming Mail Folders 2.0 Procmail ... 2.1 Setting Up Procmail ... 2.2 Tracking Your Incoming Mail ... 2.3 Explanation of Test Recipe ... 2.4 Troubleshooting: Alternate .forward Files for Procmail ... 2.5 Procmail References 3.0 Filter ... 3.1 Setting up Filter ... 3.2 Tracking Your Incoming Mail ... 3.3 Filter References 4.0 Reading Incoming Mail Folders 5.0 Contributing to this FAQ ---------------------------------------------------------------------- Date: 17 Nov 1994 23:00:00 GMT From: Nancy McGough Subject: 0.0 About this Article Q: How can I have my incoming mail messages automatically put into an appropriate folder? This is one of the most frequently asked questions about email. This article is the first release of an FAQ that addresses this question. This version gives basic instructions for how to set up either procmail or Elm's filter to filter incoming mailing list messages. Future versions of this FAQ will include instructions for doing other things like automatically replying to certain messages. If your system has both procmail and filter installed then you should use procmail which is *much* more robust and powerful than filter. This recommendation is almost universal; even the Elm and filter developers recommend procmail over filter. ------------------------------ Date: 17 Nov 1994 23:00:10 GMT From: Nancy McGough Subject: ... 0.1 Copyright Notice Copyright (c) 1994 by Nancy McGough. No portion of this work may be sold or put to commercial use without express written consent of the author. This restriction covers publication in any form, or distribution by any method, which permits this work to be visually perceived, either directly or with the aid of any machine or device. Permission is granted to republish or redistribute this article in its entirety for noncommercial use if this copyright notice is not removed or altered. ------------------------------ Date: 17 Nov 1994 23:00:20 GMT From: Nancy McGough Subject: ... 0.2 Acknowledgements Thanks to these people who sent suggestions: David L. Miller Cookie Monster Jim Showalter David W. Tamkin Rick Troxel Stephen R. van den Berg Syd Weinstein Special thanks to: Thomas A. Fine for setting up and maintaining the hypertext archive of FAQs. Congratulations to him for winning O'Reilly and Associates' "The Best of the Net" award! Please let me know if I've left you, or anyone else, out of this list. ------------------------------ Date: 17 Nov 1994 23:00:30 GMT From: Nancy McGough Subject: ... 0.3 Terminology and Notation Term Meaning ==== ======= FTP File Transfer Protocol HTML HyperText Markup Language HTTP HyperText Transport Protocol mailer Mail user agent (MUA) such as Pine or Elm pico PIne COmposer - friendly editor that's part of the Pine package RFC Internet "Request For Comments" document URL Uniform Resource Locator (specified in RFC1630) ^x Press the Ctrl key and then, while holding down the Ctrl key, press the x key ~ Your home directory $HOME (see * below) * In this article I use ~ (tilde) to mean your home directory. Note that some shells, such as the Bourne shell (sh), do not understand the tilde notation so you will need to use $HOME instead. You can always get to your home directory by typing ``cd''. ------------------------------ Date: 17 Nov 1994 23:00:40 GMT From: Nancy McGough Subject: ... 0.4 Getting the Latest Version of This Article If this article is over two months old then there is probably an updated version of it in all the usual archives. Please get the latest version from one of these places: World Wide Web (the nicest format for online reading!): http://www.cis.ohio-state.edu/hypertext/faq/usenet/mail/filtering-faq/faq.html Anonymous FTP: ftp://rtfm.mit.edu/pub/usenet/news.answers/mail/filtering-faq Email: Send mail to mail-server@rtfm.mit.edu containing the following: send usenet/news.answers/mail/filtering-faq UUCP: uunet!/archive/usenet/news.answers/mail/filtering-faq ------------------------------ Date: 17 Nov 1994 23:00:50 GMT From: Nancy McGough Subject: ... 0.5 Reading the Hypertext Version of This Article The best way to read this FAQ (and most other FAQs too) is to view the hypertext version using a Web browser such as Lynx, Mosaic, Netscape, WinWeb, or Cello. This will allow you to easily jump: * between subjects in this article * to a Uniform Resource Locator (URL) * to an Internet Request For Comments document (RFC) * to some manual pages This, and all FAQs that are crossposted to news.answers, can be accessed through: http://www.cis.ohio-state.edu/hypertext/faq/usenet/top.html This particular FAQ is at: http://www.cis.ohio-state.edu/hypertext/faq/usenet/mail/filtering-faq/faq.html ------------------------------ Date: 17 Nov 1994 23:00:60 GMT From: Nancy McGough Subject: ... 0.6 Reading the Plain Text Version of This Article If you are reading the plain text version (i.e., not the hypertext version) of this FAQ it is in RFC1153 digest format which means each subject is formatted as a mini mail message with its own Date, From, and Subject (and sometimes Followup-To) headers. Some newsreaders and mailers make it really easy to jump to a subject in an article that is in digest format. Here are instruction for some newsreaders: * NN's default is to split a digest and present each digest item on the menu. If you have changed the default by putting ``set split off'' in your ~/.nn/init file then you can split an article that is in digest format by typing G% while viewing it. This will present each digest item on a submenu. You can then read, followup-to, save, print, etc. individual digest items. * In trn (and its relatives like rn and strn) you can type ^g (Ctrl key and g key pressed together) to skip to the next line that begins with "Subject: " * In emacs GNUS, C-c C-n will skip to the next digest article (C-C C-p will go back to the previous digest article). If your newsreader or mailer does not have a built-in command that allows you to easily read a digest you can pipe the article to ``formail -ds'' which will split the article into separate mail messages, and then you can use your mailer or newsreader (if it can read mail folders) to read the folder. For example, here's what you would do in Tin while viewing the article: Type... In order to... ------- -------------- | Pipe a Specify the article is to be piped formail -ds >> ~/mail/faq.split Split the digest into individual messages and put them in a file named ~/mail/faq.split. (Replace ~/mail with your folder directory.) pine -if faq.split Use Pine to read the newly created folder. For more information see the formail(1) man pages. ------------------------------ Date: 17 Nov 1994 23:01:00 GMT From: Nancy McGough Subject: 1.0 Naming Your Incoming Mail Folders For my incoming mail folders I use names that start with ``IN''. For example, I put mail sent to the procmail mailing list into a folder named IN.procmail. I do this so that when all my folders are listed alphabetically the incoming folders are together and near the top. They are near the top because Unix is case sensitive and upper case letters come before lower case letters in an ascii sort. Of course, you can use any names you like for your mail folders. ------------------------------ Date: 17 Nov 1994 23:02:00 GMT From: Nancy McGough Subject: 2.0 Procmail Procmail is a powerful mail processor that can be used to process your mail messages either as they arrive or after they are in a mail folder. This version of the FAQ describes the basics of setting up procmail to filter incoming mailing list messages. To find out how to process existing mail folders see the NOTES section of the procmail(1) man page. ------------------------------ Date: 17 Nov 1994 23:02:10 GMT From: Nancy McGough Subject: ... 2.1 Setting Up Procmail 1] Find out if procmail is on your system and what the full path to it is. If you are using csh type: which procmail Or if you are using sh or ksh type: type procmail If neither ``which'' nor ``type'' are on your system try ``whereis'' and ``where''. If your system doesn't have procmail ask your system administrator to install it. If your sys admin isn't able to do this, use a different mail processor like deliver, mailagent, or filter (described in part 2 of this FAQ). 2a] Create ~/.procmailrc. (Note that throughout this article I use pico for editing files. Replace ``pico'' with your editor.) cd pico .procmailrc 2b] Enter a modified version of the following in your ~/.procmailrc. Note that lines that begin with # are comments and are ignored by procmail. #Set on when debugging VERBOSE=off #Replace ``mail'' with your mail directory (Pine uses mail, Elm uses Mail) MAILDIR=$HOME/mail #Directory for storing procmail log and rc files PMDIR=$HOME/.procmail LOGFILE=$PMDIR/log INCLUDERC=$PMDIR/rc.test INCLUDERC=$PMDIR/rc.folders 3] Create the directory where you will store your procmail log and rc files (this is $PMDIR that you set above). cd mkdir .procmail 4a] Create an rc (run commands) file for testing: cd cd .procmail pico rc.test 4b] Enter the following in ~/.procmail/rc.test: :0: * ^Subject: .*test IN.testing Note that the first line contains a zero (0), not the letter "oh". For now, don't worry about the meaning of this recipe. It is explained in the subject "Explanation of Test Recipe" below. 5a] Create a ~/.forward file by typing the following. (Pico's -w flag tells pico not to auto wrap lines.) cd pico -w .forward 5b] Enter a modified version of the following in your ~/.forward: "|IFS=' ' && exec /usr/local/bin/procmail -f- || exit 75 #nancym" == IMPORTANT NOTES == * Make sure you include all the quotes, both double (") and single ('). * The vertical bar (|) is a pipe. * Replace /usr/local/bin with the correct path for procmail (see step 1). * Replace ``nancym'' with your userid. You need to put your userid in your .forward so that it will be different than any other .forward file on your system. * Do NOT use environment variables, like $HOME, in your .forward file. * Do NOT expect ~ to mean your home directory in the .forward file: If procmail resides below your home directory write out the *full* path. 5c] Note that on many systems you need to make your .forward world readable and your home directory world searchable in order for the mail transport agent to "see" it. To do this type: cd chmod 644 .forward chmod a+x . 6] Send yourself two test messages: one with ``test'' in the subject and one without ``test'' in the subject. 7a] Start your mailer (pine, elm, etc.) and check that the messages were delivered correctly. The one with ``test'' in the subject should be in the folder $MAILDIR/IN.testing and the one without ``test'' in the subject should be in your inbox. If these were delivered correctly go on to step 8. 7b] TROUBLESHOOTING * If the two messages were not delivered correctly look at your $LOGFILE (~/.procmail/log) to see if you can determine what the problem is. * Check these three files for typos: ~/.forward ~/.procmailrc ~/.procmail/rc.test * Check the file and directory permissions of your .forward (set in 5c above). Type... In order to... ------- -------------- cd Go to your home directory. ls -l .forward Check the permission: it should say -rw-r--r-- ls -ld . Check permission of home dir: it should say drwx?-x?-x The ?'s may be r's or hyphens or one of each (i.e., drwx--x--x, drwxr-xr-x, drwxr-x--x, drwx--xr-x). * If none of these turn up the problem edit your ~/.procmailrc so that it contains: VERBOSE=on And repeat steps 6 and 7. If you are still having problems see the subject "Troubleshooting: Alternate .forward Files for Procmail" below. 8a] Once you have successfully tested procmail in steps 6 and 7, create rc.folders for filtering incoming messages into mail folders. cd cd .procmail pico rc.folders 8b] Enter a modified version of the following in ~/.procmail/rc.folders :0: * ^TOwww-talk IN.www-talk :0: * ^TOprocmail IN.procmail The first recipe filters the www-talk mailing list and the second recipe filters the procmail mailing lists The meaning of the first recipe is as follows: Notation Meaning ======== ======= :0 Begin a recipe : Use a lock file * Begin a condition ^TO Match ``To:'' ``Cc:'' or other synonyms for To at the beginning of a line, followed by any or no characters, followed by.... www-talk ``www-talk'' IN.www-talk If successful match, put in folder $MAILDIR/IN.www-talk Create a recipe for each of your mailing lists. Make sure that you use ``^TO'' with no space between the caret (^) and the word ``TO'', and that both letters are capitalized -- if you don't it won't work. Note that ^TO is not a normal regular expression. It is a special procmail expression that is designed to catch any destination specification. See the MISCELLANEOUS section of the procmailrc(5) man page for details. For examples, see procmailex(5) man page. 9] Repeat steps 6 and 7 to make sure that things are still working. 10] Comment out the rc.test line in you .procmailrc file so that it looks like this: VERBOSE=off MAILDIR=$HOME/mail PMDIR=$HOME/.procmail LOGFILE=$PMDIR/log # INCLUDERC=$PMDIR/rc.test INCLUDERC=$PMDIR/rc.folders Note that it's useful to leave the rc.test line there for future testing. ------------------------------ Date: 17 Nov 1994 23:02:20 GMT From: Nancy McGough Subject: ... 2.2 Tracking Your Incoming Mail There is a useful script, which is part of the procmail package, for checking your procmail log file called mailstat. Check to see if it is on your system by typing either ``which mailstat'' or ``type mailstat''. If it's on your system type: mailstat $HOME/.procmail/log This displays a concise version of your log file and moves your log file to log.old. You may want to put the above line in your .login so that each time you log in you will see a listing of how many messages you've received since the last time you ran mailstat, and what folders these messages were delivered to. You can get a mailstat listing of log.old by using the -o flag: mailstat -o $HOME/.procmail/log If mailstat is not on your system ask your system administrator to install it. It is located with all the other procmail tools at: ftp://ftp.informatik.rwth-aachen.de/pub/packages/procmail/procmail.tar.gz ------------------------------ Date: 17 Nov 1994 23:02:30 GMT From: Nancy McGough Subject: ... 2.3 Explanation of Test Recipe The recipe we used for testing is: :0: * ^Subject: .*test IN.testing The meaning of this recipe is: Notation Meaning ======== ======= :0 Begin a recipe : Use a lock file * Begin a condition ^ Match the beginning of a line followed by.... Subject: ``Subject:'' followed by.... . a space followed by any character (.) followed by.... * 0 or more of preceding character (any character in this case) followed by.... test ``test'' IN.testing If successful match, put in folder $MAILDIR/IN.testing ------------------------------ Date: 17 Nov 1994 23:02:40 GMT From: "Stephen R. van den Berg" Subject: ... 2.4 Troubleshooting: Alternate .forward Files for Procmail If the .forward template in 5b above doesn't work the following alternatives might be helpful: In a perfect world: "|exec /usr/local/bin/procmail #nancym" In an almost perfect world: "|exec /usr/local/bin/procmail USER=nancym" In another world: "|IFS=' ';exec /usr/local/bin/procmail #nancym" In a different world: "|IFS=' ';exec /usr/local/bin/procmail USER=nancym" In a smrsh world: "|/usr/local/bin/procmail #nancym" These formats can be tried in different combinations, the leading "| can be tried as |" instead, or vice versa. Some systems do not need a .forward file (i.e., having a .procmailrc file suffices if procmail already is the local delivery agent). ------------------------------ Date: 17 Nov 1994 23:02:50 GMT From: Nancy McGough Subject: ... 2.5 Procmail References Manuals: procmail(1) - autonomous mail processor procmailrc(5) - procmail rc file procmailex(5) - procmail rc file examples procmailsc(5) - procmail weighted scoring techique egrep(1) - search file for regular expression (procmail uses egrep-style regular exprssions along with some of its own expressions like ^TO) formail(1) - mail reformatter sendmail(8) - send mail over the internet Newsgroup: comp.mail.misc Mailing List: procmail@informatik.rwth-aachen.de Subscribe to the procmail mailing list by sending mail: To: procmail-request@informatik.rwth-aachen.de Subject: subscribe Procmail Archives: Get a list of files available at the procmail mail server by sending mail: To: procmail-request@informatik.rwth-aachen.de Subject: archive ls Get Best of the Procmail mailing list by sending mail (you'll need gzip and a MIME-decoder to unpack it): To: procmail-request@informatik.rwth-aachen.de Subject: archive get best_of_procmail_list* ------------------------------ Date: 17 Nov 1994 23:03:00 GMT From: Nancy McGough Subject: 3.0 Filter Filter is part of the Elm package of tools. Note that you can use filter to filter your incoming mail even if you are not using Elm to read your mail. IMPORTANT NOTE ============== If your system has both procmail and filter installed then you should use procmail which is *much* more robust and powerful than filter. This recommendation is almost universal; even the developers of Elm and Filter recommend procmail over filter. IT IS POSSIBLE TO LOSE MAIL MESSAGES WHEN USING FILTER; this is rare but it has happened. ------------------------------ Date: 17 Nov 1994 23:03:10 GMT From: Nancy McGough Subject: ... 3.1 Setting up Filter Followup-To: comp.mail.elm 1] Find out if filter is on your system and what the full path to it is. If you are using the C shell (csh) type: which filter Or, if you are using the Korn shell (ksh) or the Bourne shell (sh) type: type filter If neither ``which'' nor ``type'' are on your system try ``where'' and ``whereis''. If your system doesn't have filter ask your system administrator to install it; or even better ask her to install procmail. 2] Note the full path of your home directory by typing: cd pwd 3a] Create ~/.elm/filter-rules. (Note that throughout this article I use pico for editing files. Replace ``pico'' with your editor.) cd mkdir .elm cd .elm pico filter-rules 3b] Enter a modified version of the following in your ~/.elm/filter-rules if (subject contains "test") then save "/j/nancym/Mail/IN.testing" Replace /j/nancym with your home directory path (see step 2). Replace /Mail with the name of the directory where your mail folders are stored. Pine and Berkeley Mail use /mail (lower case m) and Elm uses /Mail (upper case M). 4] To see what the filter rule will do type the following at your Unix prompt: filter -r 5a] Create a ~/.forward file by typing the following. (Pico's -w flag tells pico not to auto wrap lines.) cd pico -w .forward 5b] Enter a modified version of the following in your ~/.forward: "|/usr/local/bin/filter -o /j/nancym/.elm/filter-errors" == IMPORTANT NOTES == * Make sure you include the quotes ("). * The vertical bar (|) is a pipe. * Replace /usr/local/bin with the correct path for filter (see step 1). * Replace /j/nancym with your home directory (see step 2). * Do NOT expect environment variables, like $HOME, to work in your .forward file. * Do NOT expect ~ to mean your home directory in the .forward file. 5c] Note that on many systems you need to make your .forward is world readable and your home directory world searchable in order for the mail transport agent to "see" it. To do this type: cd chmod 644 .forward chmod a+x . 6] Send yourself two test messages: one with ``test'' in the subject and one without ``test'' in the subject. 7a] Start your mailer (pine, elm, etc.) and check that the messages were delivered correctly. The one with ``test'' in the subject should be in the folder IN.testing and the one without ``test'' in the subject should be in your inbox. If these were delivered correctly go on to step 8. 7b] TROUBLESHOOTING * If the two messages were not delivered correctly look at your ~/.elm/filter-errors to see if you can determine what the problem is. * Check these two files for typos: ~/.forward ~/.elm/filter-rules * Check the file and directory permissions of your .forward (set in 5c above). Type... In order to... ------- -------------- cd Go to your home directory. ls -l .forward Check the permission: it should say -rw-r--r-- ls -ld . Check permission of home dir: it should say drwx?-x?-x The ?'s may be r's or hyphens or one of each (i.e., drwx--x--x, drwxr-xr-x, drwxr-x--x, drwx--xr-x). * If none of these turn up the problem edit your ~/.forward so that filter will be verbose with it's output (use the -vo flag). "|/usr/local/bin/filter -vo /j/nancym/.elm/filter-errors" And repeat steps 6 and 7. After you get filter to work you will probably want to change the ``-vo'' flag back to ``-o''. 8] After you have successfully tested filter in steps 6 and 7, edit ~/.elm/filter-rules so that it contains a modified version of the following: # if (subject contains "test") then save "/j/nancym/Mail/IN.testing" if (to contains "www-talk") then save "/j/nancym/Mail/IN.www-talk" if (to contains "hopos-l") then save "/j/nancym/Mail/IN.hopos" Replace /j/nancym with your home directory path and /Mail with the name of your mail directory. Replace the mailing list string (e.g., "www-talk") and the name of the mail folder (e.g., IN.www-talk) with text for your mailing lists. Note that ``to contains...'' means either the To or Cc header contains... Rather than deleting the test line, it's useful to just turn it into a comment (by preceding it with #) so that you can easily use it for future testing. ------------------------------ Date: 17 Nov 1994 23:03:20 GMT From: Nancy McGough Subject: ... 3.2 Tracking Your Incoming Mail Followup-To: comp.mail.elm You can get a short summary of filter's activity by typing: filter -s For a longer summary type: filter -S Or you can look at the log file itself, ~/.elm/filterlog. You should regularly look at ~/.elm/filter-errors to make sure things are working. You can automatically check filter-errors each time you log in by putting the following in your .login: tail ~/.elm/filter-errors Also you probably want to regularly delete filterlog and filter-errors so they don't fill up your disk space. To get a summary of the filter log and clear it type: filter -cs ------------------------------ Date: 17 Nov 1994 23:03:30 GMT From: Nancy McGough Subject: ... 3.3 Filter References Followup-To: comp.mail.elm Web Page: http://www.myxa.com/elm.html FAQs: http://www.cis.ohio-state.edu/hypertext/faq/usenet/elm/top.html Manual: filter(1) Newsgroup: comp.mail.elm ------------------------------ Date: 17 Nov 1994 23:04:00 GMT From: Nancy McGough Subject: 4.0 Reading Incoming Mail Folders To read an incoming mail folder use your mailer or newsreader (if it can read mail folders). For more information see documentation for your mailer or newsreader. Here are some pointers. PINE ==== FAQ: http://www.cac.washington.edu/pine/faq/ ftp://ftp.cac.washington.edu/pine/docs/faq Manual: pine(1) Newsgroup: comp.mail.pine (linked to Pine mailing list) Mailing List: pine-info@cac.washington.edu (linked to Pine newsgroup) Subscribe to the pine-info mailing list by sending mail to: majordomo@cac.washington.edu With... subscribe pine-info in the body of the message. ELM === Web Page: http://www.myxa.com/elm.html FAQs: http://www.cis.ohio-state.edu/hypertext/faq/usenet/elm/top.html Manual: elm(1) Newsgroup: comp.mail.elm Emacs Mail Mode =============== Newsgroups: gnus.emacs.help and comp.emacs http://www.cis.ohio-state.edu/hypertext/faq/usenet/GNU-Emacs-FAQ/part5/faq.html MH == FAQ: http://www.cis.ohio-state.edu/hypertext/faq/usenet/mh-faq/top.html Manual: mh(1) Newsgroup: comp.mail.mh MAIL ==== Manual: mail(1) Newsgroup: comp.mail.misc NN == FAQ: http://www.cis.ohio-state.edu/hypertext/faq/usenet/nn-faq/top.html Manual: nn(1) Newsgroup: news.software.nn (Does anyone know if nn uses lock files? Is there any problem using nn to read a mail folder that is receiving mail? Please let me know!) (Also, please let me know what other newsreaders can read mail folders?) ------------------------------ Date: 17 Nov 1994 23:05:00 GMT From: Nancy McGough Subject: 5.0 Contributing to this FAQ If you have any corrections, suggestions, or new digest items to contribute to this FAQ please send them to me at nancym@ii.com. If you are reading this with a viewer that understands the following URL you can use it to send me mail: mailto:nancym@ii.com Between official releases of this FAQ the plain text (ascii) version of the most up to date version of it is at: ftp://ftp.halcyon.com/pub/ii/Internet/filtering_mail_faq End of Filtering Mail Digest **************************** -- /\_/\ ( o.o ) Nancy McGough http://www.jazzie.com/ii/ > ^ < Infinite Ink ftp://ftp.halcyon.com/pub/ii