[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [sitefinder-tech-discuss Home]
Subject: Re: [sitefinder-tech-discuss] Pseudo code please
At 11:57 AM -0400 10/24/03, Andrew Newton wrote:
>Perhaps its lack of coffee on my part, but are your stats drawing a distinction between unresolvable (as in a lookup failure) vs. non-existent?
Boy, a task master here!
I didn't used to record that information, but you're in luck, I
started to in June. So here is June through September 14 (3.5 months).
And for everyone's sanity, I haven't printed anything with less than
.01% of total bad domains.
I've changed the column order, we have count, % of total
rejects, % of bad domains, % of the particular error type. The
error "Pipelining" means that we rejected the from and they ignored
it and kept going--unfortunately in that case I lost the original
error type. After a little checking I removed domain-too-long errors,
since 99%of them appear to have come from a single mailbomb attack and
it was accounting for 25% of my rejected froms. The rest of the error
messages are all those I pulled from the log files. The logs were
generated by the CommuniGate Pro mail system--if you can call something
that does LDAP, Calendar, POP, IMAP, Radius, Web hosting, DAV, FTP and
who knows what else a "mail" system--from Stalker. The version I parsed
for these stats is created nightly by a script I run which turns
CommuniGate's SMTP logs into something that looks like a web server
log (since there are lots of tools out there for analyzing web logs).
One other note. If you look here you'll see that total bad domains
account for only 4% of rejected email. That's a lot lower than my 18 month
sample, where it was 8%. The reason is SoBig-F. Somewhere's reject
rate has been doubling every year. Last year we rejected 10 million
messages. This year we were on target for about 20 million. We were bouncing
roughly 30-40 messages a minute prior to SoBig-F. During SoBig
however, that rate jumped to 600-700 messages a minute. So for this
three and a half month period we see 17 million rejected messages
when I would have expected something between 6 and 10. It would have
been even higher, but I throttled the number of connections. I haven't
tried to figure it out exactly, but if that's the case, then it
makes sense to multiply the "% total" numbers by at least 2.
So, cutting to the chase. Percentage of somewhere.com spam with no
A record is a minimum of 2%, and probably at least 4%. Here are the
numbers.
count % total % bad % type TLD Error Type
-------- ------- ------- ------- --- ----------
35 0.00% 0.01% 0.79% ar DNS A-record is empty
73 0.00% 0.01% 1.65% uk .
182 0.00% 0.03% 4.11% net .
342 0.00% 0.05% 7.71% edu .
3700 0.02% 0.54% 83.46% com .
4433 0.02% 0.65% 100.00% ----- DNS A-record is empty
34 0.00% 0.01% 0.07% sk DNS server failure
40 0.00% 0.01% 0.08% tk .
42 0.00% 0.01% 0.08% za .
54 0.00% 0.01% 0.11% cc .
60 0.00% 0.01% 0.12% tw .
62 0.00% 0.01% 0.12% br .
69 0.00% 0.01% 0.13% de .
82 0.00% 0.01% 0.16% ar .
91 0.00% 0.01% 0.18% gr .
97 0.00% 0.01% 0.19% uk .
108 0.00% 0.02% 0.21% au .
143 0.00% 0.02% 0.28% mx .
146 0.00% 0.02% 0.28% cl .
208 0.00% 0.03% 0.41% kr .
299 0.00% 0.04% 0.58% ve .
485 0.00% 0.07% 0.95% cn .
545 0.00% 0.08% 1.06% info .
554 0.00% 0.08% 1.08% hr .
567 0.00% 0.08% 1.11% biz .
678 0.00% 0.10% 1.32% us .
691 0.00% 0.10% 1.35% nu .
3809 0.02% 0.56% 7.43% org .
8061 0.05% 1.19% 15.72% net .
34018 0.19% 5.01% 66.35% com .
51271 0.29% 7.55% 100.00% ----- DNS server failure
35 0.00% 0.01% 0.09% ru DNS transaction timeout
39 0.00% 0.01% 0.10% ie .
39 0.00% 0.01% 0.10% hu .
40 0.00% 0.01% 0.10% info .
51 0.00% 0.01% 0.13% mil .
52 0.00% 0.01% 0.13% de .
61 0.00% 0.01% 0.16% yu .
65 0.00% 0.01% 0.17% uk .
74 0.00% 0.01% 0.19% za .
76 0.00% 0.01% 0.20% br .
150 0.00% 0.02% 0.39% kr .
189 0.00% 0.03% 0.49% edu .
196 0.00% 0.03% 0.50% cn .
283 0.00% 0.04% 0.73% ar .
309 0.00% 0.05% 0.79% ca .
447 0.00% 0.07% 1.15% org .
616 0.00% 0.09% 1.58% biz .
725 0.00% 0.11% 1.86% pl .
732 0.00% 0.11% 1.88% pk .
1225 0.01% 0.18% 3.15% tw .
1949 0.01% 0.29% 5.01% ve .
2442 0.01% 0.36% 6.28% us .
6085 0.03% 0.90% 15.64% net .
22595 0.13% 3.33% 58.06% com .
38916 0.22% 5.73% 100.00% ----- DNS transaction timeout
34 0.00% 0.01% 1.75% cn Incorrect Address Format
35 0.00% 0.01% 1.81% de .
38 0.00% 0.01% 1.96% tw .
41 0.00% 0.01% 2.12% tw, .
48 0.00% 0.01% 2.48% c .
175 0.00% 0.03% 9.03% uk .
199 0.00% 0.03% 10.27% net .
1099 0.01% 0.16% 56.71% com .
1938 0.01% 0.29% 100.00% ----- Incorrect Address Format
37 0.00% 0.01% 0.51% de Pipelining
37 0.00% 0.01% 0.51% kr .
49 0.00% 0.01% 0.67% cn .
54 0.00% 0.01% 0.74% org .
63 0.00% 0.01% 0.86% uk\t .
75 0.00% 0.01% 1.03% tw .
84 0.00% 0.01% 1.15% yahoo .
101 0.00% 0.01% 1.38% txx\t .
147 0.00% 0.02% 2.01% pw .
378 0.00% 0.06% 5.17% tx .
479 0.00% 0.07% 6.55% ru .
528 0.00% 0.08% 7.22% net .
630 0.00% 0.09% 8.62% br .
4453 0.03% 0.66% 60.92% com .
7309 0.04% 1.08% 100.00% ----- Pipelining
57 0.00% 0.01% 5.59% net Unknown Error
117 0.00% 0.02% 11.47% andy .
142 0.00% 0.02% 13.92% c .
149 0.00% 0.02% 14.61% comnce .
259 0.00% 0.04% 25.39% com .
1020 0.01% 0.15% 100.00% ----- Unknown Error
35 0.00% 0.01% 0.41% tw bad Q data format in DNS response
37 0.00% 0.01% 0.43% cn .
38 0.00% 0.01% 0.45% it .
61 0.00% 0.01% 0.72% uk .
74 0.00% 0.01% 0.87% il .
86 0.00% 0.01% 1.01% us .
122 0.00% 0.02% 1.43% org .
443 0.00% 0.07% 5.20% info .
467 0.00% 0.07% 5.48% hr .
609 0.00% 0.09% 7.15% net .
1112 0.01% 0.16% 13.05% edu .
5235 0.03% 0.77% 61.46% com .
8518 0.05% 1.25% 100.00% ----- bad Q data format in DNS response
536 0.00% 0.08% 96.23% com failed to route the address
557 0.00% 0.08% 100.00% ----- failed to route the address
34 0.00% 0.01% 0.02% pyrenees host name is unknown
35 0.00% 0.01% 0.02% loc .
35 0.00% 0.01% 0.02% hitslink .
36 0.00% 0.01% 0.02% deom .
37 0.00% 0.01% 0.02% specificpop .
37 0.00% 0.01% 0.02% bsd .
38 0.00% 0.01% 0.02% a .
39 0.00% 0.01% 0.02% ol .
39 0.00% 0.01% 0.02% proprio .
40 0.00% 0.01% 0.02% nattle .
40 0.00% 0.01% 0.02% coremetrics .
41 0.00% 0.01% 0.02% gator .
41 0.00% 0.01% 0.02% no .
42 0.00% 0.01% 0.02% hh .
44 0.00% 0.01% 0.02% worm .
44 0.00% 0.01% 0.02% za .
46 0.00% 0.01% 0.02% at .
47 0.00% 0.01% 0.03% xxx .
47 0.00% 0.01% 0.03% nett .
48 0.00% 0.01% 0.03% nao .
49 0.00% 0.01% 0.03% webtrendslive .
49 0.00% 0.01% 0.03% cs .
50 0.00% 0.01% 0.03% dom .
50 0.00% 0.01% 0.03% qksrv .
50 0.00% 0.01% 0.03% gbl .
50 0.00% 0.01% 0.03% valuead .
52 0.00% 0.01% 0.03% il .
53 0.00% 0.01% 0.03% b .
54 0.00% 0.01% 0.03% hk .
55 0.00% 0.01% 0.03% by .
55 0.00% 0.01% 0.03% xx .
56 0.00% 0.01% 0.03% campus .
56 0.00% 0.01% 0.03% tr .
57 0.00% 0.01% 0.03% be .
57 0.00% 0.01% 0.03% d .
60 0.00% 0.01% 0.03% n .
61 0.00% 0.01% 0.03% adserver .
61 0.00% 0.01% 0.03% nz .
63 0.00% 0.01% 0.03% uk\t .
64 0.00% 0.01% 0.03% internetfuel .
64 0.00% 0.01% 0.03% arpa .
64 0.00% 0.01% 0.03% my .
76 0.00% 0.01% 0.04% prv .
77 0.00% 0.01% 0.04% dr .
78 0.00% 0.01% 0.04% ccil .
79 0.00% 0.01% 0.04% mx .
79 0.00% 0.01% 0.04% in .
81 0.00% 0.01% 0.04% nn .
86 0.00% 0.01% 0.05% mmm .
86 0.00% 0.01% 0.05% se .
88 0.00% 0.01% 0.05% hlp .
90 0.00% 0.01% 0.05% enc .
91 0.00% 0.01% 0.05% ar .
93 0.00% 0.01% 0.05% abca .
97 0.00% 0.01% 0.05% sg .
99 0.00% 0.01% 0.05% class .
101 0.00% 0.01% 0.05% txx\t .
102 0.00% 0.02% 0.06% trans .
104 0.00% 0.02% 0.06% pl .
104 0.00% 0.02% 0.06% yahoo .
107 0.00% 0.02% 0.06% abcb .
108 0.00% 0.02% 0.06% gov .
112 0.00% 0.02% 0.06% con .
117 0.00% 0.02% 0.06% sdfsdf .
119 0.00% 0.02% 0.06% rt .
119 0.00% 0.02% 0.06% nl .
123 0.00% 0.02% 0.07% ne .
130 0.00% 0.02% 0.07% pt .
132 0.00% 0.02% 0.07% abc .
133 0.00% 0.02% 0.07% gif .
147 0.00% 0.02% 0.08% pw .
147 0.00% 0.02% 0.08% world .
147 0.00% 0.02% 0.08% ru .
148 0.00% 0.02% 0.08% cz .
153 0.00% 0.02% 0.08% server .
158 0.00% 0.02% 0.09% ini .
160 0.00% 0.02% 0.09% email .
165 0.00% 0.02% 0.09% advertising .
167 0.00% 0.02% 0.09% fr .
170 0.00% 0.03% 0.09% gs .
181 0.00% 0.03% 0.10% ch .
196 0.00% 0.03% 0.11% cr .
196 0.00% 0.03% 0.11% cnd .
201 0.00% 0.03% 0.11% mil .
204 0.00% 0.03% 0.11% int .
204 0.00% 0.03% 0.11% es .
228 0.00% 0.03% 0.12% jp .
233 0.00% 0.03% 0.13% andy .
238 0.00% 0.04% 0.13% it .
270 0.00% 0.04% 0.15% kr .
278 0.00% 0.04% 0.15% given .
288 0.00% 0.04% 0.16% vtabachmv .
291 0.00% 0.04% 0.16% comm .
312 0.00% 0.05% 0.17% sextracker .
318 0.00% 0.05% 0.17% au .
351 0.00% 0.05% 0.19% jpg .
357 0.00% 0.05% 0.19% hitbox .
364 0.00% 0.05% 0.20% c .
383 0.00% 0.06% 0.21% tx .
419 0.00% 0.06% 0.23% biz .
438 0.00% 0.06% 0.24% ca .
522 0.00% 0.08% 0.28% htm .
527 0.00% 0.08% 0.29% tw .
571 0.00% 0.08% 0.31% edu .
599 0.00% 0.09% 0.32% html .
651 0.00% 0.10% 0.35% co .
691 0.00% 0.10% 0.37% us .
718 0.00% 0.11% 0.39% mail .
823 0.00% 0.12% 0.45% om .
1091 0.01% 0.16% 0.59% org .
1141 0.01% 0.17% 0.62% br .
1541 0.01% 0.23% 0.83% cn .
1939 0.01% 0.29% 1.05% fi .
2068 0.01% 0.30% 1.12% uk .
3295 0.02% 0.49% 1.78% ent .
5115 0.03% 0.75% 2.77% net .
24598 0.14% 3.62% 13.32% de .
39879 0.22% 5.87% 21.59% localdomain .
81760 0.46% 12.04% 44.27% com .
184704 1.04% 27.19% 100.00% ----- host name is unknown
36 0.00% 0.01% 0.01% no no DNS A-data returned
37 0.00% 0.01% 0.01% ch .
43 0.00% 0.01% 0.01% uy .
45 0.00% 0.01% 0.01% nz .
47 0.00% 0.01% 0.01% pl .
48 0.00% 0.01% 0.01% ar .
67 0.00% 0.01% 0.02% tr .
109 0.00% 0.02% 0.03% ru .
113 0.00% 0.02% 0.03% lt .
141 0.00% 0.02% 0.04% mx .
160 0.00% 0.02% 0.04% ba .
166 0.00% 0.02% 0.04% ua .
179 0.00% 0.03% 0.05% ca .
189 0.00% 0.03% 0.05% kr .
192 0.00% 0.03% 0.05% ae .
205 0.00% 0.03% 0.05% jp .
235 0.00% 0.03% 0.06% info .
245 0.00% 0.04% 0.06% arpa .
246 0.00% 0.04% 0.06% biz .
270 0.00% 0.04% 0.07% cn .
285 0.00% 0.04% 0.08% co .
339 0.00% 0.05% 0.09% us .
444 0.00% 0.07% 0.12% be .
563 0.00% 0.08% 0.15% sk .
566 0.00% 0.08% 0.15% br .
591 0.00% 0.09% 0.16% tw .
714 0.00% 0.11% 0.19% hr .
719 0.00% 0.11% 0.19% mil .
857 0.00% 0.13% 0.23% au .
902 0.01% 0.13% 0.24% it .
1380 0.01% 0.20% 0.36% il .
1553 0.01% 0.23% 0.41% nl .
1696 0.01% 0.25% 0.45% de .
1734 0.01% 0.26% 0.46% org .
2165 0.01% 0.32% 0.57% uk .
3018 0.02% 0.44% 0.79% edu .
45503 0.26% 6.70% 11.97% net .
313981 1.76% 46.23% 82.63% com .
379990 2.13% 55.94% 100.00% ----- no DNS A-data returned
58 0.00% 0.01% 13.94% net syntax error
58 0.00% 0.01% 13.94% gov .
288 0.00% 0.04% 69.23% com .
416 0.00% 0.06% 100.00% ----- syntax error
35 0.00% 0.01% 100.00% ----- syntax error in MAIL FROM parameters
679243 3.81% 100.00% 100.00% ----- Total Bad Domains
17806723 100.00% ------- 100.00% ----- Total Rejects
-------- ------- ------- ------- --- ----------
count % total % bad % type TLD Error Type
--
Kee Hinckley
http://www.messagefire.com/ Next Generation Spam Defense
http://commons.somewhere.com/buzz/ Writings on Technology and Society
I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
----------------------------------------------------------------
To subscribe or unsubscribe from this elist use the subscription
manager: <http://lists.elistx.com/unsubscribe>
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [sitefinder-tech-discuss Home]
Powered by eList eXpress LLC