Today and over the next few weeks we will be introducing some changes that will improve the deliverability of forwarded mail. There are two problems that these changes will solve or at least minimize.

We are seeing more systems that are rejecting mail that violates the sender's published SPF policy. Mail from a domain with published a SPF policy that we forward on behalf of our customer may violate the sender's SPF policy because the sender has no way to know that the recipient is forwarding mail to another system. The recipient that is forwarding the mail has no way to know in advance that the sender does not wish mail from the their domain to be forwarded.

In most all cases the rejected mail is spam with a forged sender address but we have seen a small percentage of legitimate mail rejected. It is this small percentage that we would like to see accepted by the system that the mail is forwarded to.

The SPF forwarding problem is well known and the debate about wether SPF breaks forwarding or forwarding is inherently broken still rages on.

To ensure deliveriablity of mail that is forwarded, SRS will be enabled for all forwarded mail. Currently SRS is enabled for mail forwarded to AOL and a few dozen other systems.

Another problem is that more and more systems are rejecting spam that is forwarded to them. When a system that we are forwarding mail to rejects a message, a Delivery Status Notification must be delivered to the sender. If the rejected message is truly spam, and it almost always is, the sender is most likely forged and the DSN is sent to the innocent forgery victim adding to the bad and growing backscatter problem.

Another issue with forwarding spam is that many systems pay attention to what IP addresses are sending them spam in the same way that this system does. The recipient system does not know or care that the spam is forwarded on behalf of a customer. To the receiving system the message is just spam. Too much spam from an IP address and the IP address is blocked. Blame the spammers, not the mail system operators.

To reduce the backscatter that this system generates and to reduce the possibility of our forwarding servers being blocked and preventing mail from being forwarded, messages with a spam score of 6.0 and higher will no longer be forwarded. This change has been in affect for mail to AOL and its subsidiaries since May 2005 with no customer complaints.

Starting today messages scoring 6.0 and over will not be forwarded to adelphia.net, yausi.com, yahoo.com, hotmail.com and cox.net. SRS envelope sender re-writing is also being applied to mail forwarded to those domains.

Along with this change is a real-time report showing forwarded messages that have been discarded.

To: somebody@aol.com
recipient@domain-hosted-here
1 18.2 chancei@ambienttech.com
1 28.5 charlenehipes@asamoacpa.com
1 20.3 daiangelag@0451.com
1 15.5 fpeters@globalfriends.us
1 12.5 gar@directwest.com
1 18.2 gunnersoe@rednet.com
1 12.5 kph@gfj.bdstnetd.com
1 23.9 meeny.beatrix9hm@gmail.com

Select Account -> Real-Time Reports -> Forwarded Discards to view the reports for your account.

Nov 11, 2005 Fastmail box poll errors

The errors customers are seeing for box polls configured to fetch mail from Fastmail are due to a server failure at Fastmail and not due to problems with this system.
Oct 25, 2005 Office telephone outage

Update Oct 31, 2005: Telephone service was restored this past weekend.

We had dialtone and working lines yesterday but this morning nothing. Problem is most likely due to batteries failing in one of the BellSouth fiber huts. We have no estimate to when telephone service will be restored.

Oct 24, 2005 Desktop client access

Late this afternoon a problem with the IMAP and POP multiplexors caused some connections to timeout. Webmail and SMTP services were not affected. The exact time and duration of this problem is lost in the blur of the day resulting from dealing with hurricane Wilma.

Oct 15, 2005 Real-time reports for rejects, virus discards, and deliveries

Real-time reporting of messages rejected by the MX servers, messages infected with viruses that were discarded, and messages delivered to mailboxes was added to the Account Manager last week in an expanded beta test. Real-time reporting is available for hosted email accounts and for will be for MX relay customers later this month.

The last known bug was fixed today and with that fix the delivery reports now include messages accepted for forwarding to external accounts as well as messages that will be delivered to mailboxes. Fixing this bug required deleting all delivery data prior to 1400 EDT, 1800 UTC, today.

Because of the huge number of unknown address rejections due to backscatter from addresses forged by spammers and viruses, the reject reports include only a count of those messages.
Sep 22, 2005 Another Power Failure

Update: All services were back online by 15:55 EDT (19:55 UTC). The outage was caused when the building electricians accidently tripped the main panel breaker while removing the panel cover for further investigation into why the breaker failed yesterday.

The datacenter has contracted with an engineering firm to determine where the weak points are in their power distribution systems and what they can do to improve the power system.

Staring tomorrow evening we will begin moving servers and routers that provide load sharing and redundant routing into another cage. This work will take several days to complete and service outages are expected to be very brief. This work will be done between 20:00 and 00:00 EDT (00:00 - 04:00) UTC tomorrow evening and between 16:00 and 20:00 (20:00 - 00:00 UTC) Saturday and Sunday with the same schedule the following weekend.

Update on the power failure during hurricane Katrina. Lightning struck one of the generators supplying power to the first floor taking out the fuel pump that feeds fuel from the large ground level storage tanks to the generator day tank. The generator continued to operate untill its day tank was empty. To help prevent failures of this nature in the future, additional monitoring and pumps are being installed.

15:10 EDT (19:10 UTC) Power has been restored and services will be back online shortly.
Sep 21, 2005 Power Failure

Update: Power was restored around 12:10 EST (1610 UTC) and IMAP access via the webmail clients was restored shortly after that. Full access to the IMAP and POP servers followed and finally SMTP access was restored. We are still working to bring up the backup raid systems.

The cause of the failure was an overheated 208V 3-phase circuit breaker in the distribution panel that supplies to our racks. The breaker failure was complete and it had to be replaced.

Our second cage in the same datacenter is powered from a different 208V distribution panel. The redundant servers, routing, and switching, will be moved into that cage starting later this week.

We have lost all power to one of our cages at the datacenter. We will be moving enough servers to our second cage to bring services back online.
Sep 9, 2005 Scheduled Maintenance Saturday, Sep 10

Between 0300 and 0400 EDT, 0700 and 0800 UTC, our colocation provider will be upgrading their core routers to Cisco layer 3 switching technology. There will be a few short periods during this time when some customers may not be able to reach our servers.
Aug 26, 2005 Power Outage

The datacenter where our racks are located lost power to the first floor around 2230 EST (0330 UTC) taking out most carriers in the center. Power was restored to the first floor around 2300 EST (0400 UTC).

An update will be posted when we know more about what caused the power outage and what steps will be taken to prevent future occurances.
Aug 17, 2005 Scheduled Maintenance Thursday, Aug 18

Between 0100 and 0130 EDT, 0500 and 0530 UTC, our colocation provider will be testing failover to backup circuits. There will be a few periods of routing instability during this period. We do apologize for the inconvience this will cause for some customers.
Aug 6, 2005 Scheduled Maintenance Sunday, Aug 7

Between 0100 and 0200 EDT, 0500 and 0600 UTC, our colocation provider will be upgrading line cards in one of the border routers. We do apologize for the inconvience this will cause for some customers.
Jul 21, 2005 Scheduled Maintenance Friday, July 22

Between 0030 and 0045 EDT, 0430 and 0445 UTC, our colocation provider will be upgrading line cards in one of the border routers. There will be two periods of brief routing instability while routing is moved from and back to the router being upgraded.
Jul 2, 2005 Scheduled Maintenance Sunday, July 3

Between 1200 and 1230 EDT, 1600 and 1630 UTC, we will be updating the software on the backend IMAP servers in preparation for supporting near real-time replication of IMAP events to the backup IMAP servers.

This update requires restarting the IMAP servers. IMAP and POP3 access will not be available during the update. This update is expected take between 5 and 15 minutes.
May 21, 2005 Effective May 21, 2005, mail that is forwarded to AOL accounts that scores as spam will be silently discarded. This change is necessary to insure that legitimate non-spam mail can continue to be forwarded to AOL accounts by our customers.

Before you push that AOL 'Report Spam' button, please check the message headers to see if the spam was sent directly to your AOL account or forwarded from your domain. Some threshold of spam reports from our IP addresses will result in AOL blocking all mail from those IP addresses.

Apr 29, 2005 Update 23:10 UTC: 50% packet loss persisted for about 15 minutes due to a flapping router interface. There may be a few brief periods of packet loss during the next hour.

22:30 UTC: We are seeing about 50% packet loss at an upstream router. We are working to resolve this issue.

Mar 17, 2005 The routing issue caused 50% to 80% packet loss for about 20 minutes for some customers. One of the routers causing this has been taken offline and its backup has taken over active routing duty. The problem router will be restarted later tonight EST.

We are currently experiencing an internal routing problem at the data center. We hope to have this issue resolved ASAP.

Feb 23, 2005 We are testing new versions of the Squirrelmail and IMP webmail clients at our beta test site. Both clients have many new features and configuration options. Please try these new clients and contact support if you have any questions or comments about these new clients.

Feb 15, 2005 Update: We are accelerating the process of moving customer mailboxes off of this machine and all mailboxes will be moved tonight.

At 1410 EST, 1910 UTC, the ill mail store machine paniced. We expect to have it back online by 1430 EST, 1930 EST.

Feb 9, 2005 At 2330 EST yesterday, 0430 UTC today, we experienced another authentication server problem that affected some customers for approximately 20 minutes. We have changed the configuration to switch to backup servers rather quickly which will reduce the impact to customers if this occurs again. This change will also allow us to leave the failed server in that state for further diagnosis.

Jan 28, 2005 At 1037 EST, 1537 UTC, the authentication server for the mail storage servers stopped responding to authentication requests. Unable to determine the reason for failure we restarted the server at 1055 EST, 1555 UTC, and it is now functioning properly. We will continue to investigate the cause of this failure.

Update on the Jan 26, 2005 power outage.

Batteries for the failed UPS system arrived by truck from Tampa and were installed late yesterday afternoon. By 1830 EST, 2330 UTC, we were back on UPS power.

Jan 26, 2005 Power Outage Update: January 27 0330 EST, 0830 UTC

At 1505 EST, 2005 UTC, a UPS in the data center failed and our systems were without power for approximately 20 minutes. The engineers at the data center attempted to bring the UPS back online without success and utility power was switched to the circuits fed by the failed UPS. When power was restored one router failed to boot and its HSRP backup failed to announce our routes contributing to the problem.

All servers suffered some file system damage requiring another 10 to 90 minutes to repair before the servers could be brought online. Approximately 80 minutes after the power failure the incoming MX servers, the outbound SMTP servers, the MX relay servers, and two mail storage servers were back in service.

The last mail storage server was back online just before 1800 EST, 2300 UTC. The web sites were brought up shortly after all mailbox servers were up and functioning. The Account Manager followed shortly after that.

The UPS failure was not an inverter failure as reported earlier. When the Powerware engineers arrived on site diagnostics showed that the UPS was healthy but the batteries were not up to the task. Throughout today the facility was experiencing short but serve utility power fluctuations causing all of the UPS systems to switch in and out of battery operation and the batteries were not being fully recharged when returning to utility power. This frequent discharge and inadequate recharge caused the batteries in this UPS to fail resulting in this outage.

The data center is expediting delivery of new batteries for this UPS and the other UPS systems in the data center. The new batteries may arrive as early as tomorrow afternoon, Friday.

We sincerely regret the inconvience to our customers caused by this unfortunate outage.