Lunarpages Web Hosting Forum

Author Topic: server ZANE down ?  (Read 28355 times)

Offline poverbeeke

  • Trekkie
  • **
  • Posts: 10
Re: server ZANE down ?
« Reply #210 on: December 18, 2006, 09:48:51 PM »
Lunarpages,

WHY did at&t block?

By mail you said :
"We have removed the problem however AT&T needs to remove the block"
Meilena Hauslendale
Customer Service Representative

I was able to take out my domain. And I want to move it back. But OK, what was the problem? Who was the client who misbehaved? And what happens in the future? Are you checking the behaviours on the server?

There are a lot of unanswered questions... Your behaviour scares the hell out of me. I have to answer to others too, I can't give them a reasenable answer.

So, please, answer. What with future same problems? How are you going to avoid it?

Priest

  • Guest
Re: server ZANE down ?
« Reply #211 on: December 18, 2006, 11:11:11 PM »
Hello,

Having completed an investigation on this matter and all your questions, I can post the following information (warning, its a long one):

The issues with the Zane server being blocked by AT&T were caused by a phishing scam that was being operated on the server.

Generally, when this sort of incident occurs, the abuse team is notified by the party that discovered the phishing scam and they promptly remove the scam and deal with the issues on the account that caused this to occur.  However, it appears that AT&T did not follow that generally accepted practice and instead decided to place a block on the IP address of the server.  This could have been done for any number of reasons, most likely to ensure their ISP customers were unable to be scammed via the site.

This case is an example of why it is extremely important for any customer on a shared server to update their scripts frequently.  Belonging to a mailing list for the script they are using, checking the script's site, etc are important ways to find out about security related issues with an installed script.  Had a customer's script not been exploited, the server would never have become blocked (Please do not ask who it was, we cannot reveal private customer information).

Once the reason for the block was relayed to us, the scam was quickly removed and the remainder of the time was waiting for AT&T to remove the block.  This frequently happens to servers for email related abuse, though the block is restricted to email service to the blockers network.  These types of blocks are generally removed in less than 24 hours.  This was the first time in my ~3 years here at Lunarpages that an entire server IP had been blocked by a router on the internet backbone. Once we request the block be lifted, all we can do is wait until the blocker lifts the ban.

It has been asked why the server of the IP was not changed.  A server's shared IP is tied into every domain hosted on the server in apache configuration files that must be updated, cpanel data that must be updated, email and ftp daemon configurations, etc.  It is also tied into every DNS entry for every domain, subdomain, addon domain, and parked domain on an account.  It must be updated on both nameservers in addition to the server.  It is not a simple task and requires a great deal of planning to ensure that all changes are made in an appropriate sequence and that no configurations are screwed up by the change, missed in the process, and that they actually take hold.

It has been asked what will happen in the future.  Since this is the first time this has occured, there is no way to know.  Most companies report the offending site to the admin of the server it is on so it can be handled internally.  If companies continue to follow the example of emailed each others abuse teams, this issue should not occur again.  However, there is always the possibility that it will.  This cannot be helped in a shared server environment and we cannot control every aspect of what each account has installed.  That responsibility falls to the owner of each shared account.

It has also been asked why we don't scan servers for this type of issue and resolve them auto-magically.  Quite frankly, this is impossible.  There are so many 3rd party and custom written scripts out there that it would be inconceivable to try and manage what is and what is not permitted to be installed or in use.  To even begin to consider this, we would have to have a list of scripts that are permitted.  Any script not on the list would be disabled.  This is impractical because new versions of scripts come out, so versioning would have to be used.  This means we would also have to disable older versions when new ones came out, which would undoubtedly cause customer complaints.  It is also far too limiting for our customer to be confined to a limited set of scripts, say no more than 5 installations per account and they have to be chosen from this list of say 15 allowed scripts.

Finally, compensation has been requested in a couple of posts.  There are no plans for any form of compensation as there was no actual issue with the server or our network.  Sparing you all more analogies, it would not be appropriate to demand that company A compensate loss due to the activities of company B when the services offered by company A have not been diminished by its own activity, or lack thereof.  This block effected us as much as it has effected our customers.  Several members of our own support staff were unable to access the server for customer services related issues.

The statement in the move email, 'this is a one time offer only', has also been criticized.  Quite frankly, we cannot expend the staff to move every effected account off a server any time a block occurs.  There is much more to a server move than clicking a few buttons.  Account information must be updated, the old account must be compared to the new to ensure that all data and databases moved correctly, DNS entries must be checked to ensure they have updated, etc.  It is a time consuming process.

It should again be noted that there was nothing wrong with the server itself or our internal network, the majority of people could still access the server without issue.  This would mean that changing the shared IP of the server or moving all accounts to another server would cause the server to appear offline to everyone.  In shared server hosting, a balance must be struck.  We offered moves to those of you that could not reach the server out of generosity, we understand what it is to not be able to get to your site.

I hope this has been informative and should address all questions or concerns raised in this thread.

~Priest

Offline poverbeeke

  • Trekkie
  • **
  • Posts: 10
Re: server ZANE down ?
« Reply #212 on: December 18, 2006, 11:49:57 PM »
Thank you, Priest,

It's a clear answer. It works for me.
Understeand our point of view. The answer that a scam was found could have been given earlier. It could have saved you a lot of pain. You could have published the at&t contact, so we could send the unblock-question too.

I have been promoting Lunarpages, so I have to answer too...

On the nebula-server, a few years ago, there was a ddos-problem. It was solved perfectly as it never happened again. I hope this zane-problem will never happen again also.

Please, don't say too fast that it is always possible on a shared server. It's the same as saying: "better not professionally use Lunarservers".

Monitoring the server: if it is not possible, please do the impossible... there must be ways of comparing server-content with scam-reports... Or a big button on the main LP-website to report inside-abuse.

Peter

Offline mark worthington

  • Space Explorer
  • ***
  • Posts: 6
Re: server ZANE down ?
« Reply #213 on: December 19, 2006, 12:36:13 AM »
Hello,

Having completed an investigation on this matter and all your questions, I can post the following information (warning, its a long one):

....

The issues with the Zane server being blocked by AT&T were caused by a phishing scam that was being operated on the server.I hope this has been informative and should address all questions or concerns raised in this thread.

~Priest

Priest,

Thanks for the informative and professional feedback. I agree with all you say, personally. If I had a real business, one that depended on my host, I would not use a shared server anyway, and have a mirror "backup" host as well. But then I don't know exactly what I'm saying, technically, so that might be tosh!

In the spirit of long posts, I'll continue …. At the beginning of December I was convinced that the time had come to pay for a good web host, and after much research I chose Lunarpages. This was triggered by Cabspace, my free web host, going totally off-line for over a week. They came back, with this message :

"The CabSpace.com domain was shut off by the registrar (DirectNIC) on November 20th, 2006 because of an abuser using the service to phish MySpace.com passwords. We were not given warning to remove the abuser.

When we requested that the domain name be reactivated, their response was that cabspace.com would not be reactivated because it was associated with illegal activity, and we were fined $1000. After explaining our situation as a webhost, and that the illegal activity in question was performed by a criminal user, they agreed on late November 22nd to reactivate it this one time. They would do this after we wired them $1000.

Since this was after 4:30 PM EST on Wednesday before Thanksgiving, we were unable to send the wire until after the holiday weekend. DirectNIC acknowledged receiving the wire, and CabSpace.com has been reinstated as of November 28th, 2006.

We understand users' frustration in not knowing what was going on over the course of the week. Unfortunately, we didn't know what the outcome was exactly going to be most of the week either. And you can imagine our frustration of having to pay $1000 for a service we provide for free, and having a week of downtime for 12,000 users, all because some inconsiderate lowlife wants to wreak havoc on somebody's myspace page. Stupid. What a waste.

So what are we doing to prevent it from happening again?

First, we are most likely going to switch registrars. After making our case, DirectNIC was quick and efficient at handling our request. But the $1000 fine per instance is pretty steep, and we have yet to find any record of them sending us an alert email (that they state in their terms will be sent) 5 days before taking action.

But changing registrars doesn't remove the primary reason for these troubles. Selfish idiots. Phishers, scammers and spammers. If you know anyone that participates in these activities, please let them know that the consequences of their actions affect everyone. The are the #1 reason so many free services have fallen by the wayside.

Anyways, we regret the recent downtime, and look forward to a long future ahead. Let us know your suggestions to do better in the future."


So, it must be following me about  :smiling: But seriously, this highlights the problems the internet and web hosts face, and makes the excellent point that self-regulation is critical.

I must say that in my experience, Forums are life-savers … in this example, it was user-feedback, primarily via the Forum, that brought this problem to Lunarpage's attanetion, I believe.

So, LP, I would think long and hard about this and heartily concur with poverbeeke, that "a big button on the main LP-website to report inside-abuse" is a very good idea. At least it will give comfort to customers, and provide a simple, single point-of-entry for abuse reporting.

Regards,

Mark

Priest

  • Guest
Re: server ZANE down ?
« Reply #214 on: December 19, 2006, 01:36:06 AM »
The answer that a scam was found could have been given earlier. It could have saved you a lot of pain. You could have published the at&t contact, so we could send the unblock-question too.
...
Please, don't say too fast that it is always possible on a shared server. It's the same as saying: "better not professionally use Lunarservers".
...
Monitoring the server: if it is not possible, please do the impossible... there must be ways of comparing server-content with scam-reports... Or a big button on the main LP-website to report inside-abuse.
The Information regarding the phishing page was actually new information.  If I had the information before this point, I most definitely would have provided it to our customers.

Unfortunately, the statement that this is always a possibility is simply the truth.  When you have an environment with so many people running so many different things, there is no way to guarantee this won't occur again.  To say that it would never occur again would be a deception all to itself.  But that is also why we provide VPS and Dedicated plans for those professional sites that require a way to be separated from other customers on the same server (or operating system as it were in the case of VPS plans)

There is no way possible to actively scan all scripts that may be in use on a server.  The differences in scripting languages, programming styles, version information, etc is too much to effectively handle programmatically.  I would imagine the code the this beast alone would be horrific in its complexity and the resources required to operate on all accounts would likely be in the realm as to make the server unusable.  This is really where the owner of the account comes into play.  If the owner of every account maintained all scripts and secured their sites and code, these issues would not occur.  You can always email abuse related issues such as phishing sites to abuse@lunarpages.com

If that fails for any reason, an email sent to support@lunarpages.com will also find its way to the Customer Support Supervisors (such as myself) and our abuse team.
« Last Edit: December 19, 2006, 01:38:35 AM by Priest »

Offline wektech

  • Master Jedi
  • *****
  • Posts: 1038
    • Yuma Arizona Information
Re: server ZANE down ?
« Reply #215 on: December 19, 2006, 06:42:54 AM »
:yikes: Although my site was not impacted by this outage, I am now wondering how best to protect my site from this type of problem. As the revenues from the site do not even cover current hosting costs, I am not willing to spend a lot of money. It seems under these conditions that the best protection would be to have a dedicated IP address assigned to my account. Does LP still offer the dedicated IP address service at $2.50 a month? I am not sure as there is no link on the main site explaining this add on. I read in the forum the cost is 2.50 per month.
What is entailed in assigning a dedicated IP address? Does it require a change in the server used (in my case fyodor)? Are there any inherent issues with using subdomains and add on domains with a dedicated IP address? 

Offline solar

  • Intergalactic Superstar
  • *****
  • Posts: 183
Re: server ZANE down ?
« Reply #216 on: December 19, 2006, 12:16:27 PM »
Wanted to say that Lunarpages jumped right in for us when a support ticket was submitted. The service was excellent.

Wekkie, the dedicated IP is easy...just send in a support ticket requesting one. We have done this  many times, and once quite recently. However, I believe I'm correct in saying that a dedicated IP wouldn't prevent the problem AT&T caused by their impossibly slow response.

I am hoping it's not the way of the future...it's clear that AT&T and other giants have a master plan for restricting web access with tiered pricing. Write your congressperson. Send those cards and letters and spread the word online. It can happen and things could get much worse.

In the meantime, Lunarpages has our vote for the best of everything...the best packages at the least cost, combined with the single most important feature: great customer support. :yey: We have business websites running on shared servers at Lunarpages and they perform awesomely. Clients love the cost-saving aspect and performance.

Lupine1647

  • Guest
Re: server ZANE down ?
« Reply #217 on: December 19, 2006, 12:56:08 PM »
However, I believe I'm correct in saying that a dedicated IP wouldn't prevent the problem AT&T caused by their impossibly slow response.

Actually, I believe a dedicated IP address can solve situations such as these as AT&T wasn't blocking a dedicated IP address, but they were blocking the shared server's IP address. Other addresses such as the login.zane.lunaservers.com were still working as that address has a different IP address than the shared IP.

bryantrv

  • Guest
Re: server ZANE down ?
« Reply #218 on: December 19, 2006, 02:13:55 PM »
Thanks Mr. Priest for the "lowdown" on the situation. It does seem very odd that AT&T would simply block the IP address rather than contacting the host, with a CC to the webmaster- when a rare phishing email gets through my spam filters, I often visit the site and if it's an obviously legit site with the phishing files buried deep down, I fire off an email to webmaster@ alerting them to the scam (though I've never had a response to date).

While a dedicated IP would be a solution, realistically, there is a very finite number of IP addresses (until ipv6 gets more common- I'm not sure how I feel about that)- while I would be upset if my site was down for several days, this is such a rare happening that I'm not sure if my piddly little site warrants using up a whole IP address just for it ;).

RandyT

  • Guest
Re: server ZANE down ?
« Reply #219 on: December 19, 2006, 02:53:17 PM »
Quote
Does LP still offer the dedicated IP address service at $2.50 a month? I am not sure as there is no link on the main site explaining this add on. I read in the forum the cost is 2.50 per month.

Yes. $2.50 per month .Info on add ons found here:

http://desk.lunarpages.com/faq.php?do=article&articleid=31

Hope this helps,
RandyT

katrina1

  • Guest
Re: server ZANE down ?
« Reply #220 on: December 19, 2006, 10:13:40 PM »
During the block, sites with dedicated IP addresses were accessible on zane. However, since the mail server runs from the main server IP, they also had issues with email just as the shared IP customers did. The login.zane.lunarpages.com access to webmail was still functioning for everyone however.

Offline BMW_CHUBB

  • Intergalactic Cowboy
  • *****
  • Posts: 69
Re: server ZANE down ?
« Reply #221 on: January 27, 2007, 12:14:03 PM »
So whats this error mean :-?

Since the zane server block i have been running sweet ! But this error occurs.  :?

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@bmwchubbsdesigns.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.


Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.


--------------------------------------------------------------------------------

Apache/1.3.37 Server at www.bmwchubbsdesigns.com Port 80

Lupine1647

  • Guest
Re: server ZANE down ?
« Reply #222 on: January 27, 2007, 01:07:44 PM »
BMW,
You may want to check out this thread about 500 errors: http://www.lunarforums.com/forum/index.php?topic=20444.0

Offline BMW_CHUBB

  • Intergalactic Cowboy
  • *****
  • Posts: 69
Re: server ZANE down ?
« Reply #223 on: August 04, 2007, 12:52:26 PM »
BMW,
You may want to check out this thread about 500 errors: http://www.lunarforums.com/forum/index.php?topic=20444.0

Yup checked  and repaired  and now it returns :?

Lupine1647

  • Guest
Re: server ZANE down ?
« Reply #224 on: September 09, 2007, 10:07:12 AM »
Whoa, a blast from the past. What does the error log say?