May 28th Server Issues
The server is currently being restored from the monthly May 1 backup. While this is being done we are checking the files from the May 25th backup for the files that caused the error. Once the monthly backup is restored we will move over the files from the May 26th backup.
Restore is still currently running. The files are still moving over
4:00 AM May 28th
We have a total of 78 GB of files from the backup and currently have 21 GB left to move
5:30 AM May 28th
I am working with the data center and the backup service this morning to find a way to transfer the backup files faster
8:25 AM May 28th
I know everyone is wondering how I could be so far off on time estimations.
I have been going off the timeline from my server admin's based on how much data we needed to transfer and the speed of the connection.
Based on a 100mbps connection on our side and the backup service, the entire transfer should have taken only 4 hours.
I tripled the time needed and estimated it at 12 hours.
I did not take into account nor did I know that the backup service was running at less than 10mbps. Our service contract states 100mbps
After talking with the backup service this morning, they agreed to allow us to pay a huge fee to "upgrade" the speed to 100mbps until the restore is complete.
While this will reduce the transfer time greatly, I am unable to give an exact estimate of time, however I will keep posting updates here.
9:50 AM May 28th
I am meeting with a local data center and their server tech's to try to find a faster solution to this immediate issue, then set up a permanent solution that will allow for fail over service if a server is unavailable.
4:21 PM May 28th
We are currently having an issue with the speed of the backup transfer and are trying to find out where the bottleneck is. At this current speed it could take another 18 hours. If we can fix the bottleneck it should only take a few hours.
If you are considering moving to a new hosting company, I will be happy to help you. I just want to make sure everyone is taken care of as quickly as possible. If you sign up with another company and the domain is registered with us, all we need is the name servers for the new host. Just send the information through the contact page and we will verify via a phone call prior to making the change.
11:00 PM May 28th
We have done all we can to make the speed faster, however the speed has still not improved by much. We are looking at an option now of getting the backup service to send our backup via fedex same day air to the datacenter and are awaiting an answer.
5:30 AM May 29
The migration of backup data is still continuing to run. We are anxiously awaiting a reply from the backup service this morning.
2:15 AM May 30
I apologize for not updating sooner. I have spent all of the day manually transferring the system files for the server. If all goes well we will be up late this evening.
8:00 AM May 30
We are still working. Yesterday the data center declined on the fedex transfer stating that "the transfer shouldn't take much longer". I am not sure what the consider "much longer", however this is waaaay longer than I expected. I will try to keep updates going more often today as we progress.
1:40 PM May 30
We currently have 15% of the files left to install. There are literally thousands of tiny files that are transferring. This is causing a very high load on the server which is causing the process to slow a little, however we are at least starting to see progress and are much closer to completion.
I know some of you are trying to call and are frustrated not to get a live person. I can assure you it is in all of our best interest for me to stay focused on completing this restoration. Keeping the backup service and data center motivated and on track along with finding other ways to get the files to the server faster are very important. Every phone message is transcribed to a text and shows up on my screen so I am able to read every message that comes through.
4:10 PM May 30
We are currently at 13% of files left to install.
9:30 PM May 30
We are currently at 11% of files left to install.
1:20 AM May 31
We are currently at 10% of files left to install.
9:15 AM May 31
We are currently still at 10% of files left to install. I sent out requests for any additional ideas to make this last 10% go faster to every server admin I have met or known in the last 5 years. We are trying some of the options/ideas that we received so far this morning.
11:20 PM May 31
We have three companies working with us on this, the data center, the backup company and the admin company. Yet we have still lost another day. Why? I do not know. The communication I am getting is (for lack of a better word) worthless.
I do know that we can no longer just keep waiting, so here is my plan for tomorrow.
A local back-up service -Digital Infrastructures has loaned us a server to set up your accounts on. Considering the expense and quality of this server, I would have to say that this goes beyond the word generous. Thank you to Mark Milroy!
We will be installing this server in the Hillsboro data center first thing in the morning. Our first step will be to set up everyone's accounts and get your email running again. I will personally call each account as we have it set up.
I will post more information about our future plans later in the day. I can tell you however after having lightning strike us twice I will never allow this to happen to us again no matter what the cost.
1:24 June 1
Judging from the amount of emails and phone calls from the people who have already read the above information. I apologize that I must not have been clear on what is I am doing.
Every account is being set up manually. When I get to your account you will get a phone call from me. Please understand, this takes time. However we are talking about hours here, not days.
Just to give you an idea, in the time it took to post this I could have had two accounts already set up.
12:47 AM June 02
We now have a server online and are able to start setting up accounts.
3:53 AM June 03
I have not left you, I am still here setting up accounts. I have two people I trust that will be helping in the morning.
3:53 AM June 04
We are only waiting now for the DNS (Name Servers)to resolve to the new server.
3:45 PM June 06
First I would like apologize for the long laps in communication. For the last 10 years it has been a goal to treat everyone as if you were our only customer. I know the lack of communication now in this critical time is very frustrating for everyone. I can assure you that I have only taken short breaks as necessary to remain alert while continuing to work through this. While I tried calling each account as it was set up, this took so much more additional time due to everyone wanting to vent and let me know how frustrating this is.
There has been an overwhelming number of problems to overcome.
In short this is what has happened so far;
- We had a failed license issue from an upgrade, rolled back the software, then found that the hard drive was failing.
- The data center attempted to fix the hard drive and after 28 hours I asked them to install a new drive so we could restore from our backups.
- The new hard drive was installed and we started restoration only to find out that the operating system installation had errors and had to be reinstalled
- The restoration was started again, and I continued to get "Everything is running great, we will let you know when it is completed". Verbally I got "It should just take a couple more hours".
- While this was going on, that data center set up another server for us to move clients to while we were getting set up with a server locally
After having only 10% of files left to add to that server, they let us know there was another problem on that server and it was taken offline.
- Luckily by that time a local company had loaned us a server to use until our new parts came in.
- We unfortunately had the task of re-installing the operating system and cpanel in this system, which left it unavailable for another day.
- Once this was done, there were issues with a huge lag time in pointing the name servers to the new data center. We were able to figure this out the issue with the help of the data center and our registration company and the name servers are finally pointing properly now.
What's happening now:
The configuration of the new server was completed yesterday. We have added 95% of the accounts to the server with the original user names and passwords.
We are restoring each account from the backup manually one by one. While this takes time, this allows us to verify the data has been moved from the backup.
I would expect that we will still be doing restorations late into the evening on Monday and perhaps Tuesday depending on how everything goes.
If you are one of the lucky ones who created a backup of your website, you should be able to log into your account and upload the backup to restore your website.
You can check to see if your account is online by visiting your domain name. If the account is set up it will show the message Index of / until we restore your files.
Once your website is online, we strongly suggest you do a backup using the backup tool we provide for you in your cpanel account.
June and July are the months that over 70% of accounts do annual renewals. We have suspended all billing until this has been resolved.
What's New:
The new server is set up with Raid 5. This is a multiple hard drive "self healing" setup where if one of the drives starts to fail, the data is written to a waiting drive that takes over so we can replace the failing drive.
Our plan is to upgrade to Raid 6 when possible which will add another level of redundancy to the server.
We will continue to do remote backups.
We are researching the set up of a "Mirrored" fail over server in another local data center. This would automatically take over in the event of a network outage or server problem. If there are admin's reading this that have experience setting up a mirrored fail over solution, please drop us an email through the contact page.
9:15 PM June 06
We are currently doing an update that will improve reliability on the server. You may see a default index page on your domain while this is taking place.
5:12 AM June 07
Some of the accounts on the server are still pointing to the default page for cpanel. We are working directly with cpanel's priority support team to resolve this issue this morning.
We also have 78 accounts left that we are setting up today.
2:39 AM June 08
The cpanel and dns issues earlier today and are continuing to restore files and set up the remaining accounts.
6:26 AM June 09
I am still here and working to finish the restoration process.
9:13 AM June 10
It looks like this should be the last day needed for restoring files. What is taking so long? Each of the account files has to travel over the internet from the backup center to restore on to the server. Every single one of the 538 accounts has to be manually moved then permissions must be set for all of the files that have been moved over. Permission settings are different for several of the directories in the account. Once this is complete, then we have to check the account to make sure it is working. Several accounts have databases that also need to be connected. This all is a painstakingly time consuming process.
If the data center who had the servers fail would send me the hard drives this would be a much faster process. Unfortunately they have cut off communication over a week ago, leaving me with nothing but the monthly backup.
I know it seems like nothing is getting done for the accounts that are not up yet, however I can assure you that I have been working non stop since day one.
5:30 AM June 11
We are still working on some configurations on accounts and database issues today.
3:02 PM June 12
I apologize to everyone, my post for this morning apparently did not save. Some accounts folders are not attached to their account. We are going through each account to verify and link each up properly.
I promise we will not stop until eveyone is back online.
3:02 PM June 17
We have been trying to keep up with all of the issues of databases not being connected and dns issues with sites not resolving to the server. If you are having trouble seeing your website, please check to see if your domains nameservers are pointed to ns1 & ns2.secure10.com or ns1 & ns2.secure33.com. You can check this by going to www.whois.net, and yo can see the nameservers listed at the bottom of the page. If you domain is pointed to the server using an ip address, make sure it is pointed to 96.225.194.201
If you have no idea what the above means, please drop us an email on the contact page and we will help you.
If you need access to cpanel you can also use the ip address http://96.225.194.201/cpanel.
If you were with a reseller, we do not have reseller passwords so your password may have changed when the account was set back up. Please let us know and we will contact your reseller for you.
|
Web Hosting Control Panel 
