First of all, I'd like to personally apologize for the problems we've been having with database corruption errors. As you might imagine, it has been frustrating for us, and I'm sure it's been more than an inconvenience for y'all, as posts, threads and even registrations have been lost upon restoration from backups.
The Problem
The primary problem has been with corrupt tables in the database. What's causing the tables to become corrupt isn't fully clear, and that's part of the frustration. Chuck, Jon Paul and others have spent many hours trying to determine the root cause, and it appears to be a problem with improper reboots and shutdowns of the server, which can result in the corruption. Unfortunately, we can't tell what is causing the shutdowns.
Short Term Fix
Unfortunately, the database tables become corrupt beyond repair, and when that happens, it has to be restored from backups. Fortunately, we have automatic daily backups, but they generally occur overnight, sometime around 2 am. Having to revert to even the previous backup can sometimes result in significant data loss.
Additional Complications
Compounding the problem is that the first occurrence happened right near the beginning of hunting season. We had hoped that it would be a one time occurrence, but obviously that wasn't the case. We've had multiple outages that happened throughout the season. Not wanting to make significant changes during our peak usage (hunting season), we hoped that the issue would ultimately correct itself.
Searching for a Permanent Solution
As you might imagine, making wholesale changes to the server can be both costly and time consuming. We attempt to avoid that until all other options are exhausted. We've attempted to find a solution that we could apply to the existing hardware with the existing host, but we've been unable to do so.
Fortunately, we've had the luxury of having outside expertise to help us in each of the previous outages, in the form of Jon Paul's friend, Scott. He has spent considerable hours following each database error researching and restoring the database from backups.
We've made the decision to take advantage of Scott's server, database and vBulletin expertise by upgrading and migrating the entire site over to a server hosted by Scott's company's datacenter.
As mentioned, moving to a new server and a new host for the site will require considerable investment in time and finances, but will afford us the expertise of Scott's team, which should result in vastly improved site performance and uptime, as well as provide an upgrade to the hardware that will provide room for continued growth.
The Timeframe
We've been working with Scott and his team to determine hardware specifications (for now and to allow for growth), and Scott ordered the hardware earlier this week. Once it arrives (in the weeks following Bownanza), Scott's team will assemble the server, and then begin the migration process. Most of the work will be performed in advance of the migration, and the migration itself should require very little, if any, downtime. However, there may be occasions where we need to take the site or the forum offline for temporary periods.
What it Means to You
As mentioned, it will still take a few weeks to receive the equipment and get it configured before beginning the migration. Unfortunately, that means that we are still susceptible to the same potential problems, including database corruption (and loss of data associated with restoring from backups). As such, I would suggest if you compose a lengthy post or thread, that you save that thread elsewhere in the event that it is lost. Scott has graciously volunteered to "babysit" the site while it remains on the existing server, even though he has no affiliation with the host, or the site for that matter. As you might imagine, I'm both apprehensive and excited about the forthcoming changes!
In addition, we have also been evaluating upgrades to the discussion software. Soon after the migration of the server, we will begin implementing those changes as well. Having tested the new features on another forum, I'm extremely excited about the changes associated with that upgrade, as well!
Thank You
Finally, I'd like to reiterate how grateful I am for all of our members that make this site what it is. Obviously, many have been frustrated with the downtime and lost data. Most of you have remained patient as we have attempted to resolve the issues. Others, perhaps, not so much, but it might not be a bad idea to allow those members that seemingly no longer find enjoyment in our little community to cull themselves from it.
We all look forward to even bigger and better things from our site and its members! Thank you for your continued patience and feedback.
Michael
The Problem
The primary problem has been with corrupt tables in the database. What's causing the tables to become corrupt isn't fully clear, and that's part of the frustration. Chuck, Jon Paul and others have spent many hours trying to determine the root cause, and it appears to be a problem with improper reboots and shutdowns of the server, which can result in the corruption. Unfortunately, we can't tell what is causing the shutdowns.
Short Term Fix
Unfortunately, the database tables become corrupt beyond repair, and when that happens, it has to be restored from backups. Fortunately, we have automatic daily backups, but they generally occur overnight, sometime around 2 am. Having to revert to even the previous backup can sometimes result in significant data loss.
Additional Complications
Compounding the problem is that the first occurrence happened right near the beginning of hunting season. We had hoped that it would be a one time occurrence, but obviously that wasn't the case. We've had multiple outages that happened throughout the season. Not wanting to make significant changes during our peak usage (hunting season), we hoped that the issue would ultimately correct itself.
Searching for a Permanent Solution
As you might imagine, making wholesale changes to the server can be both costly and time consuming. We attempt to avoid that until all other options are exhausted. We've attempted to find a solution that we could apply to the existing hardware with the existing host, but we've been unable to do so.
Fortunately, we've had the luxury of having outside expertise to help us in each of the previous outages, in the form of Jon Paul's friend, Scott. He has spent considerable hours following each database error researching and restoring the database from backups.
We've made the decision to take advantage of Scott's server, database and vBulletin expertise by upgrading and migrating the entire site over to a server hosted by Scott's company's datacenter.
As mentioned, moving to a new server and a new host for the site will require considerable investment in time and finances, but will afford us the expertise of Scott's team, which should result in vastly improved site performance and uptime, as well as provide an upgrade to the hardware that will provide room for continued growth.
The Timeframe
We've been working with Scott and his team to determine hardware specifications (for now and to allow for growth), and Scott ordered the hardware earlier this week. Once it arrives (in the weeks following Bownanza), Scott's team will assemble the server, and then begin the migration process. Most of the work will be performed in advance of the migration, and the migration itself should require very little, if any, downtime. However, there may be occasions where we need to take the site or the forum offline for temporary periods.
What it Means to You
As mentioned, it will still take a few weeks to receive the equipment and get it configured before beginning the migration. Unfortunately, that means that we are still susceptible to the same potential problems, including database corruption (and loss of data associated with restoring from backups). As such, I would suggest if you compose a lengthy post or thread, that you save that thread elsewhere in the event that it is lost. Scott has graciously volunteered to "babysit" the site while it remains on the existing server, even though he has no affiliation with the host, or the site for that matter. As you might imagine, I'm both apprehensive and excited about the forthcoming changes!
In addition, we have also been evaluating upgrades to the discussion software. Soon after the migration of the server, we will begin implementing those changes as well. Having tested the new features on another forum, I'm extremely excited about the changes associated with that upgrade, as well!
Thank You
Finally, I'd like to reiterate how grateful I am for all of our members that make this site what it is. Obviously, many have been frustrated with the downtime and lost data. Most of you have remained patient as we have attempted to resolve the issues. Others, perhaps, not so much, but it might not be a bad idea to allow those members that seemingly no longer find enjoyment in our little community to cull themselves from it.
We all look forward to even bigger and better things from our site and its members! Thank you for your continued patience and feedback.
Michael
. I know yall work hard and am just happy I've got another family I can come to here. I can deal with a little "break time" so to speak
Comment