September 19, 2020

Joe the Plumber Gives a Brief Post-Mortem

Poster: Computer Errorsphoto © 2007 Rasmus Olsen | more info (via: Wylio)Presented by Chaplain Mike

First of all, let me say how thankful I am for all of you in our Internet Monk community. You have patiently hung with us as we’ve gone through an absurd number of site problems over the past couple of weeks. I have received emails and messages on Facebook expressing concern, promising prayer, and even offering financial support to help us overcome our difficulties. Thank you all for your loyal love. Please know that we continue to work hard to resolve all issues and put the site on a platform that will enable us to be more secure and stable.

Second, I can’t say enough about the work of our tech guru, Joe Stallard. Joe has generously and kindly given his time and efforts to work with our hosting company to deal with volume issues and hacker attacks. It has at times been frustrating trying to fix complex problems while patiently persevering with online and telephone tech support. Having recently spent over two hours trying to set up a piece of equipment with such tech guidance, I know how irritating and discouraging that can be. I have promised Joe many indulgences for his good works on our behalf.

Now I’ll let him tell you more about what’s been going on.

Joe the Plumber Gives a Brief Post-Mortem
In an effort not to bore you with too much geek gibberish, I’ll try to keep this short.

Following the previous hack of the iMonk website, I had gone through the files located on the server and deleted out several old programs (old forums, old WordPress, etc.) that had just been “stored” on the server.

When our web host restored the site, the backup that they used included those deleted files – which made us vulnerable once again.

Sad Mac! :-(photo © 2008 Brad | more info (via: Wylio)Whenever we ask the web host to perform a backup/restore, they always revert to the most recent “good” backup for doing this. In this case – the backup used was a good one but it included the vulnerable files.

The attack that took us down used an automated script. The script is designed to automatically search the internet and looks specifically for old programs that are known to be vulnerable (easy to hack) and easily infected.

Once the script finds a vulnerable program, it will then infect the whole site. In this instance, no damage was done aside from shutting the site down. This shut-down was due to the interaction between WordPress and the platform that we’ve used for iMonk.

When we brought the site back online, I didn’t realize that the “back door” for the attack was once again present.

As a result, it was just a matter of time before we were hit again – and we didn’t have to wait long.

On this occasion, I initially (and mistakenly) thought that we could bring things back rather quickly. After all, we had just been through it and knew the steps to take.

That’s when the real train wreck took place.

A quick call to the web host put the plan into action. In this case, the web host’s security team became deeply involved. After all, the site was hacked, and who better to bring in than the security experts, right?

Wrong.

It seems that Moe, Larry and Curly were on duty.

First of all, they couldn’t get the backup files to compile. The site files were too large and the server overloaded.

So they tried an “alternative method” of performing the backup. One that would run slowly in the background to avoid the overload.

Instead of taking a couple of hours to do the backup/restore – it took some 16 hours to do the backup alone.

Finally finished, we were told that the restore was underway.

A couple of hours later, the site was back. Except for one thing – there was no site and the complete site structure was publicly exposed.

This was a loud and public invitation to all the hackers – “Here I am! Come on in!”

Another phone call. “Yes, this is not a good thing,” they agreed, and the site was taken back down.

At this point, our conversation became really interesting. They insisted that there were no WordPress installations on the site’s backup and that was why there was no WordPress re-installed.

Pointing out the fact that the site had run exclusively on WordPress for some years now resulted in silence from their end.

By now, we were well into the weekend – entering our fourth day without a backup to work from.

Enter the weekend crew.

Our web host has a support “chat” interface that always displays a photo of the support technician on the other end.

The technician that I was in contact with looked to be all of 12 years old and my initial thought was that we now had a high-school kid who worked weekends for some gas money. It really didn’t look good.

Fortunately, my first impression was dead wrong. Within a couple of minutes of back and forth, I realized that the kid on the other end was an uber-geek. Not only did he understand server configuration, he knew WordPress inside out.

We came up with a plan to try and get the site restored. The only thing that appeared to be intact from the piece-meal (and failed) backup on their end was the database. And the database contained ALL the iMonk content for the last several years.

I say that the database “appeared” to be intact because, at this point, my trust in their backup process was just a bit lacking.

So, I began to work through the plan that he had suggested. And, yes, the thought had occurred more than once that this was a plan that shifted the burden of restoring the site away from the web host and put it squarely on us.

Pretty shrewd on their part, I thought. After all, each and every step of what they had been doing had taken us that much further from ever getting the site restored and back online. I’ll be the first to admit – the thought that their failed attempts had done permanent damage was starting to creep in. If this didn’t work, the only remaining option was to rely upon one remaining backup that (hopefully) was in place and not corrupted.

Yes, it’s a “when all else fails” scenario.

The good news is – it worked, but not without a few bumps along the way. First of all, the fresh WordPress install would not connect or communicate with the database.

Computer Kidsphoto © 2008 Eden, Janine and Jim | more info (via: Wylio)Another call to the host (we were now well into Saturday night) only to discover that our uber-geek had left for the day. But – I was assured that “one of our “senior-level” technicians had just come on duty and would look into it right away.

The not-so-good news was that the “senior-level” technician was not familiar with the WordPress database structure. The good news was that he was, in fact, a knowledgable database administrator and, after a couple of hiccups, managed to get the database restored and communicating with WordPress.

Except for one minor glitch – we couldn’t log into the site. So, another round of “chats” and support ticket communication resulted in some further tweaking of the database settings and we were finally able to get back into our site.

At this point, we were approaching midnight on Saturday, so I just left things alone for a while.

With Sunday dawning, a quick check of the site showed us still online and running with just a couple of glitches.

A little housekeeping brought things back pretty close to normal. At least the train was back on the track – just not quite full throttle.

There are still some widgets to re-build and a tweak here and there, but things seem pretty good for now.

Lest you breathe too deep a sigh of relief, or become a bit complacent, I want to leave you with these words of comfort:

Sometime this week, we move the site to our dedicated server.

Comments

  1. Good Job Joe…

    I know WordPress pretty well and it can have a mind of it’s own, you have to actually work through it to actually understand some of it’s glitches. I run two WP sites, one I custom fit into an existing site (I even replaced the terrible WP login screen) and the second site I gave up and bought a skin 🙂

    I’m so glad the site is back!!!

    Blessings…

    -Paul-

    • Thanks, Paul. And yes, it seems like it only misbehaves when you really need for it to play nice. It sometimes takes a while, but once you do understand some of its glitches, you can make it do some amazing things.

      On this occasion, our normally reliable web host almost turned an “incident” into a tragedy.

  2. One more Mike says

    Well done Joe. Though the only word in your explanation I understood was “corrupted” (I do live near DC so understand corruption all too well) I admire you guys who inhabit Geekdom, and appreciate everything you do to keep the IMonk community buzzin’.

  3. Tenacity… Great job! Any more these days i am usually solving my own issues having run into so many ‘guru’s” that make things worse. It is good news that you were able to find competent help to work through the issues. I run into this even wthin my own company, the “It didn’t work – oh well it’s your problem” syndrome. Internet Monk is lucky to have someone like you who has the stick-to-it-ness…..

    • Thanks, Radagast. It seems that “stubborn” streak finally did something right.

      Let’s, um, not discuss the other side of that coin, OK?

  4. I paint houses for a living and I honestly don’t know what any of that meant. I can say that it’s apparent you worked your tail off so now we get to type stuff. Thank you!

    • Hey, Chris – my part here is to make sure the plumbing works – and it’s as if somebody slipped in and flushed a hand grenade.

      Right now, it’s still pretty wet here in the basement but it looks like the leaks have stopped and the pipes are holding.

  5. This reads like an episode of Star Trek, “The Galileo Seven”. Spock resorts to an act of desperation as the only logical course of action.

    SPOCK: We don’t have to maintain [orbit] very long, Mister Scott. In less than twenty four hours, the Enterprise will be forced to abandon its search in order to make a rendezvous. If we can’t maintain orbit after that time, it won’t make any difference. If we burn up in a decaying orbit or die here on the planet’s surface, we shall surely die. Doctor, your phaser. Go to work, Mister Scott.
    SCOTT: Aye, aye, sir.

    [Shortly after that, Mr. Spock fired off the shuttlecraft’s remaining fuel as a distress flare. Looked great, even in black and white. And the Enterprise came to the rescue! Beaming them aboard just in the nick of time, as I remember, before they fried to death in re-entry…]

  6. I’m glad this site is up and running. I love it here…I feel like this is a part of home for me. Of course I can’t chill with people in person, and while I do have a couple of people here that I can talk with over a brewsky or coffee…I like the thought that others here have struggled with evangelicalism or are licking their wounds. That brings me comfort in knowing that I was not the only one singed….

  7. Now, back to figuring out how to un-hack evangelicalism. That task seems far more daunting.

  8. Three cheers for you Joe! Whew. Job well done.

  9. God bless and keep you, Joe.

    Reminds me of an old quote from somewhere or other: to err is human. To really foul things up, you need a computer.

    🙂

    • Amen Martha!! Thanks for all the hard work you all did. I would miss you terribly as you have been my guiding light through my “scarey freeing journey” .

  10. Joe,

    Having had to reconstruct a web server and an email server after it shred itself due to an overload of spam, I understand what you’ve been through and salute you. Thanks

  11. Thanks so much again, Joe! What a process! I have a love/hate thing with computers. When they work well, I love them. When I get some trojan that has done damage that even my paid internet PC guys can’t figure out, I am not happy. The last time that happened, the guy told me that I was going to have to reinstall Windows which he said would remove everything else on the PC. I have an external drive where I save documents but we have software we bought online and so much stuff, I didn’t want to do that. So, I tried one last thing myself and fixed it! I was so happy.

    • Dana Ames says

      Joanie,

      get yourself some ESET. They have never had a virus get through; cost is reasonable; it’s a sleek program that doesn’t take a lot of room in your computer; if there is any question, you can talk to an actual human being. In your browser window, add the dot com to those four letters and you will be at their site.

      Dana

      • Thanks, Dana. I just contracted with my phone company who is also my DSL internet provider to provide protection with something simply called Internet Security (Powered by F-Secure). It used to be called Defender, I think. I have been using McAfee, but too many things are getting past McAfee, I think. I also have other Spyware removal programs I use as well as Registry Patrol software which is what saved me the last time.

  12. My experience is to call about midnight with a problem. They usually have the guys who love to get their teeth into a big problem but don’t always play by the rules. They get the problem solved and don’t care how long the clock runs. It has worked 3 times when no one else could fix it.

    • You’re spot on with that, Liz. Our previous problem was solved pretty fast with the weekend crew at our web host. And it was the weekend crew again that was able to help us get things sorted after the “day shift” almost destroyed the site.

  13. And speaking of computer problems: I have been trying to read Rachel Held Evans blog about Mark Driscoll and the page will start to show but then never fully load and it stops responding. Maybe because she has over 500 comments according to the number I can see near the top of the page!

    • JoanieD

      What OS are you using? And how fast is your computer and home much memory/ram does it have?

      • David, I just looked at the System Information on my PC and I have Microsoft Windows XP Professional, Media Center Edition, Version 2002 with Pentium 4 CPU 3.20Ghz 3.19 GHz, 1.00GB of RAM. I don’t know what most of that means. We have had the computer maybe 5 years and have never reloaded Windows which the tech guy said I should have done at least twice by now.

        • Also, I decided that instead of clicking on the link Jeff or Mike gave us here to go to Rachel’s blog post, I just typed in the main blog address and then worked my way to her post. I see that she took down all the comments about her Driscoll post even though the page still says there are 513 comments. She says she is still getting some hate email though. That’s sad. 🙁 You can still see what her post says about why she is bothered by Mark. I share her concerns.

        • If you’re not there yet you really should install the SP3 update. Which might require you to install the SP2. And that might require the SP1. I can’t remember if they are sequential or cumulative.

          That should really help out your AV issues. And switch to Chrome for surfing the internet. MS is NOT releasing any updates to IE running on XP. I would reccomend FireFox but they seem to be driving off the cliff lately with their version change policies.

          • Oh, David, I do have Service Pack 3. My husband uses FireFox. I can check out Chrome. Thanks for your concern!

  14. Amory Ewerdt says

    Just and FYI in case others are still in my predicament. I have been unable to get to your site (at least on my personal laptop) until just now. Going to “internetmonk.com” brought me to a Host Dime site. On that site there are several options, one of which is live chat. I spoke with someone and they were able to get me here by entering “https://internetmonk.com.proxy.dizinc.com/” into my address bar. He said it is possible that things were being changed around when I attempted to get to IM in the usual way and so maybe it was stored in my computer that way or something. Anyhow, I just thought you might want to be aware of it in case there are others who are still unable to access this site.

    • Amory,

      I’m no geek, but have you tried clearing your browser’s cache and/or history files. That seemed to help during the time when Firefox and IE were giving me different results.

    • Amory,

      Clearing the browser cache should make it work. The address that they gave you is actually our new server that the site is being migrated to.

      Right now, the migration is still ongoing and you should be able to reach the site as normal. Once the change-over is complete, you’ll still just type in “www.internetmonk.com” as you usually do.

      There shouldn’t be any disruption or changes on your end – this change should just keep the site from crashing the web server due to the traffic load.

  15. Joe:

    Before I forget, thank you for your hard work and a job well done.

    Along with performing current backups of all your web page directories to and including WordPress files and images and saving them to your local system, I would recommend you obtaining the WordPress plugin “WordPress Database Backup” by Austin Matzko currently at Version 2.2.3 and installing/activating the plugin.

    You can set the plugin to back up the database as short of a time duration as “hourly” if you like. But you can select every 12 hours, daily (every 24 hours) , or weekly (every 168 hours).

    Also, you can set it to backup to a specified dedicated directory on the server for you to bring down to your local system via FTP (and backed up when your webhost does the backups); or even better, have it sent via e-mail to you via a file attachment that you can download to your local system. You can also do a “on-the-spot” backup to download / save to your local system.

    When I moved my WordPress install over to a new server for the WP 3.2 (now at 3.2.1) release needing the PHP 5, all I had to do was to carry the old WordPress files and image files over in the same recursive directory scheme, and then made sure the database name, “localhost” username and password was the exact same as the old one and matched the WordPress “wp-config.php” file, and then use phpMyAdmin (a GUI interface to maintain MySQL databases my web host provided) to upload the backed up database from my local system to the server.

    Because internetmonk.com is a high traffic site with lots of data, I would recommend backing up the files plus database from your local system at least nightly and after that, copy it over to either a writable CD/DVD or get an USB external hard drive like the Seagate that also has a backup software that you could set up to back up your local system to the external drive.

  16. I hope the (whole) site made it safe to your server. Blessings for your hard work!
    ~ Teodor C.