OMG. What a nightmare the past three weeks have been.
Started out with Facebook changing security priviledges for "apps", which affected our server's ability to send weather bulletins to Facebook. Until we were able to figure out a workaround for that, everything on Facebook was down. It also affected our postings to the pages of several northern Florida Emergency Management agencies, and made us look very bad...even though it wasn't our fault. And I don't like looking bad.
After spending a god-awful lot of time daytime and into many late evenings and sometimes early mornings, I and a bunch of other beta testers were able to pool our own hands-on, experimental knowledge until we were able to figure out how to work with the Facebook changes.
Right on the heels of getting things going again on Facebook, then the Fireline server crashed. Not sure on the deeper, more technical specifics of how it happened. But...
...Something to do with the server software being old...originally 2003, upgraded to 2007. Then came an automatically downloaded "upgrade" and all hell broke loose. Nothing worked anymore. All dirs on the server had their permissions reset. Dirs became "unmapped". We could work with a top dir, and a third level dir, but the middle dir...it was as if the server "forgot" it was there. ...Things like this. We had to literally start from scratch with Fireline. All web pages ceased to function. People couldn't see them because permissions had been reset and no one was "allowed" access and they got errors when they tried. The maps on the ACS page stopped updating. The marquee scroller and the bulletin scroller both stopped working. Access ceased to the online EMWIN bulletins archive. Bitly links in Facebook postings no longer worked because access to the online EMWIN bulletins archive couldn't be had anymore. You can see all the cascading problems that resulted.
Both I and the Fireline sysop tried really hard, staying up late at night, to try to stay ahead of all of the problems and to try to figure out how to get the Fireline and the AC-EMWIN servers to work together again. We felt like Data trying to stay ahead of all the "cascading failures" with Lal. (...If you remember the ST:TNG episode where Data had built a daughter.) In Data's case, he was unsuccessful and Lal ceased functioning. In our case, I think we actually licked it, and things are again functional. But I'm going to say that with wariness because on Facebook I've said that before and then something drastic happened and we had to start all over again - and I don't want to jinx myself. :)
The Fireline server software is old and no longer supported by Microsoft. The sysop wants to upgrade it but the cost is just ridiculously prohibitive. We're talking on the order of $4500. Double that for the backup server in case the first one goes down.
At this point, I believe we've finally been able to get everything back up and operating EXCEPT for a couple of Emergency Management agencies. But Facebook pages for Alachua Co SKYWARN, the Alachua County EMWIN Project, GVLStorms, GVLWeather, and GNVWeather, should again be operating as normal from this point onwards.
Let's hope something like that that doesn't happen again for a while. I think I've had about enough this past few weeks and I need a serious vacation. (sigh) It's been nerve-racking.
All that being said, we believe in a couple of mottos wholeheartedly, though...
First of course is "NEVER give up! NEVER surrender!" -Peter Quincy Taggert, Galaxy Quest
Secondly, you've got "Failure is not an option!" -Eugene Krantz (during the Apollo 13 disaster)
And well, of course, you have Coln. Jack O'Neill too, who once said, "So, when your back's up against the wall, and there's no tomorrow, just take one day at a time, and remember...the bigger they are, etcetera."
But the best, most important words of advice came from my cat, Stormy, who with a stare quite serious, sternly advised..."Meow." ...And I believe he was right on.
UPDATE - MAY 24, 2015: Everything again functional. texts to email, pagers, cellphones, listgroups, Facebook, Twitter, web pages...everything...including all our Emergency Management Agency customers. Even gained an additional EM customer: Liberty County. So at the moment, we're serving the following EMA Facebook pages: Bradford Co, Gulf County, Holmes Co, Liberty Co, and Washington Co...with the possibility of Bay Co soon joining the bandwagon, too. :)
