Announcement
Collapse
No announcement yet.

Harmony central outage

Collapse
X
  • Time
  • Show
Clear All
new posts

  • Harmony central outage

    http://www.harmonycentral.com/forum/...central-outage
    Keeping the Harmony at Harmony Central

  • #2
    The below is taken from the link


    "Our apologies, we've been down for two days due to a catastrophic hardware failure at the server companythat hosts Harmony Central. The good news—we’re back. The bad news is we lost some data. Luckily our software technology partner, Nucleus Creative, was able to restore the data up to January 10th, 2016. We thank you for your patience and continued support. We've been assured that the root cause has been isolated, and that this specific hardware failure will not happen again

    Dendy Jarrett | Director Harmony Central Communities"
    Last edited by billybilly; 02-07-2016, 11:33 AM.
    MIA Fender Strat / Gibson Les Paul Studio / Custom Telecaster / Washburn WI66 / Custom Stratocaster / Martin D15S / Guild D55 / Simon & Patrick Cedar / Martin HD16R LSH​

    Comment


    • #3
      I'm really struggling to find anything constructive to say so I won't.

      Why don't any other forums experience such problems?
      MIA Fender Strat / Gibson Les Paul Studio / Custom Telecaster / Washburn WI66 / Custom Stratocaster / Martin D15S / Guild D55 / Simon & Patrick Cedar / Martin HD16R LSH​

      Comment


      • Dendy Jarrett
        Dendy Jarrett commented
        Editing a comment
        Harmony Central maintains our 15 years of database on the servers. There is no other music community that has this much data maintained. We are the oldest, so it would be natural to deduce this is why we have so much data. Removing any of that data interferes with SEO and Google search loves HC.

        The good news is we have 99.998% of HC back. The bad news, we lost a month of good content, threads and posts. But, hey, ... we're back.

        This was one of those "being struck by lightening twice" situations where we had a cataclysmic hardware failure that took out our back up drives (2 of them).
        Stuff happens out of our control. Earthquakes, Tornadoes, Cat 5 Hurricanes. What matters is what we do with it, learn from it and move forward from it.

        No one will ever know how much work went into the last 48 hours to get us back to where we are.

    • #4
      Originally posted by Dendy
      No one will ever know how much work went into the last 48 hours to get us back to where we are.
      As someone who has worked in a data center, I have an idea. If the hardware failure was a hard drive (and this suggests so) then there's really nothing you can do but restore from the last backup. There's also a very good chance that the same machine hosting HC forums also hosted about a hundred other sites, and it wasn't that people were sitting on their thumbs for those two days- They were working 'round the clock to get everyone up and running.

      TL;DR: **************** happens, glad we're back!
      __________________________________________________ __________________________________________________

      It's funny how nowadays the only decent metal around is the theme music for a cartoon that makes fun of metal bands.

      Comment


      • #5
        test
        "These riffs were built to last a life time". Keith Richards

        "All the women in the world want a phony rock star who plays guitar". John Cougar Mellencamp

        "
        If you're listening to a rock star in order to get your information on who to vote for, you're a bigger moron than they are." Alice Cooper

        Comment


        • #6

          Sounds like 2 drives failed in a RAID-5 array?



          Comment


          • #7
            Wow. That's a lot of posts gone.

            Comment


            • AlamoJoe
              AlamoJoe commented
              Editing a comment
              I know. Wonder where they are? You think someone is gonna wake up and find their lawn all covered in posts?

          • #8
            Just testing...
            - formerly vbadmin

            "If it was easy, everyone would do it."

            Comment


            • #9
              Not the end of the world folks. Glad it's back. But if it goes down like that again, I'm going to have to ask for my membership dues back.
              Signature...

              Comment


              • Ratae Corieltauvorum
                Editing a comment
                You're actually 7 years behind in your subs, didn't wanna say nothin, but you brought it up

            • #10
              Sorry but "this specific hardware failure will not happen again" isn't that reassuring because it implies that someone somewhere is busy inventing new hardware failures. As someone who used to build and repair PCs as a small business, I know just how many ways computers can fail. A big round of applause to the folks behind the scenes but pardon me if I'm still a wee bit nervous.
              Official HCAG “Theory-Challenged Hack”
              Member of the IBANEZ ACOUSTIC ASSASSINS
              Proud Member of The Alvarez Alliance
              Member of the Schecter Society
              Person-2-Person on the Web

              Comment


              • 1001gear
                1001gear commented
                Editing a comment
                I find the mystery intriguing but am not too hip on the technicalities.

              • Dendy Jarrett
                Dendy Jarrett commented
                Editing a comment
                It is as mysterious to us as to you, but we are being as transparent as we can be based on the pre forensic info we have. We have always tried to be forthright with you guys. The site has more support and better support right now than we ever have. We don't take an outage or loss of data lightly. And we will take all steps to prevent it in the future. We are, however, just people like you.

              • 1001gear
                1001gear commented
                Editing a comment
                I don't doubt it. Just curious. Thx.

            • #11

              1 production server (i.e. H-C) with RAID-5 or RAID-6 protection and at least one hot spare drive.

              1 smaller dedicated backup server in same equipment rack as production server, on same network as production server, with dedicated 1 Gb/s network connection between servers.

              Backup server has two very large mirrored (RAID-1) hard drives with a hot spare drive.

              Dedicated backup server performs daily server backups of all file systems -- incremental, weekly differential, and monthly full backups as production server stays online.

              Daily (at least) database snapshots are performed.

              Backups and snapshots stay on dedicated backup server as a mirror image of each backup is trickled off asynchronously to "cheap" cloud storage (that way, you have at least two full copies of backups).

              Backups are kept for at least 90 days if not a year.



              Comment


              • #12
                Dendy...Please don't ever say the phrase "Pre Forensic" again. It creeps me out.

                I'm just glad the server is back up. And frankly...there were too many repeats happening in "Music Association Challenge". I'm personally grateful for the freshen up.
                http://thebasement.createaforum.com/

                Comment


                • #13
                  If you have multiple , simultaneous, drive failures it is almost certainly power frying the controller boards. In this case the data is still on your discs and can be restored for a few hundred bucks.
                  if some numpty formatted them you will get 99% back with a surface scan on the discs.
                  if somebody is telling you otherwise i would question those responsible, something does not sound right.
                  Last edited by knotty; 02-08-2016, 04:02 AM.
                  “One day your life will flash before your eyes. Make sure it's worth watching”
                  Gerard Way

                  Comment


                  • #14
                    As a veteran of watching the Bourne series on TNT I always wonder about schitt. Hope nobody got hurt.
                    Originally posted by Unconfigured Static HTML Widget...








                    Write Something, or Drag and Drop Images Here...

                    Comment


                    • #15
                      Looks like a lot/most of the loss was recent postings?
                      Music is a moral law. It gives soul to the universe, wings to the mind, flight to the imagination, and charm and gaiety to life and to everything. - Plato

                      Comment


                      • Ratae Corieltauvorum
                        Editing a comment
                        Aaaaaah ****************, there's been a lot of quality traffic since the new year

                      • Chordite
                        Chordite commented
                        Editing a comment
                        Dang, that's my Pulitzer Prize out the window

                      • Ratae Corieltauvorum
                        Editing a comment
                        Lotta good posts ain't goin home ta there families boy😓
                    Working...
                    X