1. #1
    Forum Member

    Join Date
    Oct 2005
    Posts
    1,154

    Default We're the phone company. We don't care. We don't have to.

    Backup a computer?

    Why on earth should we keep a backup of data stored on a computerized system? That would cost money!

    Monkeys. Absolute Monkeys.

    (BTW...You don't "send this equipment to the manufacturer" for analysis...the disk systems that would store this volume of data the manufacturer comes to you. Heck, the units have "phone home" capability built in that calls the manufacturer to send a tech whenever they detect a problem. Fracking PR spin doctor idiots.)

    Verizon vendor accidentally deletes 911 recordings
    By Ken Maguire, Associated Press Writer | June 20, 2006

    BOSTON --A Verizon Communications Inc. subcontractor deleted nearly a year's worth of recordings of 911 calls to the Massachusetts State Police, a mistake which could weaken criminal cases if the data cannot be retrieved, authorities revealed Tuesday.

    The lost data includes 1 million emergency calls from cellular phones, and a half-million radio communications between troopers and headquarters. All 911 calls from cell phones in Massachusetts go to State Police.

    A technician hired by a Verizon contractor, caused the problem last Tuesday while upgrading equipment at the Framingham headquarters, according to preliminary investigations by state officials and Verizon.

    District attorneys and police departments around the state have been notified that 11 months of recordings were deleted, said Kelly Nantel, a spokeswoman for the Executive Office of Public Safety.

    "Rarely are 911 tapes crucial in an investigation, but they certainly can be part of an investigation," she said.

    Nantel said no affected cases had come to their attention yet, however, "it's certainly possible, and we won't know that until we move forward."

    Verizon has assigned in-house experts and may hire outside specialists to try to recover the files. If necessary, the equipment will be sent to the manufacturer for forensic analysis, a Verizon spokesman said.

    "We're going to lead the charge to see what can be done to retrieve the data," Verizon spokesman John Bonomo said.

    Verizon has an $80 million annual contract to provide service to State Police, Nantel said. The company holds similar contracts with other states, including New York.

    Bonomo said a technician from Needham-based Dictronics Inc. was upgrading the system. A call to Dictronics was not immediately returned.

    Despite the apparent mistake, Bonomo said Verizon is satisfied that its subcontractors are competent.

    "We feel confident that a lot of the companies we do business with are experts in their field," he said. "There's a certain set of procedures that need to be followed. It appears that there was some kind of error when the technician did the work."

  2. #2
    Forum Member
    backsteprescue123's Avatar
    Join Date
    Apr 2004
    Location
    Ohio
    Posts
    4,319

    Default

    OH MY FREAKING GOD!

    What a bunch of bums looking to save a buck.
    ------------------------------------
    These opinions are mine and do not reflect the opinions of any organizations I am affiliated with.
    ------------------------------------

  3. #3
    MembersZone Subscriber
    MalahatTwo7's Avatar
    Join Date
    Mar 2002
    Location
    Loco madidus effercio in rutilus effercio.
    Posts
    12,837

    Default

    Ya know. Y'd figure that of all the companies on the planet that rely on computers, the telephone co would be the most up to date and online with current technology and backup systems.
    If you don't do it RIGHT today, when will you have time to do it over? (Hall of Fame basketball player/coach John Wooden)

    "I may be slow, but my work is poor." Chief Dave Balding, MVFD

    "Its not Rocket Science. Just use a LITTLE imagination." (Me)

    Get it up. Get it on. Get it done!

    impossible solved cotidie. miracles postulo viginti - quattuor hora animadverto

    IACOJ member: Cheers, Play safe y'all.

  4. #4
    dazed and confused
    Resq14's Avatar
    Join Date
    Dec 2000
    Location
    New England
    Posts
    1,993

    Default

    I don't believe Verizon is responsible for the actual recording system. They aren't in my state, at least, and I believe we have similar contracts for E911.

    We also use Dictronics digital "dictaphone" system, and its on our shoulders if something goes awry. It is hard drive based, but is *backed up* to removeable digital media routinely and automatically.

    I can't believe that they don't at least do the cheap removeable media backup option, let alone redundant hard drives. Seems a little... dangerous.

    And as you said, Dal, they send the tech to you! haha
    God Bless America!Remember all have given some, but some have given all.
    Google Is Your Friend™Helpful forum tip - a "must see" if you're new here
    Click this to search FH Forums!

  5. #5
    Forum Member
    nmfire's Avatar
    Join Date
    Nov 2002
    Location
    Maryland (DC Suburb)
    Posts
    5,738

    Default

    Since when was the phone company responsible for recording equipment? That stuff is usually seperate equipment maintained by the agency... and usually it isn't even a computer. It is a totally seperate piece of hardware and they can't be "deleted".
    Even the burger-flippers at McDonald's probably have some McWackers.

  6. #6
    MembersZone Subscriber

    Join Date
    May 2000
    Location
    Wheaton IL
    Posts
    1,767

    Default

    It is just a case of it won't happen to me, why worry.
    I bet they back stuff up now, good lesson for the rest of us, thanks Mass State Police.

  7. #7
    MembersZone Subscriber
    N2DFire's Avatar
    Join Date
    Feb 2000
    Location
    S.W. Virginia
    Posts
    1,286

    Default

    ************ Computer Geek Alert ***************
    O.K. before we go off on everyone involved let's examine a few things.

    1) The owner of the Data is ultimatly responsible for backing it up or seeing that it get's backed up. Conclusion MSP is Primarily at fault for not having a back-up system.

    2) No one said the Tech. was incompetent - a little over relaxed maybe. Computer hardware fails every day (that's why back-up solutions exist in the first place). The Tech. should have performed a back-up prior to begining any work on the system.

    "There's a certain set of procedures that need to be followed. It appears that there was some kind of error when the technician did the work."
    BINGO - Fault # 2

    3) Sorry Dal but with your background in journalisim - I'm disapointed.

    Quote Originally Posted by Dalmatian190
    (BTW...You don't "send this equipment to the manufacturer" for analysis...the disk systems that would store this volume of data the manufacturer comes to you.
    You've taken the information out of context
    Verizon has assigned in-house experts and may hire outside specialists to try to recover the files. If necessary, the equipment will be sent to the manufacturer for forensic analysis, a Verizon spokesman said.
    First -
    Let's make an analogy we can all relate to - if you make an EMS run do you carry EVERYTHING in the Bus to the Pt? No You take the basic common things and if the Pt's needs are more complex than what you can handle - you move them to the bus to work on.

    Same principle - they will be on site and try a few common "tricks" to recover the data, in the event that they fail - then the entire unit may indeed be carted off to a lab somewhere for more invasive data recovery methods.

    Second -
    Honestly at this point all we "know" is that the data was lost "while upgrading equipment" - We have no clue exactly what he was attempting to do or what cause the data loss. My guess is it's not something as simple as "Move to recycle bin" or "Format C:" That stuff happns too often and that type of data recovery is pretty easy. Heck I just pulled some "lost" files off my g/f's laptop AFTER I did a fresh install of Windows on it about 2 weeks ago. Took less time to get the data back that it does to install windows!!

    Depending on exactly what did happen, they may very well have to ship off a drive (or drives) to have a "post mortem" data recovery performed (i.e. remove the platters and place then in a new enclosure - basicly build a new drive with the old disks in it. Not exactly an on site operation since it requires a steril dust free environment).

    4) Just for the Geekly among us - let's see just how much data were talking. Let's assume that MP3 quality recordings (128kHz) were used (Not likely).
    On average that's 1 Megabyte (MB) per minute of audio.
    Now assuming a constant sream of audio 24/7 for 11 months (Also Not Likely)
    60 minutes per hour X 24 hours per day X 30 days per month X 11 Months -
    that's "only" 27843.75000 gigabytes (GB)** or 27.191162 terabytes (TB)**

    More realisticly it's probably 15TB or less

    ** Although data storage capacity is generally expressed in binary code, many hard drive manufacturers (and some newer BIOSs) use a decimal system to express capacity. For example, a 30 gigabyte drive is usually 30,000,000,000 bytes (decimal) not the 32,212,254,720 binary bytes you would expect.
    Source: http://www.t1shopper.com
    Last edited by N2DFire; 06-21-2006 at 10:38 AM.
    Take Care - Stay Safe - God Bless
    Stephen
    FF/Paramedic
    Instructor

  8. #8
    Forum Member
    nmfire's Avatar
    Join Date
    Nov 2002
    Location
    Maryland (DC Suburb)
    Posts
    5,738

    Default

    I'm still trying to figure out what they use to record audio that uses a PC in the first place.
    Even the burger-flippers at McDonald's probably have some McWackers.

  9. #9
    Forum Member

    Join Date
    Oct 2005
    Posts
    1,154

    Default

    Well, first my background is NOT journalism.

    My last full-time job prior to going into freelance consulting was at a newspaper, and I learned a lot about journalism there.

    My background is systems administration. Specifically, mid to high end systems administration.

    The "15 TB" or the full "27 TB" you calculate is trivial to achieve with today's storage systems.

    At the T&G I had direct responsibilities over an EMC Clariion CX700 with 2.6 TB installed -- and a capacity for 27 TB. That unit was the best technology of the three "storage area networks" we ran.

    And that's at best a "mid-range" machine by EMC. And that was at a newspaper that barely broke the top 100 in circulation -- storage systems of this size certainly should not be considered ground breaking or unusual for agencies handling the volume of a Mass State Police or any other mid-size enterprise.

    Freelancing, I've done disaster recovery planning & testing on disk systems far larger and more complex than that, including wide-area replication.

    Why would you keep "tape" as digital data on a hard disk system (and it's clear if they've lost 11 months that quickly, that's how it had to be handled)?

    Ease of management.

    No physical tapes to change; no physical tapes to lose.

    No SINGLE POINT OF FAILURE -- unless you're recording to two tapes, the failure of one tape / recording head /etc and you're not recording. High performance disk units spread the data across multiple, redundant disk systems...preferably even physically seperate chassis; and ideally with them in geographically disperse areas (geo-clustering in live time is real challenge because of latency of the speed of light...but near live-time replication is possible).

    Speed in pulling up specific information in the future -- you're not looking for and mounting a tape, then zipping through the tape to find what you're looking for. Instead you tell the computer you need radio traffic on channel Z starting at such a time on such a date...wham, there it is.

    Ease of backing up -- even with replication, backups to physical media (like computer tapes, or dedicated lower-cost disk systems) is still a good idea to guard against a malicious or other problem that would replicate to the other system. This is much easier with data, then copying traditional audio style tapes.

    How was the data lost?

    Several scenarios spring to mind.

    One is overwriting disk configuration information. Which sucks when it happens -- had a hardware swap test go horribly wrong on me once, and had the RAID Configuration on the disks (where DEC / Compaq / HP stored it in that series of RAID systems) overwritten even though it *shouldn't* have happened. Had the backups, started streaming back the backup tapes, and smile sheepishly at management, "Stuff happens..." But I had known good backup prior to doing the change.

    Another is a deletion of the data, which should be recoverable by software as long as the data isn't overwritten. Don't need to ship the disks to a clean room for that.

    Loss of encryption keys also come to mind, although I suspect knowing encryption isn't used frequently enough...this probably wasn't it. But having encrypted data and forgetting the key is a bad thing depending on what you have...and shame on you for not having the key securely stored somewhere.

    The physical failure of a drive -- where you would have to open it and start reading direct from the platter in a clean room -- is inconceivable. Systems for this volume of data simply aren't vulnerable to the failure of a single disk or usually disks.

    Again, going back to systems I've worked on, one of my systems again with over a TB of storage, but lower-end then the EMC, which only handled transient data (mostly thrown away after each night), was stored on a mirrored mirrored RAID system -- to physically seperate chassis that mirrored each other, and in each unit the disk sets where mirrored and striped. We could lose one 1 out of 2 chassis, or 3 out of 4 disk sets, and if you included hot spares, 15 out of 18 disks...and still get out the paper without calling IT at night to fix anything.

    While I do not know exactly what went wrong...I know enough of the minimum size the systems need to be to simply function, and I certainly know *how* they should've been set up, and basic procedures that *should* have been in place.

    There are situations where you get caught having to take a gamble. But in those, you keep executive managment / customer in the decision making loop to go ahead understanding an unrecoverable loss may occur. Stuff like that gets an actual old fashion, physically printed memo signed by the customer so it's crystal fracking clear to them if data is lost by me, it's not because of an error or ommission but a known risk I've been authorized to take -- I want it, and so does my Errors & Ommission insurance carrier! (They like written work orders...a lot.)

    Something went wrong here -- just on the face of it, it's obvious the system wasn't designed right or had never been tested properly. Beyond any specific technical procedures, anytime you're making a change that could affect stored data...you first make sure you have a good backup (or if you can't, see the previous paragraph!)

  10. #10
    Forum Member
    Bones42's Avatar
    Join Date
    Mar 2001
    Location
    Pt. Beach, NJ
    Posts
    10,703

    Default

    you first make sure you have a good backup
    As another IT person, this is a statement to live by.

    Have it happen to you once, and after all the work to rebuild what you just lost of a customer, you will never make that mistake twice.
    "This thread is being closed as it is off-topic and not related to the fire industry." - Isn't that what the Off Duty forum was for?

  11. #11
    IACOJ BOD
    FlyingKiwi's Avatar
    Join Date
    May 2002
    Location
    New Zealand
    Posts
    4,757

    Default

    And TEST the backup, how many times I have seen someone say...

    "We take backups every day."

    The look on their face when they learn..

    "yes you did, but none of the tapes were useable."

    Is bloody priceless. I have seen businesses hit the wall on this point.

    And I can get 500 Gb hard drives for USD $ 282.00 for a single unit, let alone bulk price.

    8 of these units are being built into our backup system (x 2 computers) at remote locations from our Internet Servers.

    And that is just off the shelf hardware without going to any extreme.

    IBM have the DS8000 series for enterprises with 1.1TB to 192 TB capacity available.

    And you can dasiy chain them.

    SOOOO....

    allowing for 12 months recording lets say 30TB

    192 / 30 = 6.4

    Oh, only 6 years of storage in that box, bugger.

    Cost vs the amount of tapes, off site storage, man hours etc is actually less over the lifespan of the box.
    Psychiatrists state 1 in 4 people has a mental illness.
    Look at three of your friends, if they are ok, your it.

  12. #12
    MembersZone Subscriber
    EFD840's Avatar
    Join Date
    Jun 2000
    Location
    Eclectic (no, NOT electric), Alabama
    Posts
    1,510

    Default

    Quote Originally Posted by nmfire
    I'm still trying to figure out what they use to record audio that uses a PC in the first place.
    Our recording system is entirely digital - it goes to magnetic disk and is then archived to DVD-RAM.

    Quick and easy.

    Oh, and add me to the IT manager group chanting 'backup, backup, backup'

    We're, RAID-5 with hot spares and full tape backups which are stored off site. You just can't take chances...

  13. #13
    Forum Member
    nmfire's Avatar
    Join Date
    Nov 2002
    Location
    Maryland (DC Suburb)
    Posts
    5,738

    Default

    We have a Mirra (built by NICE Systems) unit that records to it's own drive and archives to double-sided DVD RAM's. We can hold about three months of telephone and radio recordings on one side of one disk. They are not erasable or overwritable unless I put the DVD back in the drive and use my administrator override key to force the unit into overwrite mode. The thing is idiot proof until they make a better idiot.
    Even the burger-flippers at McDonald's probably have some McWackers.

  14. #14
    MembersZone Subscriber
    ullrichk's Avatar
    Join Date
    Jan 2002
    Location
    Deleted by the forum gremlins
    Posts
    1,663

    Default

    Blessed are the pessimists for they hath made backups.
    ullrichk
    a.k.a.
    perfesser

    a ship in a harbor is safe. . . but that's not what ships are for

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. ISO Company Personnel
    By FIRE549 in forum Firefighters Forum
    Replies: 20
    Last Post: 05-16-2007, 07:15 PM
  2. Safety Food For Thought
    By MalahatTwo7 in forum The Off Duty Forums
    Replies: 3
    Last Post: 06-02-2006, 09:50 AM
  3. 911 Phone Glitch
    By MalahatTwo7 in forum Firefighters Forum
    Replies: 14
    Last Post: 04-27-2006, 12:48 AM
  4. Niagara Active Hose fire company feeling the heat
    By Lewiston2FF in forum Firefighters Forum
    Replies: 27
    Last Post: 04-20-2005, 08:33 PM
  5. Lawsuit Blames Gas Company for a fire when the gas was off!!??
    By thefyreman in forum Firefighters Forum
    Replies: 14
    Last Post: 04-05-2002, 09:01 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Log in

Click here to log in or register