Topic and Location

Incident description

Flood – loss of electricity

Mississippi

This flood occurred due to floodgate control (at the dam) being overlooked in such a way as to protect the homes built along the river way.

The flood suddenly overflowed the dam and the local “elite” community could not manage the volume of water crashing down upon it.
Close-by staff required assistance in relocating family to other areas or States and was accomplished by direct and indirect funding and assistance.

The computer room facility was barely accessible but the power was off for days. Being a production facility room power and Air Conditioning were the crisis. Several home-style generator sets were brought to the building and tapped into wall power sockets. Adequate power became available to start the computer room and minimal lighting. Large blocks of dry ice were brought in by pickup and placed behind the computer cabinetry. Small fans were used to draft across the dry ice to cool the computers of which lasted for a few hours at a time before replacement ice was needed. The site met monthly production.

My role was to assist management in coordinating environmental adjustments suitable to sustain production, communicate with home-office c-level regarding the status of personnel and assets and to stay onsite for other duties as required in this minimal staff situation.

Monsoon – loss of electricity

Brunei

An expected seasonal monsoon arrived and flooded the coastline business structure where Shell housed its seismic operations.

Fortunately the original facility design provided for raising the floor by 1-foot for the computer room – which proved sufficiently for this flood. Water seepage was detected only on the windward side of the building and was contained at the wall.

To provide for contractual commitments the data processing software, key project data and selective paperwork was temporarily relocated to the nearby hotel (3^rd floor) to isolate it from immediate risk.

The facility survived the 1-week issue as power became very erratic.

My role included communicating with the home-office and with the client in regards to facility, production and personnel safety and production. I stayed in the computer room that night to ensure water retention which allowed the operators to concentrate on production. Over the next few weeks environmental changes were made to better mitigate future issues. My design changes were accepted and construction revisions were submitted.

Repeating Brown-Out

Singapore

A repeating brownout was causing data processing problems (varied degree of crash and restart) every Friday afternoon around 5:30pm. A local machine shop operated some equipment that they powered up early Monday morning (~4:30am) and left online until Friday afternoon (~5:30 to 6:00pm). The interruption caused an array of computer related problems on jobs that needed to run for days to weeks before completion. Extended conversation with all personnel lead to knowledge of hearing a noise every Friday about time as various workers waited for their ride home or bus. Visiting with the local companies led to the understanding of their power & production needs.

The processing center shifted its start/stop time and break time of night shift such that the day shift paused all production early enough as to miss the brown-out and started the night shift late enough to accomplish the same.

Monthly production increased as all other working hours realized clean power.
The local power company was contacted and they started monitoring and later added an additional transformer on the block.

My role was to educate c-level as to the nature of what needed to be done to correct this situation. From that discussion we selected the correct employee to communicate with the local utility company and from there own the communications to see the issue resolved.

Flood – 1-foot above ground floor

Summer Rain storm

Houston TX

This flood started as just another rain around 4:30 pm. By 6:00 pm, it was obvious that a flood was imminent and the basement and first floor were in jeopardy.

Sandbags that were available were placed at entrance ways and along first floor walls where leaks were detected. They were soon overwhelmed.

Seismic data (stored in basement) was moved via “human chain” to first floor and then to higher floors.

First floor computer rooms were shutdown, equipment that normally resided under the false floors were obtained and moved. Lower floor power was turned down to minimum areas.

In the following weeks all cables that previously laid on the concrete floor were moved to hanging trays about 1-foot up from the floor.

After considerable time of vacuuming water, the production center returned to operation with little loss other than time. Man of us quickly purchased supplies from neighborhood stores and spent the night in the computer room or offices.

My role was to identify problems that could affect production. From this the street level water was the primary issue as power was stable and the storm was low order in regards to lightning. From the maintenance staff, a team was established such that sandbags could be coordinated at perimeter doorways and at the basement to control the volume of water accessing the corporate data storage area. The team concluded water cleanup steps the following morning.

Electricity loss of City grid

Africa
Lagos

The location is a multi-floor-tenant building, which was accessed via a single elevator and of which was suddenly found without electricity.

It was summer time, on the equator, and 115 degrees F in the shade.

The cause of the citywide grid failure is unknown.

The problem is that the building engineer had not only falsified months of emergency generator test results; he had sold the on-board alternator (for engine starter batteries) so that a delivery truck could be repaired. I believe a family member owned the vehicle.

Contracts were violated as it was the major tenants responsibility to provide clean & full time electricity to the building such that vendors, contractors & visitors could use the building 24/7. Practically, a full day was lost before the emergency system could bring the building back online. Local "fees" were paid to expedite the procurement of replacement parts and repairs.

My role was to communicate with the local manager so that the proper contacts were made to invoke repair, increase the security guard count and define additional steps to better protect the staff. I further requested changes to include security updates in the monthly reports to accompany medical issues above $100.

Embezzlement

Taipei

A field manager working near China was in charge of the corporate working account for his field crew. This included operational and payroll expense responsibility. The funds were knowingly held in his personal bank account as to satisfy the local needs he often required cash to conclude business. The account had approximately $40,000 USD in it at the time of this incident.

For personal reasons and gain, he utilized the funds for procurement and debt relief to benefit himself.

The local authorities would take no action, as it was not their responsibility of his being a bad manager for that company.

The company had to make a decision of which meant leaving him onsite to conclude his contract or be fired and relocated home – also as per his contract.

The cost of hiring a new manager and transporting that person was also a consideration, which was further complicated by contractual obligations to return the field crew to work in a few days.

The weather was yet another consideration as it was approaching storm season and the total number of production days were few.

The final decision was to transfer the funds to him as a bonus and leave him onsite to conclude his overseas mission. Immediately upon return to Houston, his contract was reviewed and NOT renewed.

My role was to communicate with field services management to identify what expenses needed to be managed via the field work-crew and that of the corporate office. Banking procedures were changed such that lower funds were immediately on-hand in the field but better procurement and support funds were available for issue as needed. I determined that there were insufficient insurance policies in place for business operations and personnel. The submission was made to increase the level of coverage so as to mitigate future issues.

Theft by Deception

Africa
Lagos

Contractor obtained several sheets of corporate letterhead from a manager’s office of an expatriate company. The letterhead was utilized in the form of purchase orders to obtain oilfield equipment and a vehicle.

The company realized the matter upon being contacted by one of the oil field equipment shipping companies to clarify an address. This is when they determined that a 6-digit transaction was about to erupt at their bank.

Local authorities were quick to point out the company’s error in that the manager should not have had the letterhead in open/public view.

They described that the documents should have been under lock and key as within this country it was legal tender to use letterhead as a procurement tool.

About a year later, many legal fees (bribes or Dashes) later the local DA had the native African arrested for petty theft. The DA proclaimed that his crime was of stealing the paperwork and that the negligence of the local manager was the temptation for the overall act. Communications between the corporate office and local attorney was maintained throughout this process.

Some of the equipment was recovered and the vehicle was sold to cover the legal expenses of the contractor.

The legal system and assorted laws have improved.

A lockable filing cabinet was installed by weeks-end.

My role was to communicate with the regional office & home corporate office on the problem, the source, the extent and the resolution. Procedure were changed at the local office and policy was defined in both global regions by the regional office and the corporate office.

Authority abuse

South America Caracas

A call for an emergency computer repair was issued from South America. The most expedient means of getting several boxes of parts through customs was to assign someone to the task of hand-carrying them. A local manager was selected as she could secondarily perform software updates when the system was back online.

Upon arriving in a foreign country, she began to display her short-temper fuse and decided to claim only a small fraction of the actual value of the many boxes of computer parts. The actual value was ~$50,000 USD.

The attending customs agent took her information and departed to review past import records and examine the part numbers.

Upon returning he asked her to follow him and she was promptly escorted to a security holding area that within the airport on a Friday afternoon. Her increased level of tone and arrogance was met with calm and professionalism.

She was allowed to speak with the local employees; there to pickup her up and in so doing formulated a response to send her back to the corporate office. Funds were wire transferred ($150.000USD) to resolve this matter, however it was a couple of days later before she was allowed out of the security holding area.

In the meantime, a new shipment of parts (also hand-carried) was shipped and delivered to the computer center. A policy was generated to clear all parts shipments through the division procurement officer to declare the means and amount approved for future shipments.

My role was to coordinate the original and secondary equipment and shipping which involved customs paperwork. The inventory was relieved from the inventory system that I created for our division which was there to support the then fifteen data centers. Discussions with the VP's identified the issue and a policy was established for shipping into foreign countries. The foreign travel policy was altered to include customs and shipping expenses and management. Staff security was left at the educational level of each remote office - not by my advise.

Sniper Shots

Saudi Arabia Oil Field

Think about this, a day at work with normal issues to resolve, dodging sniper bullets, jumping from multi-story structures to the ground in emergency evacuation moves, hiding in closets; trashcans; or anything large enough to house your body.

After trying to get to the emergency escape line (on a Saudi oil Derek) a worker used a red rag to cling to and ride a support cable down to the ground. Unfortunately, the friction burned through the rag while he was about 30 feet in the air. The crashing plunge he made to the ground was actually to his advantage as the sniper had just repositioned and was only another shot or two away from hitting him.

Keep in mind that the weapon of choice was an AK47 or similar fully automatic assault weapon.

Once home (Singapore) the native Hindu seemed normal as we sat around the dinner table.

Yet, an event occurred that really made it obvious that he was still recovering from believing that he was going to die that day at work. The event was the taunting of him by his brother of which had dropped a spoon to the floor. The brother screamed, wrapped his face with his hands, and jumped to and fro knocking over the chair and crashing into other people and household objects.

After a few minutes of some rather “sharp” communications, we all began laughing as the attention moved on to his leapfrog dance.

Clearly, he was in no condition to return to the Saudi oil field. He remained home through the next year and finally began working at a small business in Singapore. The business concern was that he was still too unstable after this event to spend a day among other employees without relaying this ill event to them and creating an unstable working environment.

My role here was to recommend to regional management my view on his ability to perform his job. In this case, he needed more time for recovery before returning to work. Therapy was recommended. The system (policies & procedures) worked as they were.

Earth Quake (The Big One)

San Jose CA

A computer service & repair partnership was running along between Houston, San Jose and Tokyo.

One day while on the phone (Hou-San J.) a series of wild calls were made the voice that had been muttering part numbers had stopped. I dropped the line after waiting a few seconds and then recalled a few minutes later.

The remote voice did return but was exceptionally out of character. Suddenly a set of instructions began flying across the phone line, send a fax to this VP, send email to that Manager, don’t hang up the phone for any reason, send more faxes, etc.

Some time in the process of sending faxes to Tokyo I received a call from them indicating that they had no evidence of a problem in CA and were questioning my data. After explaining to the secretary that the call was real and that I demanded that she wake her boss (VP was taking a nap) she reluctantly did and relayed the fax.

Shortly after confirming to the frantic voice in CA that the Tokyo office was updated the phone line was dropped and did not return for about 3 days.

At this time there was no BC/DR process even thought of for this business venture, and so it was observed by the lack of knowledge that when a emergency call came in it was going to be real and that no pranks would be ever utilized in business communications.
Production and inventory shipping delays continued for weeks as the lack of forecasting a disaster did what it is capable of doing - closed the doors for weeks.

My role was to create the outline for a BC-DR plan such that the domestic and foreign partners could collaborate on a final business continuity plan.

Felony Theft

Houston TX

Paperwork, paperwork, who has the paperwork.

Well one thing is for sure, you best know who has the insurance papers when handling custom designed HiPPI to FIDDI interfaces.

It was a simple task of moving from one office to another. The move was just between one building and the next. It was Saturday and there was no traffic in the parking lot.

What could possibly go wrong????

Well, the traffic changed for about one minute.

Upon emerging from the new office, headed for the next equipment load, a strange vehicle was discovered at the corner of the sidewalk, which was being used as a staging area.

A discussion pursued with the unknown men whom had collected several items from the “staging area” and politely proclaimed that they though the items were there for them to pickup as they were "The Scrap Men". As best that it could be determined, things had been returned and they departed.

The fact that there was no vehicle license plate was further disturbing.

No problem, just turn in an insurance claim and get a replacement moving.

Right – NOT!

The vendor had been shipping millions of dollars of equipment around the US, for this project and others, and had no method to relieve an expense of this nature. At this point I was shaking in my boots as I did not want to be stuck with this ~$50,000 problem. Fortunately, a compromise was made between the manufacturer and the vendor and a replacement arrived two weeks later at a mere ¼ the original value.
Hmmm! Is this a good time to discuss discounts, now that the real value of the board is known?

Assumed, Direct or Indirect Liability is still LIABILITY. Always obtain proof of protection via hardcopy and don't operate a business w/o liability coverage.

My role was to define the incident and provide common data and literature as if filing an insurance claim. This documentation was issued to the shipping vendor for their internal report as there was no other action. The ROI was based on the mass quantity of shipping of this nature they did (across 3-years) and the minimal claims did not warrant the additional insurance premium cost.

Offshore “Man Overboard”

US
East Coast

Seismic production work-boat. Approaching hurricane headed from Atlanta toward Rhode Island - New England.

Seismic data cable is over 2-miles long. Lots of repair underway due to storm and leaks (kerosene) from data cable.

Workman slips and was able to grab lifeline rope. Workman was underwater the entire time while off deck.

Crew Chief properly notified Captain of “Man Overboard” status.

Engines were shutdown and all hands moved to their emergency stations. Workman retrieved, via safety rope, with new respect for horsepower and moving water. No injury beyond minor scratches and bruising.

The “rehearsed” emergency procedures proved successful.
A valuable lesson for the crew that the practice Fire Alarms and other drills were there for a purpose. Upon review of current procedures additional steps and drills were added for all vessels.

My role was to report what I saw as I was on the back deck when the crewman went overboard. It was apparent that the cause was a simple accident which occurs when working on a foundation that is moving and wet. No changes were recommended as the process of announcing the incident and the wheel-house's immediate response worked as needed.

Computer Equipment

Guarantee Cancellation

Freeport, TX.

A seismic boat was outfitted with multiples of millions of dollars of computer, navigation, power protection-regulation and other equipment for collecting and processing data.

The disk drive technology was that of a removable media pack. This 3-foot tall near “washing machine” sized unit was equally heavy to its mass.

The problem was that upon entering the open seas the manufacturer would cancel the warranty of this or any other equipment of theirs.

The concern was regarding a crash (r/w heads contacting the media surface) and subsequent replacement parts.

A unique method of using oversize Air Bag suspension mounts (under inflated) allowed the unit to operate in seas so high that the waves were clearing the wheel-house. (over 12' seas)

Though full liability was passed on to the user the unique mounting process allowed the disk drive to operate flawlessly throughout its multi-year journey.
Proper environmental design was the key to this production story. A standard sea-bound mounting scheme would have been too rigid and would have crashed the heads on this unit which would shut down production and invoking a $250,000 per day contract penalty.

My role was to define my alternative mounting design as it proved successful in a high-seas environment. The design criteria for future seismic vessels was altered to allow for rough seas which would allow the newer & higher capacity disk drives to function on the boats.

Diesel Fire

On-board in the Red Sea

As with any vehicle a boat needs maintenance. This situation involved general diesel engine maintenance concerning filters. (fuel, oil & air)

An unexpected interruption caused the mechanic and other crew members to halt their current activities and move forward within the vessel. Unbeknown to anyone the last fuel filter connection “apparently” may have not been as tight as possible or perhaps the new fuel filter was faulty. In any event, a fuel leak began to cover the back-deck with diesel fuel. At some point the fuel came in contact with an electrical device or something may have fell to cause a spark that started a fire.

Before the wheelhouse detected the diesel smoke a crewman returned to the back deck and immediately depressed the fire alarm. Automatically an SOS signal was initiated and all ship members hit the deck in a full run. A water-hose was used to attempt to guide the diesel fuel off the back deck and away from other flammable items, including diesel storage tanks. Keeping in mind that water, diesel and fire are not a mutual mix of elements the fire got worse.

A crew member announced the Abandon Ship notice and emergency procedures began. Lifeboats were launched, key items were secured from the Skippers cabin and wheelhouse and dozens of people now migrated into the Red Sea.

Emergency procedures worked, no injuries were incurred, the crew was fished from the sea by local fishing boats (what coast guard), the boat sank and the insurance company was not happy. The daily work procedure was altered to include a watchman (secondary to the fireman) during any procedures that could expose any flammable fuel or oil to the deck area to include all maintenance operations or unscheduled repairs.

My role was to discuss the situation with the onboard processing center manager and the involved crewman. The report was issued to the corporate office for review. Emergency and maintenance procedures were updated on all boats.

Murder -

“Eye for Eye”

Justice

Africa - Lagos

The living quarters were located some ~30 miles from the general office and computer system.

The living quarters (quadplex) used the ground floor utility & storage rooms as a home for the family that serviced the quarters and the guards that protected the property and its personnel.

One weekend morning a small disturbance was noticed near the front gate. Upon returning upstairs, (breakfast time) the cook was questioned about the number of people downstairs. When questioned, the cook notified one of the expatriates (wife of computer processing crew member) that a thief attempted to enter the compound last night. She innocently continued asking questions and the cook continued to supply data. Mistaking the curiosity for interest the cook took the wife downstairs to see the results of the incident.

Upon getting downstairs, they exited the gate and approached the trashcans. As they approached the two remaining guards moved away one returned to his post outside of the gated area. The cook (attempting to please) concurrently lifted the lid from two trashcans and seemed quite surprised when the wife passed out.

The thief had been caught and local justice was prevailed the thief was killed, decapitated and quartered like a cow off to butcher.

By now the additional commotion stirred other members of the quadplex. Upon determining the situation, various people scattered (some calling Europe) and others helped to carry the wife upstairs. The next few weeks were difficult as production diminished with the wife leaving the country and the husband had to stay or forfeit their work contract. Several other crew members also missed time and most were very vocal in their attempts to take permanent leave from their foreign post.

The attempts to leave and request for replacements fell on deaf ears, so the same request were issued back to the US corporate office. To the best of my knowledge, the company policy of reverting to the decision of the regional office is still active.

It took about a month to get the wife back to England and into proper psychiatric treatment of which lasted for several years and inevitable outlived the remaining time of their marriage.

The local police department stopped by, I believe the next day, to take a report. The command decision was made to hire an additional guard and make certain fence-line repairs as entry had become easy in the darkest corner of the backyard.

My role was to document the situation and report. The procedural change submitted was to better inform (during interview) the contractor and stipulate that communications had followed the interview to include any family members expected to spend time on the foreign assignment.

Corruption and Bribes

Traffic Police

Africa
Lagos

One day several of us gathered to plan a Friday night-out. Apparently, the selection of the day coincided with some event seen on the news and so what the heck, lets go.

The news event turned out to be that the police were looking for a criminal whom was suspected to be fleeing from a neighboring city and headed in our direction. Later I learned that the police would be setting up several police checks (roadside) to search vehicles in the pursuit of this criminal.

When Friday arrived, additional instructions were given. After taking inventory of the money that each individual had we were instructed to place different amounts of money into different pockets, setting ourselves up to draw amounts as needed when called upon as we paid our bribes to get through the police checks. I had to wonder about driving ~100 miles to spend a few hours at a Greek dance club (ballet or something).

As we returned, after midnight, we encountered the first of three police checks. The drivers quick description lead us to understand that the higher the rank of the officer the higher the bribe (or dash). The driver quickly estimated what we were up against and so we made our plans to produce funds.

This, the first stop was short and not very expensive as Privates were the top ranking. By the time we reached the final checkpoint (third) one of the passengers was really freaking out. Our “fun trip” then started to lean toward a potential night of “hell in a foreign jail”.

We arrived at the final checkpoint and all had a sinking feeling. There must have been a dozen cops with one of them having several pounds of braid, silver & gold metals and looked more like a military general than an experienced police officer. Last minute instructions were for everyone to stay calm, do NOT speak unless spoken DIRECTLY to, and recount your remaining money. More effort was invested in keeping the panic level down.

The span of three officers talking with us of which included several of us having various weapons pointed at us and for enticement they would poke the driver, later myself as well, with a pistol (45 automatic) and an automatic assault weapon (AK47). The good news is that I think the “Thompson Machine gun” was inoperable but it did make a small dent in my temple where it had impacted several times.

Bottom line, they got our money, we had fun, and we got back to the apartment.

I stayed down stairs with the guards for a while and tossed down a few additional beverages. As I told them of the venture, they laughed. I was (finally) glad to be such a source of entertainment for them, "behind that locked gate".

The point of this - ALWAYS know the RULES when in someone else’s BACKYARD.

My role was to outline a security segment for the interview and policies & procedures manual. This outline defined normal and abnormal situations which could be faced in that country.

Excellent electrical power for some.

Indonesia

The company assigned me that task of setting up a seismic data processing center in Indonesia.

Generally, we knew that the power was bad and that a UPS system would be required. At that time it was not known that we would be on the 9^th floor and that the only place to locate a UPS system was in the basement and within a small room that would have to be built and air conditioned.

Upon arrival, I made contact with the building management and their engineer. We made appointments to visit the building, the building designer, the local French company that provided electricity to the block and several surrounding businesses.

The feedback provided that power outside of the “block” (Indonesian government area) was undependable and out for hours or days at a time. The French power system was excellent and completely backed up to cover 110% of the needs of the block.

Specifying the correct size of the UPS system was somewhat like playing with perpetual motion in that you had to add power to compensate for air conditioning to compensate for the heat within the UPS room. Wiring compensation, oversize wire, was made to provide for being 10+ stories away from the computer.

Other than a few incidents: the air conditioner being stolen from the UPS room (yanked from window from within a secure parking garage), the unit overheated from the AC thermostat going out in the summer time and “roasting” the UPS electronics, the system provided clean power for years.
BC-DR plans need to allow for bribes and theft along with other normal environmental conditions.

My role was to design, order, implement a stable environment (HVAC, AC, access control) for the center. The interview with the utility company, and contractors, lead to the installation and site start-up. I hired a local technical assistant that would aid the field engineer that was to be selected for the site. After performing onsite training with the assistant I departed from Indonesia. Long-term inventory control and Level-1 technical support was provided from the staff that I originally work within and then managed from the company home office.

Backup - never tested
Recover - never tested

Houston, TX

A local database service provider had backup/recover scripts written for their rather old (70’s vintage) computer. The company operated for over five years before exploring the avenue of upgrading their ½” tape drives to something more modern.

I visited with them and showed them a custom controller that could be adapted to fit their exact needs and further be compatible with their application software.

In order to perform our task we had to understand their current backup process and dataset volume. Within that exploration, it was discovered that their BU tapes were blank. They had apparently operated for at least 3-years since updating the BU script and had NEVER tested the output tapes as it was too much trouble to be their at midnight when the script ran. A simple error (data density setting) caused the script to error immediately upon initiation. The script did not test for error conditions and simply skipped to the rewind/unload phase after detecting the tape density.

The manager began mumbling something about being fired, never being able to work again, and a few other things before realizing that he was venting directly in my presence.

Backup’s that are not tested are avenues of disaster looking for the time to happen.

By not testing the recovery process, the business liability grew from small to enormous (~$100 million).

The tape upgrade project and the manager were terminated. The new manager (two months later) claimed no knowledge of this matter or the need to alter their system or their procedures.

My role was to demonstrate the product and assist the client in determining their needs with regards to implementing our hardware solution. We altered the marketing approach to better interview potential clients. No other actions were needed.