Flood – loss
- This flood occurred due to
floodgate control (at the dam) being overlooked in such a way as
to protect the homes built along the river way.
The flood suddenly overflowed the
dam and the local “elite” community could not manage
the volume of water crashing down upon it.
required assistance in relocating family to other areas or States
and was accomplished by direct and indirect funding and
The computer room facility was
barely accessible but the power was off for days. Being a
production facility room power and Air Conditioning were the
crisis. Several home-style generator sets were brought to the
building and tapped into wall power sockets. Adequate power
became available to start the computer room and minimal lighting.
Large blocks of dry ice were brought in by pickup and placed
behind the computer cabinetry. Small fans were used to draft
across the dry ice to cool the computers of which lasted for a
few hours at a time before replacement ice was needed. The site
met monthly production.
My role was to assist management
in coordinating environmental adjustments suitable to sustain
production, communicate with home-office c-level regarding the
status of personnel and assets and to stay onsite for other
duties as required in this minimal staff situation.
loss of electricity
- An expected seasonal monsoon
arrived and flooded the coastline business structure where Shell
housed its seismic operations.
Fortunately the original facility
design provided for raising the floor by 1-foot for the computer
room – which proved sufficiently for this flood. Water
seepage was detected only on the windward side of the building
and was contained at the wall.
To provide for contractual
commitments the data processing software, key project data and
selective paperwork was temporarily relocated to the nearby hotel
(3rd floor) to isolate it from immediate risk.
The facility survived the 1-week
issue as power became very erratic.
My role included communicating
with the home-office and with the client in regards to facility,
production and personnel safety and production. I stayed in the
computer room that night to ensure water retention which allowed
the operators to concentrate on production. Over the next few
weeks environmental changes were made to better mitigate future
issues. My design changes were accepted and construction
revisions were submitted.
- A repeating brownout was
causing data processing problems (varied degree of crash and
restart) every Friday afternoon around 5:30pm. A local machine
shop operated some equipment that they powered up early Monday
morning (~4:30am) and left online until Friday afternoon (~5:30
to 6:00pm). The interruption caused an array of computer related
problems on jobs that needed to run for days to weeks before
completion. Extended conversation with all personnel lead to
knowledge of hearing a noise every Friday about time as various
workers waited for their ride home or bus. Visiting with the
local companies led to the understanding of their power &
The processing center shifted its
start/stop time and break time of night shift such that the day
shift paused all production early enough as to miss the brown-out
and started the night shift late enough to accomplish the same.
Monthly production increased as
all other working hours realized clean power.
The local power
company was contacted and they started monitoring and later added
an additional transformer on the block.
My role was to educate c-level as
to the nature of what needed to be done to correct this
situation. From that discussion we selected the correct employee
to communicate with the local utility company and from there own
the communications to see the issue resolved.
1-foot above ground floor
- Summer Rain storm
- This flood started as just
another rain around 4:30 pm. By 6:00 pm, it was obvious that a
flood was imminent and the basement and first floor were in
Sandbags that were available were
placed at entrance ways and along first floor walls where leaks
were detected. They were soon overwhelmed.
Seismic data (stored in basement)
was moved via “human chain” to first floor and then
to higher floors.
First floor computer rooms were
shutdown, equipment that normally resided under the false floors
were obtained and moved. Lower floor power was turned down to
In the following weeks all cables
that previously laid on the concrete floor were moved to hanging
trays about 1-foot up from the floor.
After considerable time of
vacuuming water, the production center returned to operation with
little loss other than time. Man of us quickly purchased supplies
from neighborhood stores and spent the night in the computer room
My role was to identify problems
that could affect production. From this the street level water
was the primary issue as power was stable and the storm was low
order in regards to lightning. From the maintenance staff, a team
was established such that sandbags could be coordinated at
perimeter doorways and at the basement to control the volume of
water accessing the corporate data storage area. The team
concluded water cleanup steps the following morning.
of City grid
- The location is a
multi-floor-tenant building, which was accessed via a single
elevator and of which was suddenly found without electricity.
It was summer time, on the
equator, and 115 degrees F in the shade.
The cause of the citywide grid
failure is unknown.
The problem is that the building
engineer had not only falsified months of emergency generator
test results; he had sold the on-board alternator (for engine
starter batteries) so that a delivery truck could be repaired. I
believe a family member owned the vehicle.
Contracts were violated as it was
the major tenants responsibility to provide clean & full time
electricity to the building such that vendors, contractors &
visitors could use the building 24/7. Practically, a full day was
lost before the emergency system could bring the building back
online. Local "fees" were paid to expedite the
procurement of replacement parts and repairs.
My role was to communicate with
the local manager so that the proper contacts were made to invoke
repair, increase the security guard count and define additional
steps to better protect the staff. I further requested changes to
include security updates in the monthly reports to accompany
medical issues above $100.
- A field manager working near
China was in charge of the corporate working account for his
field crew. This included operational and payroll expense
responsibility. The funds were knowingly held in his personal
bank account as to satisfy the local needs he often required cash
to conclude business. The account had approximately $40,000 USD
in it at the time of this incident.
For personal reasons and gain, he
utilized the funds for procurement and debt relief to benefit
The local authorities would take
no action, as it was not their responsibility of his being a bad
manager for that company.
The company had to make a
decision of which meant leaving him onsite to conclude his
contract or be fired and relocated home – also as per his
The cost of hiring a new manager
and transporting that person was also a consideration, which was
further complicated by contractual obligations to return the
field crew to work in a few days.
The weather was yet another
consideration as it was approaching storm season and the total
number of production days were few.
The final decision was to
transfer the funds to him as a bonus and leave him onsite to
conclude his overseas mission. Immediately upon return to
Houston, his contract was reviewed and NOT renewed.
My role was to communicate with
field services management to identify what expenses needed to be
managed via the field work-crew and that of the corporate office.
Banking procedures were changed such that lower funds were
immediately on-hand in the field but better procurement and
support funds were available for issue as needed. I determined
that there were insufficient insurance policies in place for
business operations and personnel. The submission was made to
increase the level of coverage so as to mitigate future issues.
Theft by Deception
- Contractor obtained several
sheets of corporate letterhead from a manager’s office of
an expatriate company. The letterhead was utilized in the form of
purchase orders to obtain oilfield equipment and a vehicle.
The company realized the matter
upon being contacted by one of the oil field equipment shipping
companies to clarify an address. This is when they determined
that a 6-digit transaction was about to erupt at their bank.
Local authorities were quick to
point out the company’s error in that the manager should
not have had the letterhead in open/public view.
They described that the documents
should have been under lock and key as within this country it was
legal tender to use letterhead as a procurement tool.
About a year later, many legal
fees (bribes or Dashes) later the local DA had the native African
arrested for petty theft. The DA proclaimed that his crime was of
stealing the paperwork and that the negligence of the local
manager was the temptation for the overall act. Communications
between the corporate office and local attorney was maintained
throughout this process.
Some of the equipment was
recovered and the vehicle was sold to cover the legal expenses of
The legal system and assorted
laws have improved.
A lockable filing cabinet was
installed by weeks-end.
My role was to communicate with
the regional office & home corporate office on the problem,
the source, the extent and the resolution. Procedure were changed
at the local office and policy was defined in both global regions
by the regional office and the corporate office.
South America Caracas
- A call for an emergency
computer repair was issued from South America. The most expedient
means of getting several boxes of parts through customs was to
assign someone to the task of hand-carrying them. A local manager
was selected as she could secondarily perform software updates
when the system was back online.
Upon arriving in a foreign
country, she began to display her short-temper fuse and decided
to claim only a small fraction of the actual value of the many
boxes of computer parts. The actual value was ~$50,000 USD.
The attending customs agent took
her information and departed to review past import records and
examine the part numbers.
Upon returning he asked her to
follow him and she was promptly escorted to a security holding
area that within the airport on a Friday afternoon. Her increased
level of tone and arrogance was met with calm and
She was allowed to speak with the
local employees; there to pickup her up and in so doing
formulated a response to send her back to the corporate office.
Funds were wire transferred ($150.000USD) to resolve this matter,
however it was a couple of days later before she was allowed out
of the security holding area.
In the meantime, a new shipment
of parts (also hand-carried) was shipped and delivered to the
computer center. A policy was generated to clear all parts
shipments through the division procurement officer to declare the
means and amount approved for future shipments.
My role was to coordinate the
original and secondary equipment and shipping which involved
customs paperwork. The inventory was relieved from the inventory
system that I created for our division which was there to support
the then fifteen data centers. Discussions with the VP's
identified the issue and a policy was established for shipping
into foreign countries. The foreign travel policy was altered to
include customs and shipping expenses and management. Staff
security was left at the educational level of each remote office
- not by my advise.
Saudi Arabia Oil Field
- Think about this, a day at
work with normal issues to resolve, dodging sniper bullets,
jumping from multi-story structures to the ground in emergency
evacuation moves, hiding in closets; trashcans; or anything large
enough to house your body.
After trying to get to the
emergency escape line (on a Saudi oil Derek) a worker used a red
rag to cling to and ride a support cable down to the ground.
Unfortunately, the friction burned through the rag while he was
about 30 feet in the air. The crashing plunge he made to the
ground was actually to his advantage as the sniper had just
repositioned and was only another shot or two away from hitting
Keep in mind that the weapon of
choice was an AK47 or similar fully automatic assault weapon.
Once home (Singapore) the native
Hindu seemed normal as we sat around the dinner table.
Yet, an event occurred that
really made it obvious that he was still recovering from
believing that he was going to die that day at work. The event
was the taunting of him by his brother of which had dropped a
spoon to the floor. The brother screamed, wrapped his face with
his hands, and jumped to and fro knocking over the chair and
crashing into other people and household objects.
After a few minutes of some
rather “sharp” communications, we all began laughing
as the attention moved on to his leapfrog dance.
Clearly, he was in no condition
to return to the Saudi oil field. He remained home through the
next year and finally began working at a small business in
Singapore. The business concern was that he was still too
unstable after this event to spend a day among other employees
without relaying this ill event to them and creating an unstable
My role here was to recommend to
regional management my view on his ability to perform his job. In
this case, he needed more time for recovery before returning to
work. Therapy was recommended. The system (policies &
procedures) worked as they were.
Earth Quake (The
San Jose CA
- A computer service &
repair partnership was running along between Houston, San Jose
One day while on the phone
(Hou-San J.) a series of wild calls were made the voice that had
been muttering part numbers had stopped. I dropped the line after
waiting a few seconds and then recalled a few minutes later.
The remote voice did return but
was exceptionally out of character. Suddenly a set of
instructions began flying across the phone line, send a fax to
this VP, send email to that Manager, don’t hang up the
phone for any reason, send more faxes, etc.
Some time in the process of
sending faxes to Tokyo I received a call from them indicating
that they had no evidence of a problem in CA and were questioning
my data. After explaining to the secretary that the call was real
and that I demanded that she wake her boss (VP was taking a nap)
she reluctantly did and relayed the fax.
Shortly after confirming to the
frantic voice in CA that the Tokyo office was updated the phone
line was dropped and did not return for about 3 days.
At this time there was no BC/DR
process even thought of for this business venture, and so it was
observed by the lack of knowledge that when a emergency call came
in it was going to be real and that no pranks would be ever
utilized in business communications.
Production and inventory
shipping delays continued for weeks as the lack of forecasting a
disaster did what it is capable of doing - closed the doors for
My role was to create the outline
for a BC-DR plan such that the domestic and foreign partners
could collaborate on a final business continuity plan.
- Paperwork, paperwork, who has
Well one thing is for sure, you
best know who has the insurance papers when handling custom
designed HiPPI to FIDDI interfaces.
It was a simple task of moving
from one office to another. The move was just between one
building and the next. It was Saturday and there was no traffic
in the parking lot.
What could possibly go wrong????
Well, the traffic changed for
about one minute.
Upon emerging from the new
office, headed for the next equipment load, a strange vehicle was
discovered at the corner of the sidewalk, which was being used as
a staging area.
A discussion pursued with the
unknown men whom had collected several items from the “staging
area” and politely proclaimed that they though the items
were there for them to pickup as they were "The Scrap Men".
As best that it could be determined, things had been returned and
The fact that there was no
vehicle license plate was further disturbing.
No problem, just turn in an
insurance claim and get a replacement moving.
Right – NOT!
The vendor had been shipping
millions of dollars of equipment around the US, for this project
and others, and had no method to relieve an expense of this
nature. At this point I was shaking in my boots as I did not want
to be stuck with this ~$50,000 problem. Fortunately, a compromise
was made between the manufacturer and the vendor and a
replacement arrived two weeks later at a mere ¼ the
Hmmm! Is this a good time to discuss
discounts, now that the real value of the board is known?
Assumed, Direct or Indirect
Liability is still LIABILITY. Always obtain proof of protection
via hardcopy and don't operate a business w/o liability coverage.
My role was to define the
incident and provide common data and literature as if filing an
insurance claim. This documentation was issued to the shipping
vendor for their internal report as there was no other action.
The ROI was based on the mass quantity of shipping of this nature
they did (across 3-years) and the minimal claims did not warrant
the additional insurance premium cost.
- Seismic production work-boat.
Approaching hurricane headed from Atlanta toward Rhode Island -
Seismic data cable is over
2-miles long. Lots of repair underway due to storm and leaks
(kerosene) from data cable.
Workman slips and was able to
grab lifeline rope. Workman was underwater the entire time while
Crew Chief properly notified
Captain of “Man Overboard” status.
Engines were shutdown and all
hands moved to their emergency stations. Workman retrieved, via
safety rope, with new respect for horsepower and moving water. No
injury beyond minor scratches and bruising.
emergency procedures proved successful.
A valuable lesson for
the crew that the practice Fire Alarms and other drills were
there for a purpose. Upon review of current procedures additional
steps and drills were added for all vessels.
My role was to report what I saw
as I was on the back deck when the crewman went overboard. It was
apparent that the cause was a simple accident which occurs when
working on a foundation that is moving and wet. No changes were
recommended as the process of announcing the incident and the
wheel-house's immediate response worked as needed.
- A seismic boat was outfitted
with multiples of millions of dollars of computer, navigation,
power protection-regulation and other equipment for collecting
and processing data.
The disk drive technology was
that of a removable media pack. This 3-foot tall near “washing
machine” sized unit was equally heavy to its mass.
The problem was that upon
entering the open seas the manufacturer would cancel the warranty
of this or any other equipment of theirs.
The concern was regarding a crash
(r/w heads contacting the media surface) and subsequent
A unique method of using oversize
Air Bag suspension mounts (under inflated) allowed the unit to
operate in seas so high that the waves were clearing the
wheel-house. (over 12' seas)
Though full liability was passed
on to the user the unique mounting process allowed the disk drive
to operate flawlessly throughout its multi-year journey.
environmental design was the key to this production story. A
standard sea-bound mounting scheme would have been too rigid and
would have crashed the heads on this unit which would shut down
production and invoking a $250,000 per day contract penalty.
My role was to define my
alternative mounting design as it proved successful in a
high-seas environment. The design criteria for future seismic
vessels was altered to allow for rough seas which would allow the
newer & higher capacity disk drives to function on the boats.
On-board in the Red Sea
- As with any vehicle a boat
needs maintenance. This situation involved general diesel engine
maintenance concerning filters. (fuel, oil & air)
An unexpected interruption caused
the mechanic and other crew members to halt their current
activities and move forward within the vessel. Unbeknown to
anyone the last fuel filter connection “apparently”
may have not been as tight as possible or perhaps the new fuel
filter was faulty. In any event, a fuel leak began to cover the
back-deck with diesel fuel. At some point the fuel came in
contact with an electrical device or something may have fell to
cause a spark that started a fire.
Before the wheelhouse detected
the diesel smoke a crewman returned to the back deck and
immediately depressed the fire alarm. Automatically an SOS signal
was initiated and all ship members hit the deck in a full run. A
water-hose was used to attempt to guide the diesel fuel off the
back deck and away from other flammable items, including diesel
storage tanks. Keeping in mind that water, diesel and fire are
not a mutual mix of elements the fire got worse.
A crew member announced the
Abandon Ship notice and emergency procedures began. Lifeboats
were launched, key items were secured from the Skippers cabin and
wheelhouse and dozens of people now migrated into the Red Sea.
Emergency procedures worked, no
injuries were incurred, the crew was fished from the sea by local
fishing boats (what coast guard), the boat sank and the insurance
company was not happy. The daily work procedure was altered to
include a watchman (secondary to the fireman) during any
procedures that could expose any flammable fuel or oil to the
deck area to include all maintenance operations or unscheduled
My role was to discuss the
situation with the onboard processing center manager and the
involved crewman. The report was issued to the corporate office
for review. Emergency and maintenance procedures were updated on
Africa - Lagos
- The living quarters were
located some ~30 miles from the general office and computer
The living quarters (quadplex)
used the ground floor utility & storage rooms as a home for
the family that serviced the quarters and the guards that
protected the property and its personnel.
One weekend morning a small
disturbance was noticed near the front gate. Upon returning
upstairs, (breakfast time) the cook was questioned about the
number of people downstairs. When questioned, the cook notified
one of the expatriates (wife of computer processing crew member)
that a thief attempted to enter the compound last night. She
innocently continued asking questions and the cook continued to
supply data. Mistaking the curiosity for interest the cook took
the wife downstairs to see the results of the incident.
Upon getting downstairs, they
exited the gate and approached the trashcans. As they approached
the two remaining guards moved away one returned to his post
outside of the gated area. The cook (attempting to please)
concurrently lifted the lid from two trashcans and seemed quite
surprised when the wife passed out.
The thief had been caught and
local justice was prevailed the thief was killed, decapitated and
quartered like a cow off to butcher.
By now the additional commotion
stirred other members of the quadplex. Upon determining the
situation, various people scattered (some calling Europe) and
others helped to carry the wife upstairs. The next few weeks were
difficult as production diminished with the wife leaving the
country and the husband had to stay or forfeit their work
contract. Several other crew members also missed time and most
were very vocal in their attempts to take permanent leave from
their foreign post.
The attempts to leave and request
for replacements fell on deaf ears, so the same request were
issued back to the US corporate office. To the best of my
knowledge, the company policy of reverting to the decision of the
regional office is still active.
It took about a month to get the
wife back to England and into proper psychiatric treatment of
which lasted for several years and inevitable outlived the
remaining time of their marriage.
The local police department
stopped by, I believe the next day, to take a report. The command
decision was made to hire an additional guard and make certain
fence-line repairs as entry had become easy in the darkest corner
of the backyard.
My role was to document the
situation and report. The procedural change submitted was to
better inform (during interview) the contractor and stipulate
that communications had followed the interview to include any
family members expected to spend time on the foreign assignment.
- One day several of us
gathered to plan a Friday night-out. Apparently, the selection of
the day coincided with some event seen on the news and so what
the heck, lets go.
The news event turned out to be
that the police were looking for a criminal whom was suspected to
be fleeing from a neighboring city and headed in our direction.
Later I learned that the police would be setting up several
police checks (roadside) to search vehicles in the pursuit of
When Friday arrived, additional
instructions were given. After taking inventory of the money that
each individual had we were instructed to place different amounts
of money into different pockets, setting ourselves up to draw
amounts as needed when called upon as we paid our bribes to get
through the police checks. I had to wonder about driving ~100
miles to spend a few hours at a Greek dance club (ballet or
As we returned, after midnight,
we encountered the first of three police checks. The drivers
quick description lead us to understand that the higher the rank
of the officer the higher the bribe (or dash). The driver quickly
estimated what we were up against and so we made our plans to
This, the first stop was short
and not very expensive as Privates were the top ranking. By the
time we reached the final checkpoint (third) one of the
passengers was really freaking out. Our “fun trip”
then started to lean toward a potential night of “hell in a
We arrived at the final
checkpoint and all had a sinking feeling. There must have been a
dozen cops with one of them having several pounds of braid,
silver & gold metals and looked more like a military general
than an experienced police officer. Last minute instructions were
for everyone to stay calm, do NOT speak unless spoken DIRECTLY
to, and recount your remaining money. More effort was invested in
keeping the panic level down.
The span of three officers
talking with us of which included several of us having various
weapons pointed at us and for enticement they would poke the
driver, later myself as well, with a pistol (45 automatic) and an
automatic assault weapon (AK47). The good news is that I think
the “Thompson Machine gun” was inoperable but it did
make a small dent in my temple where it had impacted several
Bottom line, they got our money,
we had fun, and we got back to the apartment.
I stayed down stairs with the
guards for a while and tossed down a few additional beverages. As
I told them of the venture, they laughed. I was (finally) glad to
be such a source of entertainment for them, "behind that
The point of this - ALWAYS know
the RULES when in someone else’s BACKYARD.
My role was to outline a security
segment for the interview and policies & procedures manual.
This outline defined normal and abnormal situations which could
be faced in that country.
electrical power for some.
- The company assigned me that
task of setting up a seismic data processing center in Indonesia.
Generally, we knew that the power
was bad and that a UPS system would be required. At that time it
was not known that we would be on the 9th floor and
that the only place to locate a UPS system was in the basement
and within a small room that would have to be built and air
Upon arrival, I made contact with
the building management and their engineer. We made appointments
to visit the building, the building designer, the local French
company that provided electricity to the block and several
The feedback provided that power
outside of the “block” (Indonesian government area)
was undependable and out for hours or days at a time. The French
power system was excellent and completely backed up to cover 110%
of the needs of the block.
Specifying the correct size of
the UPS system was somewhat like playing with perpetual motion in
that you had to add power to compensate for air conditioning to
compensate for the heat within the UPS room. Wiring compensation,
oversize wire, was made to provide for being 10+ stories away
from the computer.
Other than a few incidents: the
air conditioner being stolen from the UPS room (yanked from
window from within a secure parking garage), the unit overheated
from the AC thermostat going out in the summer time and
“roasting” the UPS electronics, the system provided
clean power for years.
BC-DR plans need to allow for bribes
and theft along with other normal environmental conditions.
My role was to design, order,
implement a stable environment (HVAC, AC, access control) for the
center. The interview with the utility company, and contractors,
lead to the installation and site start-up. I hired a local
technical assistant that would aid the field engineer that was to
be selected for the site. After performing onsite training with
the assistant I departed from Indonesia. Long-term inventory
control and Level-1 technical support was provided from the staff
that I originally work within and then managed from the company
Backup - never
Recover - never tested
- A local database service
provider had backup/recover scripts written for their rather old
(70’s vintage) computer. The company operated for over five
years before exploring the avenue of upgrading their ½”
tape drives to something more modern.
I visited with them and showed
them a custom controller that could be adapted to fit their exact
needs and further be compatible with their application software.
In order to perform our task we
had to understand their current backup process and dataset
volume. Within that exploration, it was discovered that their BU
tapes were blank. They had apparently operated for at least
3-years since updating the BU script and had NEVER tested the
output tapes as it was too much trouble to be their at midnight
when the script ran. A simple error (data density setting) caused
the script to error immediately upon initiation. The script did
not test for error conditions and simply skipped to the
rewind/unload phase after detecting the tape density.
The manager began mumbling
something about being fired, never being able to work again, and
a few other things before realizing that he was venting directly
in my presence.
Backup’s that are not
tested are avenues of disaster looking for the time to happen.
By not testing the recovery
process, the business liability grew from small to enormous
The tape upgrade project and the
manager were terminated. The new manager (two months later)
claimed no knowledge of this matter or the need to alter their
system or their procedures.
My role was to demonstrate the
product and assist the client in determining their needs with
regards to implementing our hardware solution. We altered the
marketing approach to better interview potential clients. No
other actions were needed.