Testing generates information that can be used to make a decision. Testing can occur at any stage in the development of a product or system; it can test a specific attribute or overall performance; it can test a component, a subsystem, a system, or a system-of-systems.
Planning a test requires making crucial decisions: Which item to test? Which test to perform? How many tests to perform?
The test plan determines the value of the information gathered and the time and cost of testing. In general the key tradeoff is that gathering more valuable information requires more time and cost.
My students and I have developed some techniques for making test planning decisions.
Which facility to use? In some cases, a system (such as a military vehicle) needs to be used in an operational environment, but there are no existing test facilities like that environment. Instead of building a new test facility, one could use a combination of existing facilities to replicate the new operational environment. We developed an optimization model to find the test plan that used the best combination of existing facilities examples. For military vehicle applications, it specified the time and number of miles that the test vehicles should run on each existing test facility. Reference: http://www.isr.umd.edu/~jwh2/papers/IEST.pdf
Which configuration to test? A system-of-systems (SoS) consists of relatively independent systems. For example, a missile defense systems has control stations, radar locations, and rocket launchers. If the reliability of the SoS is unknown because the reliability of the systems is unknown, then testing is needed, but testing a full-scale configuration is expensive. We developed a simulation technique to predict the results of tests with smaller configurations under different scenarios and estimate the expected error of the tests. With this information, the test planning decision-makers could evaluate the tradeoffs of cost and expected error and determine the best test configuration. Reference: http://www.isr.umd.edu/~jwh2/papers/Tamburello-Herrmann-JRR-2015.pdf
Which attribute to measure (test)? In a multiattribute decision making situation, measurements of the attributes are valuable for knowing which alternative is best. If these measurements have error and the total budget for measurements is limited, then it is crucial to measure the attributes in a way that provides the most valuable information and increases the likelihood of selecting the truly best alternative. We developed and tested rules for determining which attributes should be measured how many times and showed that better rules can significantly increase this likelihood. Reference: http://www.isr.umd.edu/~jwh2/papers/Leber-Herrmann-ISERC-2015.pdf
Which test to perform? Demonstrating the reliability of a complicated system (such as a liquid rocket engine) requires testing the system and its components and subsystems. These tests and the associated hardware are expensive and require time at scarce test facilities. We developed a multi-objective test plan optimization approach to determine the best test plan that meets the demonstrated reliability. Reference: http://www.isr.umd.edu/~jwh2/papers/Strunz-Herrmann-CEAS-Space-2011.pdf
Monday, November 30, 2015
Saturday, November 21, 2015
How to React to an Earthquake
One cannot prevent an earthquake, but one can protect people and property from its impact. Traditionally, this means designing and building structures that will survive the earthquake without damage (preventive actions). Now, because of earthquake warning systems, contingency planning is becoming feasible.
Earlier this month, The Washington Post's Eric Niiler wrote about technology that can tell that an earthquake is about to happen. The November 3 article ("Last-minute warnings may make quakes less destructive") states that a combination of ground sensors and satellite-based instruments can give a warning a minute or two before the earthquake occurs.
The technology works by detecting the P-waves that precede the earthquake's shock (the sound waves are an earthquake precursor). The ShakeAlert earthquake early warning system for the west coast of the United States was developed by the United States Geological Survey (USGS) and university partners. According to Niiler, Chile, Mexico, China, Taiwan, Turkey, and Israel are developing similar systems.
Although a minute is not enough time to evacuate large buildings or a city, it is enough to shut down (or put into a safe mode) critical systems such as trains, gas lines, elevators, tunnels, and bridges. Unfortunately, it is not enough time to send text messages to millions of people simultaneously, so earthquake alerts go only to emergency agencies, utilities, and similar high-priority organizations.
For more about ShakeAlert, see the fact sheet at http://pubs.usgs.gov/fs/2014/3083/pdf/fs2014-3083.pdf
Tuesday, November 3, 2015
Small pilots, big risks
On October 24, 2015, The Washington Post reported that testing of the F-35 Joint Strike Fighter has shown that the ejection seat system poses a risk of whiplash to pilots. In particular, pilots who weigh less than 136 pounds face a "high" risk of danger. "Mid-weight" pilots face a "serious" risk.
The Post also reported that the mass of the pilot's helmet increases the risk because it is too heavy.
Until the risk can be mitigated, the F-35 program has restricted "lighter-weight" pilots from flying the plane, which will be used by the the U.S. Air Force, Navy, and Marine Corps, the Royal Air Force, and other U.S. allies. The 34th Fighter Squadron at Hill Air Force Base was the first operational Air Force unit to fly combat-coded F-35s.
Of course, in normal operation, a pilot does not eject; using the ejection seat is a contingency plan for worst-case scenarios. But the story highlights the need to consider the potential problems (risks) that can occur during a contingency plan.
It also shows different levels of risk acceptance: in 2011, the Operational Test and Evaluation office had "serious concerns" about conducting training flights with the ejection seat, but the F-35 program office accepted the risk and went ahead with training.
The Post also reported that the mass of the pilot's helmet increases the risk because it is too heavy.
Until the risk can be mitigated, the F-35 program has restricted "lighter-weight" pilots from flying the plane, which will be used by the the U.S. Air Force, Navy, and Marine Corps, the Royal Air Force, and other U.S. allies. The 34th Fighter Squadron at Hill Air Force Base was the first operational Air Force unit to fly combat-coded F-35s.
Of course, in normal operation, a pilot does not eject; using the ejection seat is a contingency plan for worst-case scenarios. But the story highlights the need to consider the potential problems (risks) that can occur during a contingency plan.
It also shows different levels of risk acceptance: in 2011, the Operational Test and Evaluation office had "serious concerns" about conducting training flights with the ejection seat, but the F-35 program office accepted the risk and went ahead with training.
Tuesday, September 29, 2015
Phones, Boats, Airlines, and Commitment
The Business section of The Washington Post on Sunday, September 27, had four interesting articles about decision making.
Brian Fung discussed the new iPhone and the different ways to pay for it, including payment plans, leases, and traditional contracts (https://www.washingtonpost.com/news/the-switch/wp/2015/09/14/thinking-about-buying-a-new-iphone-read-this-first/). He identified the best plans for four different types of consumers: bargain hunters, early adopters, owners, and traditionalists.
Barry Ritzhold's column on investing (http://www.washingtonpost.com/business/get-there/never-buy-a-boat-and-other-rash-financial-advice/2015/09/25/b788db36-60a7-11e5-8e9e-dce8a2a2a679_story.html) covered financial advice (like "never buy a boat") and then discussed the importance of recognizing four key issues in any big decision: knowledge, costs, skill, and psychology. He proposed the following decision-making approach: (1) understand what you are getting into, (2) carefully consider all of the costs of ownership, (3) only buy what you can afford, and (4) make an intelligent decision about using your limited time and money.
Chico Harlan discussed the American Airlines-US Airways merger and how the combined firm is working hard to manage the risks associated with integrating their operations (http://www.washingtonpost.com/business/the-last-days-of-us-airways/2015/09/25/f5530686-60a6-11e5-8e9e-dce8a2a2a679_story.html). A key decision was retaining the larger reservations system used by American Airlines, which reduced the number of employees who had to learn a new system.
Finally, Michelle Singletary (http://www.washingtonpost.com/business/get-there/the-pope-has-left-dc-but-his-words-remain-we-must-help-the-poor/2015/09/25/c7b423c6-60a7-11e5-8e9e-dce8a2a2a679_story.html) wondered if the Americans who were inspired by Pope Francis will decide to contribute their time and talents to help the less fortunate and discussed the relevant priciples.
Brian Fung discussed the new iPhone and the different ways to pay for it, including payment plans, leases, and traditional contracts (https://www.washingtonpost.com/news/the-switch/wp/2015/09/14/thinking-about-buying-a-new-iphone-read-this-first/). He identified the best plans for four different types of consumers: bargain hunters, early adopters, owners, and traditionalists.
Barry Ritzhold's column on investing (http://www.washingtonpost.com/business/get-there/never-buy-a-boat-and-other-rash-financial-advice/2015/09/25/b788db36-60a7-11e5-8e9e-dce8a2a2a679_story.html) covered financial advice (like "never buy a boat") and then discussed the importance of recognizing four key issues in any big decision: knowledge, costs, skill, and psychology. He proposed the following decision-making approach: (1) understand what you are getting into, (2) carefully consider all of the costs of ownership, (3) only buy what you can afford, and (4) make an intelligent decision about using your limited time and money.
Chico Harlan discussed the American Airlines-US Airways merger and how the combined firm is working hard to manage the risks associated with integrating their operations (http://www.washingtonpost.com/business/the-last-days-of-us-airways/2015/09/25/f5530686-60a6-11e5-8e9e-dce8a2a2a679_story.html). A key decision was retaining the larger reservations system used by American Airlines, which reduced the number of employees who had to learn a new system.
Finally, Michelle Singletary (http://www.washingtonpost.com/business/get-there/the-pope-has-left-dc-but-his-words-remain-we-must-help-the-poor/2015/09/25/c7b423c6-60a7-11e5-8e9e-dce8a2a2a679_story.html) wondered if the Americans who were inspired by Pope Francis will decide to contribute their time and talents to help the less fortunate and discussed the relevant priciples.
Wednesday, September 23, 2015
Traveling in Pennsylvania
Last weekend we used the Pennsylvania Turnpike for our drive to the Pittsburgh area. Because we were headed away from Philadelphia, we were surprised to see roadside emergency signs "warning" us about the visit by Pope Francis. At the Somerset service plaza, we found cards on every table reinforcing the message (see photo). The back of the card had advice about safe driving that would normally be relevant to a severe winter storm.
The Turnpike's website also had this "highway advisory" (the original was in ALL CAPS):
Although the Pope poses no threat, the traffic surrounding his visit will be disruptive, so the Pennsylvania Turnpike staff prepared a standard risk communication message that describes the potential problem, its extent (time and place), and what one can do to mitigate it.
The Turnpike's website also had this "highway advisory" (the original was in ALL CAPS):
To all travelers on the Pennsylvania turnpike Interstate 76, Interstate 276, and Interstate 476. Motorists are advised of the upcoming Pope visit in Philadelphia on September 26 and September 27, 2015. This event is expected to cause significant congestion around the Philadelphia region from the Reading interchange to the Delaware River bridge, and from the Pocono interchange to the mid-county interchange. For travelers not destined for Philadelphia during this time, find an alternate route.
Although the Pope poses no threat, the traffic surrounding his visit will be disruptive, so the Pennsylvania Turnpike staff prepared a standard risk communication message that describes the potential problem, its extent (time and place), and what one can do to mitigate it.
Wednesday, September 16, 2015
Making Laundry Detergent Safer
The Wall Street Journal reported that American manufacturers of laundry detergent packets (like Tide Pods) have agreed to a voluntary standard for the packaging of these packets.
According to the American Association of Poison Control Centers, in the United States, 7,184 children (age 5 and younger) were exposed to single-load laundry packets in the first seven months of 2015 (http://www.aapcc.org/alerts/laundry-detergent-packets/).
The WSJ article described the changes that the manufacturers will make (including tougher packaging and opaque containers).
It also quoted Nancy Cowles, the executive director of Kids in Danger, a nonprofit organization dedicated to protecting children by improving children’s product safety (http://www.kidsindanger.org/about-us/).
She raised an interesting question about risk monitoring: "We are talking absolute numbers-that's what we want to see a drop in. What's important is how many children are being injured, and not the rate of injuries relative to how much companies are selling."
What is the right measure of risk in this case: the number of children injured by laundry detergent packets, the number of injured children per household that uses laundry detergent packets, or the number of injured children per million laundry detergent packets sold?
In other areas, risks are measured per unit of activity: for instance, the European Aviation Safety Agency (EASA) 2013 Annual Safety Review reported both the number of fatal accidents per year and the rate per 10 million flights. Both numbers have been decreasing since 1994, but the rate varies dramatically around the world.
For the article, see http://www.wsj.com/articles/p-g-other-laundry-pod-makers-agree-to-new-safety-standard-1441397456?mod=rss_Business
According to the American Association of Poison Control Centers, in the United States, 7,184 children (age 5 and younger) were exposed to single-load laundry packets in the first seven months of 2015 (http://www.aapcc.org/alerts/laundry-detergent-packets/).
The WSJ article described the changes that the manufacturers will make (including tougher packaging and opaque containers).
It also quoted Nancy Cowles, the executive director of Kids in Danger, a nonprofit organization dedicated to protecting children by improving children’s product safety (http://www.kidsindanger.org/about-us/).
She raised an interesting question about risk monitoring: "We are talking absolute numbers-that's what we want to see a drop in. What's important is how many children are being injured, and not the rate of injuries relative to how much companies are selling."
What is the right measure of risk in this case: the number of children injured by laundry detergent packets, the number of injured children per household that uses laundry detergent packets, or the number of injured children per million laundry detergent packets sold?
In other areas, risks are measured per unit of activity: for instance, the European Aviation Safety Agency (EASA) 2013 Annual Safety Review reported both the number of fatal accidents per year and the rate per 10 million flights. Both numbers have been decreasing since 1994, but the rate varies dramatically around the world.
For the article, see http://www.wsj.com/articles/p-g-other-laundry-pod-makers-agree-to-new-safety-standard-1441397456?mod=rss_Business
Saturday, September 5, 2015
How to build better models with end-user modeling
Many decision-makers rely upon analysts to build models for them, and they get involved in a back-and-forth with the analyst who is supporting the decision-maker by evaluating and ranking alternatives using a mathematical model (like decision analysis or optimization). This iteration (called decision calculus by John Little) can waste a lot time.
In some situations, a better option is end-user modeling, in which the decision-maker builds the model. The approach is quantitative but not analytical, the style is quick and dirty, and the purpose is to gain insight into a decision or problem. (Tom Grossman and Stephen Powell introduced this style of modeling, which exploits the power of spreadsheets. Powell and Baker's textbook describes spreadsheet modeling in detail.)
The end-user modeling process, which accelerates the process of using models to understand a situation, has three steps: (1) plan the model (with the computer off); (2) program the model, and (3) craft the user interface.
For the planning stage, start by identifying the key relationships between the inputs and outputs, sketching the layout of a worksheet to calculate the outputs, and drawing the key charts and graphs that will provide the needed insight.
When programming, build the spreadsheet one section at a time, checking the each section works correctly, and using good spreadsheet programming techniques.
For the user interface, use color and formatting and comments to make it clear how to use the model (one can quickly forget after setting it down for awhile). Clearly identifying the inputs, parameters, calculations, and outputs is very helpful.
As Grossman stated, end-user modeling gives one "the ability to roughly compute the effects of a proposed change ('what-if' modeling)" and "the ability to obtain quick, rough insight on actions that are likely to improve the business."
For further reading:
Grossman, Tom, "End-User Modeling," OR/MS Today, October 1997. Link: http://lionhrtpub.com/orms/orms-10-97/IiE.html
Little, John D.C., “Models and managers: the concept of a decision calculus,” Management Science, Volume 16, Number 8, pages B-466-485, 1970. Link: http://pubsonline.informs.org/doi/abs/10.1287/mnsc.1040.0267
Powell, Stephen G. "The teachers' forum: From intelligent consumer to active modeler, two MBA success stories." Interfaces 27, no. 3 (1997): 88-98. Link: http://pubsonline.informs.org/doi/abs/10.1287/inte.27.3.88
Powell, Stephen G., and Kenneth R. Baker, The Art of Modeling with Spreadsheets, John Wiley & Sons, Inc., Hoboken, New Jersey, 2004. Link: http://www.wiley.com/WileyCDA/WileyTitle/productCd-EHEP002883.html
In some situations, a better option is end-user modeling, in which the decision-maker builds the model. The approach is quantitative but not analytical, the style is quick and dirty, and the purpose is to gain insight into a decision or problem. (Tom Grossman and Stephen Powell introduced this style of modeling, which exploits the power of spreadsheets. Powell and Baker's textbook describes spreadsheet modeling in detail.)
The end-user modeling process, which accelerates the process of using models to understand a situation, has three steps: (1) plan the model (with the computer off); (2) program the model, and (3) craft the user interface.
For the planning stage, start by identifying the key relationships between the inputs and outputs, sketching the layout of a worksheet to calculate the outputs, and drawing the key charts and graphs that will provide the needed insight.
When programming, build the spreadsheet one section at a time, checking the each section works correctly, and using good spreadsheet programming techniques.
For the user interface, use color and formatting and comments to make it clear how to use the model (one can quickly forget after setting it down for awhile). Clearly identifying the inputs, parameters, calculations, and outputs is very helpful.
As Grossman stated, end-user modeling gives one "the ability to roughly compute the effects of a proposed change ('what-if' modeling)" and "the ability to obtain quick, rough insight on actions that are likely to improve the business."
For further reading:
Grossman, Tom, "End-User Modeling," OR/MS Today, October 1997. Link: http://lionhrtpub.com/orms/orms-10-97/IiE.html
Little, John D.C., “Models and managers: the concept of a decision calculus,” Management Science, Volume 16, Number 8, pages B-466-485, 1970. Link: http://pubsonline.informs.org/doi/abs/10.1287/mnsc.1040.0267
Powell, Stephen G. "The teachers' forum: From intelligent consumer to active modeler, two MBA success stories." Interfaces 27, no. 3 (1997): 88-98. Link: http://pubsonline.informs.org/doi/abs/10.1287/inte.27.3.88
Powell, Stephen G., and Kenneth R. Baker, The Art of Modeling with Spreadsheets, John Wiley & Sons, Inc., Hoboken, New Jersey, 2004. Link: http://www.wiley.com/WileyCDA/WileyTitle/productCd-EHEP002883.html
Subscribe to:
Posts (Atom)