Frequencies and human rationality

This post was inspired by reading a number of articles and books about whether or not humans can make rational decisions.  Different definitions of rationality lead to different answers.  Although a popular view is that certain "cognitive illusions" show that humans can be irrational, Gerd Gigerenzer has argued over the years that humans are indeed rational and that human cognition is more adapted to certain representations of information than others.  He and his collaborators have shown that changing the format of the information changes how well experimental subjects do on judgment tasks (which are used to test human rationality).  Herbert Simon made a similar argument about how the representation of a problem can affect how one solves it.

Gigerenzer identified two types of information that are called "probability": subjective probabilities (about single events) and frequencies (within a set).  He repeated judgment experiments by asking for frequency judgments instead of probabilities about single events; in his experiments, fewer subjects committed the errors that were found in the original judgment experiments.  Humans seem irrational if one considers only a certain algorithm or norm as describing true rationality, but the environment and the representation of the information are also relevant when assessing someone's judgment.

To explain human judgment, he proposed the theory of probabilistic mental models (PMM), a model of bounded rationality.  In this theory, a person uses different cues to help make judgments.  In situations with time pressure, a person may use the first useful cue, a heuristic that corresponds to bounded rationality.

Heuristics such as Take the Best and recognition are "short cuts that can produce efficient decisions" and help humans adapt to a complex, dynamic environment.   In 1991 Gigerenzer noted that the heuristics of similarity, availability, and anchoring and adjustment "are largely undefined concepts" and are merely well-known principles of the mind.  Gigerenzer has proposed ecological rationality to explain human decision making.  "Ecological rationality can refer to the adaptation of mental processes to the representation of information ... It can also refer to the adaptation of mental processes to the structure of information in an environment."

Gigerenzer cited Herbert Simon to argue that our cognitive abilites are limited and that our minds "should be understood relative to the environment in which they evolved, rather than to the tenets of classical rationality."  Because our environment included natural frequencies, not single-event probabilities, fewer subjects make errors in experiments with frequencies.  Humans acquire and update natural frequencies through experience.  Moreover, humans can recognize that the environment has changed and will ignore data about the past when it changes.

Recently, Gigerenzer has argued that the study of behavioral economics has a "bias bias"; that is, economists are looking for evidence of systematic errors and bias while ignoring psychological research that contradicts the view that humans are systematically irrational.

Related work

1. Gigerenzer, Gerd, "How to make cognitive illusions disappear: Beyond heuristics and biases," European Review of Social Psychology, Volume 2, Number 1, pages 83-115, 1991.
2. Gigerenzer, Gerd, "The bounded rationality of probabilistic mental models," in K.I. Manktelow and D.E. Over, editors, Psychology and Philosophical Perspectives, Routledge, London, 1993.
   
3. Gigerenzer, Gerd, "Ecological intelligence: An adaptation for frequencies," In D.D. Cummins and C. Allen, editors, The Evolution of Mind, pp. 9-29, Oxford University Press, Oxford, 1998.
4. Gigerenzer, Gerd, Adaptive Thinking, Oxford University Press, Oxford, 2000.
5. Gigerenzer, Gerd, "The bias bias in behavioral economics," Review of Behavioral Economics, Volume 5, Number 3-4, pages 303-336, 2018.

Reducing Noise and Improving Decision Making

 

Cover page image: https://www.littlebrownspark.com/

Noise: A Flaw in Human Judgment. By Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein

Kahneman, Sibony, and Sunstein have written a book that is both valuable and frustrating.

This book presents multiple ideas related to human judgment and decision making: (1) a review of studies that have described the variability in judgments in many domains, (2) approaches for reducing that variability, (3) an approach for making decisions when there are multiple factors that should be considered, and (4) an appeal for better procedures in the legal system.  According to the authors, the book offers an understanding of the psychological foundations of disparities in judgments, which are here classified as noise and bias.

The book’s strengths include its review of the literature on the variability of judgments and its distinction between noise and bias.  It presents examples of judgments from many domains (including medicine, business, and the legal system).  It strongly supports systematic decision-making processes (a topic that is important to me) and emphasizes the importance of accurate judgments.  It acknowledges the difficulties of reducing noise.  Finally, its notes provide references to original studies that provide context and details to the book’s discussion.

The book describes a range of best practices for judgment and decision making: employing persons who are better at making judgments, aggregating multiple judgments, using judgment guidelines, using a shared scale grounded in an outside view, and structuring complex decisions.  The first three items are meant to reduce judgment errors due to noise and bias.  The last item is a practical multi-criteria decision-making process that (a) decomposes the decision into a set of assessments, (b) collects information about each assessment independently, and (c) presents this evidence to the decision-maker(s), who may use intuition to synthesize this information and select an alternative.  In an appendix, the authors use their recommendations, for the book provides a checklist (a guideline) for evaluating a decision-making process.

When discussing ratings, the book wisely recommends that “performance rating scales must be anchored on descriptors that are sufficiently specific to be interpreted consistently.”  Scales with undefined terms such as “poor” and “good” and “excellent” should be discarded unless they are well-understood in the group of persons who are using them as a common language.

Unfortunately, two weaknesses (one minor, one major) frustrated me.  The first is the fact that the text contains no superscripts, citations, or other marks that indicate the notes that are available in the Notes section at the end of the book.  In the Notes section, each note has only a page number and a brief quote that suggest the text to which the note applies.  This unreasonable scheme reduces the value of the notes' many citations and explanations by making them harder to find and use.

The second weakness is more significant.  The book does not distinguish between judgment and decision making.  It tends to treat them as the same thing.  Indeed, a note explains that the authors “regard decisions as a special case of judgment” (page 403). 

For example, the book discusses the judgments that an insurance company’s employees make.  One example is a claims adjuster’s estimate of the cost of a future claim, which is indeed a judgment.  The other example is the premium that an underwriter quotes, which is a decision, not a mere judgment.  It is based on numerous judgments, of course, but the underwriter chooses the premium amount.  The book states that making a judgment is similar to measuring something, which is appropriate, but then goes on to say that the premium in the underwriter’s quote is also a judgment, which is not appropriate, because it is the result of a decision, not a measurement.

Elsewhere, the book claims that “an evaluative judgment determines the choice of an acceptable safety margin” for an elevator design (page 67).  This is inappropriate, however, for choosing the safety margin is a decision, not a measurement.

The book states that the process of judgment involves considering the given information, engaging in computation, consulting one’s intuition, and generating a judgment; that is, judgment is “an operation that assigns a value on a scale to a subjective impression (or to an aspect of an impression)” (page 176).  This is not the same as decision making.  Decision making is a more comprehensive process that defines relevant objectives, identifies (or develops) alternatives, evaluates the alternatives, selects one, and implements it.  In this process, judgment is an activity that may be used to evaluate the alternatives.  The book provides a relevant example that shows the distinction: job candidates get ratings, but only one gets hired.  The ratings are judgments, but choosing and hiring someone is a decision.

Those seeking to improve decision making in their organizations will find many useful suggestions in this book, but they should keep in mind that decision making is a process, not a judgment.

 

Decision making in the Ideation Toolkit

The Ideation Toolkit (from Keen Engineering Unleashed) is a collection of information that engineering students can use as they develop an idea for a product.

Although this toolkit includes the Analytic Hierarchy Process (AHP), I suggest that educators use multi-attribute utility theory (MAUT) instead of AHP.  I have been teaching engineering decision making for many years, and, although my textbook includes both AHP and MAUT, I have found that engineering students find MAUT easier to adopt and use correctly.  Using AHP in a rational way is more difficult than it looks, whereas MAUT is more straightforward for making decisions when the alternatives have multiple criteria that need to be considered (e.g., cost, strength, durability, etc.).

Improving Decision-Making Processes

The WomenCorporateDirectors Foundation’s Thought Leadership Commission and the KPMG Board Leadership Center issued a report that describes some of the problems that can cause poor decision making and recommends improving decision-making processes.  The report discusses incomplete information, groupthink, overconfidence, and other poor practices as causes of decision-making failures. 

The report presents five decision-making styles (first presented in an article by Dan Lovallo and Olivier Sibony):

• Visionary,
• Guardian,
• Motivator,
• Flexible, and
• Catalyst.

Each style has its strengths and weaknesses.  I would add that a decision-maker needs to use the right decision-making style for the decision-context that is present (see more about this at this blog post). 

The report also discusses ways to create a more inquisitive, risk-based decision-making process by considering multiple viewpoints, identifying the pros and cons of every alternative, and discussing the associated risks (what could go wrong).  The resulting process will resemble the "discovery decision-making process" described by Paul Nutt.

Finally, the report recommends evaluating the decision-making process, not its outcomes, as a way to identify opportunities for improvement.

HT: ISE Magazine

Fixing the Roof

Strong winds earlier this week in Maryland led us into two related but very different decisions.

I came home Monday afternoon to discover shingles lying next to the front door and hanging in the trees in the front yard of our house.  That was unusual, and I was further surprised to learn that they came from our roof!  The wind had lifted dozens from the roof on the back half of our house and blown them over the front. 

After viewing the damage from the ground and reading that the weather forecast included more high winds and thunderstorms this week, we hurried to find and hire someone to repair the roof quickly. 
Due to the extreme time pressure, this chaotic context (Snowden and Boone, 2007) forced us to find something that works and reestablish order (an adequate roof).  We quickly searched online resources, asked friends for references, and even opened the yellow pages (we still had one from 2010) to get a list of roofers, and we called around until we had a few lined up to call back or inspect it the next morning (that took seven phone calls).  Some of those who called back said that they didn't do repairs, so they were out.  The first roofer who actually showed up, looked at the roof, and gave us a price got the job.  It was a pure satisficing strategy - we picked the first adequate alternative.   (It was quick: one roofer called later that morning and was surprised that we had already selected someone.)  We had a fixed roof later that day, and we could sleep better that night.

That was the first decision.  We now face a second decision: to select someone to do a roof replacement (the current roof is over 20 years old, and the shingle incident this week is a precursor to more significant problems in the future).  This decision has a complicated context in which we have to get more information about the state of the roof from experts, reconcile their opinions, investigate the firms, make tradeoffs, and finally pick one. 

One roof, two decisions. Both have the same problem (pick the best roofer), but the contexts, relevant attributes, and decision-making processes are very different.

Reference cited: Snowden, David J., and Mary E. Boone, “A Leader's Framework for Decision Making,” Harvard Business Review, Vol. 85, Issue 11, pages 69-76, November 2007.

How to build better models with end-user modeling

Many decision-makers rely upon analysts to build models for them, and they get involved in a back-and-forth with the analyst who is supporting the decision-maker by evaluating and ranking alternatives using a mathematical model (like decision analysis or optimization). This iteration (called decision calculus by John Little) can waste a lot time.

In some situations, a better option is end-user modeling, in which the decision-maker builds the model.  The approach is quantitative but not analytical, the style is quick and dirty, and the purpose is to gain insight into a decision or problem.  (Tom Grossman and Stephen Powell introduced this style of modeling, which exploits the power of spreadsheets.  Powell and Baker's textbook describes spreadsheet modeling in detail.)

The end-user modeling process, which accelerates the process of using models to understand a situation, has three steps: (1) plan the model (with the computer off); (2) program the model, and (3) craft the user interface.


For the planning stage, start by identifying the key relationships between the inputs and outputs, sketching the layout of a worksheet to calculate the outputs, and drawing the key charts and graphs that will provide the needed insight.

When programming, build the spreadsheet one section at a time, checking the each section works correctly, and using good spreadsheet programming techniques.

For the user interface, use color and formatting and comments to make it clear how to use the model (one can quickly forget after setting it down for awhile).  Clearly identifying the inputs, parameters, calculations, and outputs is very helpful.

As Grossman stated, end-user modeling gives one "the ability to roughly compute the effects of a proposed change ('what-if' modeling)" and "the ability to obtain quick, rough insight on actions that are likely to improve the business."

For further reading:

Grossman, Tom, "End-User Modeling," OR/MS Today, October 1997. Link: http://lionhrtpub.com/orms/orms-10-97/IiE.html

Little, John D.C., “Models and managers: the concept of a decision calculus,” Management Science, Volume 16, Number 8, pages B-466-485, 1970.  Link: http://pubsonline.informs.org/doi/abs/10.1287/mnsc.1040.0267

Powell, Stephen G. "The teachers' forum: From intelligent consumer to active modeler, two MBA success stories." Interfaces 27, no. 3 (1997): 88-98.  Link: http://pubsonline.informs.org/doi/abs/10.1287/inte.27.3.88

Powell, Stephen G., and Kenneth R. Baker, The Art of Modeling with Spreadsheets, John Wiley & Sons, Inc., Hoboken, New Jersey, 2004.  Link: http://www.wiley.com/WileyCDA/WileyTitle/productCd-EHEP002883.html

Deciding to Save New Orleans

Ten years after Hurricane Katrina, the city of New Orleans has done much to mitigate the risk of a hurricane.  The August 22 issue of The Washington Post included an article by Chris Mooney (http://www.washingtonpost.com/sf/national/2015/08/21/the-next-big-one/) about an important decision that remains to be made: whether to use sediment diversion to protect the wetlands that protect New Orleans.  In addition to slowing the loss of wetlands, the advantages include a relatively low one-time cost and a potential economic value from sportsmen and tourists who would enjoy the wetlands.  The key disadvantage is the disruption to the local fishing industry.  A key uncertainties are whether the diversions will actually work because there are many factors that influence wetland restoration and the impact of the wetlands on the storm surge may be limited.

The decision-making process appears to be an analytic-deliberative one: a state advisory board has scientific experts, while fishermen have organized a group to oppose the diversions and (if necessary) block construction, and a state agency needs to make a decision before the end of the year.

Option Awareness

MITRE organized a Decision Making in Complex Systems Technical Exchange Meeting at their McLean, Virginia, site this week, and the meeting included valuable presentations on modeling complex systems, visualizing their performance, and supporting decision making.  (Full disclosure: I was one of the speakers.)

At the meeting, Jill Drury and Gary L. Klein discussed their research on option awareness (OA), which is "the perception and comprehension of the relative desirability of available options, as well the underlying factors and trade-offs that explain that desirability."
Their experimental research has shown that OA decision support tools that present the distribution of performance for each option can help decision-makers select the most robust alternatives, understand the factors that affect their performance, and generate new options.
Their collaborators include Mark Pfaff and others at Indiana University.
For more details and examples of the visualization, see their paper in the Journal of Cognitive Engineering and Decision Making, which can be found at http://www.iupui.edu/~grappa/publications/Supporting_Complex_Decision_Making_Through_OA_Pfaff_et_al_2012_JCEDM.pdf

MITRE also recently hosted the 12th International Naturalistic Decision Making Conference.
The conference website is http://www2.mitre.org/public/ndm/index.html.

Improving design decision making

Here at the ISERC in Nashville this week, I picked up the January 2014 issue of
IIE Transactions on Occupational Ergonomics and Human Factors (http://www.tandfonline.com/toc/uehf20/current) and found the article "Adapting Engineering Design Tools to Include Human Factors" by Judy Village et al.

The article describes how researchers at Ryerson University (in Toronto) worked with human factors specialists and engineers at a large electronics manufacturer to change how that firm designs the assembly process for its new products.  The changes led to design tools that help the firm's engineers consider human factors issues during the design process to make assembly easier, safer, and faster.  That is, the changes modified the objectives and constraints used to make assembly process design decisions.  For example, the design must satisfy a human factors target by scoring well on 22 items related to human factors.

In addition, the process used to develop these new design tools is interesting.  According to the article, "an action research approach was used, where researchers were embedded in the organization and together took action" to plan, implement, and improve the tools.  This process emphasized understanding the design process and its metrics, tools, and language and then identifying opportunities to improve the design tools with feasible, desirable changes (that is, the changes had to "fit the design process" and "provide important metrics for business performance").

Although the new design tools may be specific to this firm, the process used can be applied elsewhere.  The authors state that their contribution includes "the lessons learned about the process of adapting internal engineering tools"; that is, they have showed how an organization can improve design decision making.

Decision Making and the Panama Canal (Part II: the Americans)

(For Part I, see http://engineeringdecisionmaking.blogspot.com/2015/05/decision-making-and-panama-canal-part-i.html)

After the United States gained control of the Panama Canal effort in 1904, the type of canal was not yet specified.   The two alternatives were a sea-level canal and a lock canal.

First, President Theodore Roosevelt appointed thirteen civil engineers to the International Board of Consulting Engineers and told them that the two most important attributes were the speed of construction and the likelihood of successful completion. After conducting their research, eight members voted for a sea-level canal, and five voted for a lock canal.  The chairman of Isthmian Canal Commission (ICC), which ran the Canal Zone, and the chief engineer, recommended a lock canal and gave sound technical and financial reasons for their disinterested choice.  Eventually, the Senate approved a lock canal. The House of Representatives then concurred, and the president confirmed the choice on June 29, 1906.

To coordinate their experienced personnel, appropriate equipment, and the ingenious system for moving material, the Americans had a centralized organization with authority over every aspect of and employee in the Canal Zone. There were no contractors working in Panama.  The headquarters made a detailed plan every day for coordinating the drilling, blasting, excavating, and dumping operations to maximize productivity.  Moreover, the motivated employees felt that they were part of a community, which improved morale and productivity.

The canal was officially opened on August 15, 1914. The American decision-making process was a more effective analytic-deliberative process that was focused on clear objectives (not personal ambition), and their decision-making system in Panama was more appropriate than the French scheme. 

Decision Making and the Panama Canal (Part I: the French)

The Panama Canal, which opened in 1914, is one of the most outstanding engineering successes in history.  The history of the Panama Canal has many stories of heroic explorers, brilliant engineers, and tireless laborers.  It also includes critical choices by rational, intelligent decision-makers.  Unfortunately, the French decision-making processes and decision-making system contributed to their failure.

An early critical choice was whether to build a sea-level canal or a lock canal.  Ferdinand de Lesseps was a French diplomat who led the effort to build the Suez Canal and became part of a group interested in building a canal across Panama.  In 1879 he organized a meeting in Paris to evaluate the options for a canal across Central America.  De Lesseps was determined that the sea-level route across Panama should be approved, however, and he personally convinced many French delegates to support that alternative, which helped him and his business associates raise money to build the canal.

Approximately 100 small subcontractors worked to dig the canal, but, without central coordination, they impeded each other’s efforts to remove the excavated dirt and rock.  Moreover, the contractors chose the simplest (cheapest) way to dump the excavated dirt and rock, and these operations were often stopped by storms, which slowed the excavation of the canal.  They were not loyal to the canal company or motivated by its goals, and there was no central office to coordinate their activities.

The French effort was bankrupt within 10 years.

For more about the canal's history, check out The Canal Builders by Julie Greene and The Path Between the Seas by David McCullough.

Next time: the Americans.