How reliable are questions?

In a recent post I commented on our dependence on the reliability of knowledge. Now, I'll extend that inquiry to the reliability of questions themselves. You might wonder how could a question be unreliable? How could a question be false? How could a question be misleading? Good questions. That is the point.

A question is really simply an implied statement at a foundation level. The implied statement is a characterization of a quantity of information or knowledge that is desired, coupled with a request or demand or command that the requested information be provided.

So, how can a question be unreliable?

• The person asking the question may not really need the information. In that sense, the implied statement "I need X" may be a lie.
• The person may already have the information so that supplying the information may not be necessary. They may merely be seeking confirmation or maybe testing the other party.
• The question may be overly broad due to poor phrasing.
• The question may be overly narrow due to poor phrasing.
• The question may refer to knowledge that simply does not exist.
• The timing of the question may be inappropriate, either too early or too late to get a reasonable answer.
• The questioned party may not be a reasonable source for the answer.
• The question may be an imposition or unfair or disrespectful or discourteous of the questioned party.
• The tone of the question may be inappropriate.
• The represented need for an answer may not be appropriate.
• The implied statement may be offensive.
• The question may be illegal or a violation of the questioned party's rights.
• The complexity of the answer may be far out of proportion to the expectation of the questioner.
• The two parties may not agree on compatible interpretations of the terms used within the question.
• The questioner may be using private interpretations of some terms without disclosing those interpretations.
• The context may not be explicit in the question.
• The context may be incomplete or ambiguous in the question.
• The question may be ambiguous. Even simple English words can be ambiguous.
• The question may really be simply a statement in the form of a question with no intent that an answer is expected.
• The question may be rhetorical. No "answer" is expected, but the question is intended to "hang" over the interaction.
• The question may seem simple, but have underlying complexity that the questioner or the questioned party may be unaware of.
• The answer to a question may have a radically different context than the questioner was prepared for and that seemed implied by the question.
• The question might be loaded so that any answer might be misleading.
• The question might be leading so that the answer might be improperly biased.
• The question might be designed so that the legitimate answer might indirectly mislead a third party monitoring the interaction.
• The question might be worded in such a way that the answer might be misleading if viewed by itself without the full context of the question.
• The statement implied by the question might be internally inconsistent.
• The question might be intended as a distraction rather than a genuine query for information.
• The questioner may be asking the right question but the wrong person.
• The questioner may not have done sufficient due diligence to identify a source that can reasonably be considered reliable.

At heart, the issue is whether the questioned party or a computational system being questioned can reasonably be expected to respond with an acceptable answer. And even if the response is considered "acceptable", was the question itself reliable enough for the questioner to be able to depend on the answer (assuming the answer itself is also reliable.)

Context is essential. The questioned party may be able to infer all or part of the questioner's context, but assumptions about context can be somewhat risky and unreliable, possibly leading to an unreliable question despite the best of intentions on the part of the questioner.

Answering questions reliably certainly requires very careful attention to detail, but there is still plenty of craft if not outright art that needs to go into constructing reliable questions. There is an old saying in the data processing world, "GIGO - Garbage In, Garbage Out."

All too often, people have a false sense of confidence in the reliability of their questions which can lead to a false sense of confidence that the answers are valid for the questions they thought they were asking.

The only tried and true method I know of to even come close to assuring the reliability of a question is to ask multiple parties the same question and to ask multiple corollary questions so that the multiple answers can be examined to reinforce the most reliable answer. This doesn't even come close to avoiding all of the reliability factors I listed above, but at least it is a start.

-- Jack Krupansky