Conducting Web-based Research

The CARS Checklist

The CARS Checklist (Credibility, Accuracy, Reasonableness, Support) is designed for ease of use. Few sources will meet every criterion in the list, and even those that do may not possess the highest level of quality possible. But if you learn to use the criteria in this list, you will be much better able to separate the high-quality information from the poor-quality information.

Read each section below or jump right to the summary.

Credibility

Because people have always made important decisions based on information, evidence of authenticity and reliability or credibility, believability has always been important. If you read an article saying that the area where you live will experience a major earthquake in the next six months, it is important that you should know whether or not to believe the information. Some questions to ask about general credibility might include these:

• Is there sufficient evidence presented to make the argument persuasive?
• Are there compelling arguments and reasons given?
• Are there enough details for a reasonable conclusion about the information?

There are several tests you can apply to a source to help you judge how credible and useful it will be:

Author’s Credentials. The author or source of the information should show some evidence of being knowledgeable, reliable, and truthful.

Some questions you might ask would include the following:

• What about this source makes it believable (or not)?
• How does this source know this information?
• Why should I believe this source over another?

As you can see, the key to credibility is the question of trust. Here are some clues to credibility:

• Author’s education, training, and/or experience in a field relevant to the information. Look for biographical information, the author’s title, or position of employment
• Author’s contact information (e-mail or postal mail address, telephone number)
• Organizational authorship from a known and respected organization (corporate, governmental, or non-profit)
• Organizational authorship reflecting an appropriate area of expertise
• Author’s reputation or standing among peers.

Evidence of Quality Control

Most scholarly journal articles pass through a peer review process, whereby several readers must examine and approve content before it is published. Statements issued in the name of an organization have almost always been seen and approved by several people. (But note the difference between, “Allan Thornton, employee of the National Oceanographic and Atmospheric Agency, says that a new ice age is near,” and “The National Oceanographic and Atmospheric Agency said today that a new ice age is near.” The employee is speaking for himself, where-as a statement in the name of NOAA repre-sents the official position of NOAA.)

Evaluation Tip

Feel free to contact the author of a source by e-mail or telephone when such contact information is given. Many authors are willing to clarify points or explain details further. Communicating with the author will also give you more evidence about credibility.

Evidence of quality control of Web material includes these items:

• Information presented on corporate, government, or organizational Web sites
• Online journals that use refereeing (peer review) by editors or others
• Postings of information taken from books or journals that have a quality control process Indicators of Lack of Credibility. You can sometimes tell by the tone, style, or com-petence of the writing whether or not the information is suspect.

Here are a few clues to lack of credibility:

• Anonymity
• No indication of a third party editor or publisher to ensure the quality and reliability of the information
• Negative metainformation. If all the reviews are critical, be careful. (See the discussion about metainformation later in this chapter.)
• Bad grammar or misspelled words. Most educated people use grammar fairly well and check their work for spelling errors. An occasional split infinitive or comma in the wrong place is not unusual, but more than two or three spelling or grammar errors is cause for caution, at least. Whether the errors come from carelessness or ignorance, neither puts the information or the writer in a favorable light.

Evaluation Tip

The top-level domain .gov is used by state and federal government agencies and indicates that the information has the sanction of that government agency. Information from these domains is usually highly reliable. For example, you can obtain reliable census information from the Census Bureau at www.census.gov or information about medicine, food, or cosmetics from the Food and Drug Administration at www.fda.gov

Accuracy

The goal of the accuracy test is to ensure that the information is actually correct: up-to-date, factual, detailed, exact, and comprehensive. For example, even though a very credible writer said something that was correct twenty years ago, it may not be correct today. Similarly, a reputable source might be giving up-to-date information, but the information may be only partial and not give the full story. Here are some concepts related to accuracy:

Timeliness

Some work is timeless, like the classic novels and stories, or like the thought-provoking philosophical work of Aristotle and Plato. Other work has a limited useful life because of advances in the discipline (psychological theory, for example), and some work is outdated very quickly (like technology news). You must therefore be careful to note when the information you find was created, and then decide whether it is still of value (and how much value). You may need information within the past ten years, five years, or even two weeks. But old is not necessarily bad: nineteenth-century American history books or literary anthologies can be highly educational because they can function as comparisons with what is being written or anthologized now. In many cases, though, you want accurate, up-to-date information.

An important idea connected with timeliness is the dynamic, fluid nature of informa-tion and the fact that constant change means constant changes in timeliness. The facts we learn today may be timely now, but tomorrow will not be. Especially in technology, science, medicine, business, and other fields always in flux, we must remember to check and re-check our data from time to time, and realize that we will always need to update our facts. Comprehensiveness. Any source that presents conclusions or that claims (explicitly or implicitly) to give a full and rounded story, should reflect the intentions of completeness and accuracy. In other words, the information should be comprehensive. Some writers argue that researchers should be sure that they have “complete” information before making a decision or coming to a conclusion. But with the advent of the information age, such a goal is impossible, if by “complete” we mean all possible information. No one can read 20,000 articles on the same subject before coming to a conclusion or making a decision. On the other hand, an information source that deliberately leaves out important facts, qualifications, consequences, or alternatives may be misleading or even intentional-ly deceptive. And since no single piece of information will offer the truly complete story, even if accuracy and fairness are intended, we must rely on more than one source to provide us with a fuller view of the situation.

Evaluation Tip

You can use your browser to find out when a Web page was last modified, even though there may not be a visible date on the page itself. In Netscape, use “View,” “Page Info” and you will see a “Last Modified” field with a date. In Internet Explorer, using “File,” ”Properties” you will get the date the information was transferred to your disc, not the date the page was modified.

Audience and Purpose

For whom is this source intended and for what purpose? If, for example, you find an article, “How Plants Grow,” and children are the intended audience, then the material may be too simplified for your college botany paper. More impor-tant to the evaluation of information is the purpose for which the information was created. For example, an article titled, “Should You Buy or Lease a Car?” might have been written with the purpose of being an objective analysis, but it may instead have been written with the intention of persuading you that leasing a car is better than buying. In such a case, the information will most likely be highly biased or distorted. Such information is not useless, but the bias must be taken into consideration when interpreting and using the informa-tion. (In some cases, you may be able to find the truth by using only biased sources, some biased in one direction and some biased in the other.) Be sure, then, that the intended audience and purpose of the article are appropriate to your requirements or at least clearly in evi-dence so that you may take them into account. Information pretending to objectivity but possessing a hidden agenda of persuasion or a hidden bias is quite common in our culture.

Indicators of a Lack of Accuracy

In addition to an obvious tone or style that reveals a carelessness with detail or accuracy, there are several indicators that may mean the source is inaccurate, either in whole or in part:

• No date on the document
• Assertions that are vague or otherwise lacking detail
• Sweeping rather than qualified language (that is, the use of always, never, every, completely rather than usually, seldom, some-times, tends, and so forth)
• An old date on information known to change rapidly
• Very one-sided view that does not acknowledge opposing views or respond to them

Reasonableness

The test of reasonableness involves examining the information for fairness, objectivity, moderateness, and consistency. Fairness. Fairness includes offering a balanced, reasoned argument. ment, not selected or slanted. Even ideas or claims made by the source’s opponents should be presented in an accurate manner. Pretending that the opponent has wild, irra-tional ideas or arguments no one could accept is to commit the straw man fallacy. A good information source will also possess a calm, reasoned tone, arguing or presenting material thoughtfully and without attempting to get you emotionally worked up. Pay attention to the tone and be cautious of highly emotional writing. Angry, hateful, critical, spiteful tones often betray an irrational and unfair attack under way rather than a reasoned argument. And writing that attempts to inflame your feelings to prevent you from thinking clearly is also unfair and manipulative.

Objectivity

There is no such thing as pure objectivity, but a good writer should be able to control his or her biases. Be aware that some organizations are naturally not neutral. For example, a professional antibusiness group will find, say, that some company or industry is overcharging for widgets. The industry trade association, on the other hand, can be expected to find that no such over-charging is taking place. Be on the lookout for slanted, biased, politically distorted work. One of the biggest hindrances to objectivity is conflict of interest. Sometimes an infor-mation source will benefit in some way (usually financially, but sometimes politically or even emotionally or psychologically) if that source can get you to accept certain informa-tion rather than the pure and objective truth. For example, many sites that sell “natural” products (cosmetics, vitamins, clothes, food) often criticize their competitors for selling bad, unhealthy, or dangerous products. The criticism may be just, but because the messenger will gain financially if you believe the message, you should be very careful—and check somewhere else before spending money or believing the tale.

Moderateness

Moderateness is a test of the information against how the world really is. Use your knowledge and experience to ask if the information is really likely, possible, or probable. Most truths are ordinary. If a claim being made is surprising or hard to believe, use caution and demand more evidence than you might require for a lesser claim. Claims that seem to run against established natural laws also require more evidence. In other words, do a reality check. Is the information believable? Does it make sense? Or do the claims lack face validity? That is, do they seem to conflict with what you already know in your experience, or do they seem too exaggerated to be true? For example, does the statement, “Half of all Americans have had their cars stolen,” pass the face validity test? Have half of your friends had their cars stolen? Is the subject on the news regularly (as we might assume it would be if such a level of theft were the case)?

It is important, of course, to remember that some truths are spectacular and immoderate. If you have read about, say, the Tulipmania in Holland in 1636–1637, where some single tulip bulbs sold for the equivalent of tens of thousands of dollars each, the idea of immoderate truth will not be so strange to you. Do not, therefore, automatically reject a claim or source simply because it is astonishing. Just be extra careful about checking it out.

Evaluation Tip

Many Web sites are sponsored by news organizations that have printed or broadcast counterparts. As a general rule, if the print-ed or broadcast source is reliable, so too is the online counterpart. The online sites for Time, CNN, The New York Times, ABC News, and so forth, have the same reputation to uphold and the same resources to devote to fact checking and accuracy as their traditional media ver-sions, and should therefore be just as professional. There have been a few notable exceptions, but those have indeed been exceptions. (Similarly, the online versions of the incredible sources, such as some of the tabloids, should be viewed with the same suspicion used for the printed version.)

Consistency

The consistency test simply requires that the argument or information does not contradict itself. Sometimes when people spin falsehoods or distort the truth, inconsistencies or even contradictions show up. These are evidence of unreasonableness. Perhaps an obvious example comes from some of the chain letters that circulate on the Internet and by fax machine: they affirm that they have come to you “to bring you cheer and good luck,” only to threaten you (not very indirectly) with unemployment or death if you do not continue to circulate the letter.

World View

A writer’s view of the world (political, economic, religious—including antireligious—and philosophical) often influences his or her writing profoundly, from the subjects chosen to the slant, the issues raised, issues ignored, fairness to opponents, kinds of examples, and so forth. Some writers’ value systems permit them to fabricate evidence, lie, or falsify the positions of others for the sake of what they think is a noble cause—or perhaps political expediency. For these writers, political agendas take precedence over truth. Knowing about such distorting world views can therefore provide another evaluative test for reasonableness.

Indicators of a Lack of Reasonableness

Writers who put themselves in the way of the argument, either emotionally or because of self-interest, often reveal their lack of reasonableness. If, for example, you find a writer reviewing a book he opposes by asserting that “the entire book is completely worthless claptrap,” you might suspect there is more than a reasoned disagreement at work.

Here are some clues to a lack of reasonableness:

• Intemperate tone or language (“stupid jerks,” “shrill cries of my extremist opponents”)
• Overclaims (“Thousands of children are murdered every day in the United States.”)
• Sweeping statements of excessive significance (“This is the most important idea ever conceived!”)
• Conflict of interest (“Welcome to the Old Stogie Tobacco Company Home Page. To read our report, ‘Cigarettes Make You Live Longer,’ click here.” or “When you buy a stereo, beware of other brands that lack our patented circuitry.”)

Support

The area of support is concerned with the source and corroboration of the information. Much information, especially statistics and claims of fact, comes from other sources. Citing sources strengthens the credibility of the information. (Remember this when you write a research paper.)

Source Documentation or Bibliography. When facts or statis-tics are quoted, look to see whether their source is revealed, so that you could check their accuracy.

Some source considerations include these:

• Where did this information come from?
• What sources did the information creator use?
• Are the sources listed?
• Is there a bibliography or other documentation?
• Does the author provide contact information in case you wish to discuss an issue or request further clarification?
• What kind of support for the information is given?
• How does the writer know this?

It is especially important for statistics to be documented. Otherwise, someone may be just making up numbers. Note that some information from corporate sites consists of descriptions of products, techniques, technologies, or processes with which the corporation is involved. If you are careful to distinguish between facts (“We mix X and Y together to get Z”) and advertising (“This protocol is the best in the industry”), then such descriptions should be reliable.

Corroboration

See if other sources support this source. Corroboration or confirmability is an important test of truth. And even in areas of judgment or opinion, if an argument is sound, there will probably be a number of people who adhere to it or who are in some general agreement with parts of it. Whether you’re looking for a fact (like the lyrics to a song or the date of an event), an opinion (such as whether paper or plastic is the more environmentally friend-ly choice), or some advice (such as how to grow bromeliads), it is a good idea to triangulate your findings: that is, find at least three sources that agree. If the sources do not agree, do further research to find out the range of opinion or disagreement before you draw your conclusions.

What you are doing with corroboration, then, is using information to test information. Use one source, fact, point of view, or inter-pretation to test another. Find other informa-tion to support and reconfirm (or to challenge or rebut) information you have found. It is critical, of course, to consider the quality of the corroborative information. There is at least one situation on the Web now where more than a dozen “natural care” sites have all repeated the same false story that the surfactant sodium laureth sulfate (used in many cosmetics) causes cancer. (The owners of these sites sell products that do not contain this ingredient.) Finding the same claim on several of these sites is certainly not corroboration. For a corroborative test, you should look to disinterested third parties, in this case to the federal government, toxicology reports, and the like.

Corroboration is especially important when you find dramatic or surprising information (information failing the moderateness test, above). For example, a claim like the one just mentioned, that some commonly used substance is harmful, should be viewed with skepticism until it can be confirmed (or rebutted) by further research and by truly reliable sources. The claim may be true, but it seems unlikely that both government and consumer organizations would let a harmful chemical or other product go unchallenged.

Evaluation Tip

A quirk of human nature causes most of us to value what is scarce. Some information producers who want to manipulate us therefore pretend to have “secret” or “exclusive” information—which, of course, cannot therefore be corroborated. Be very careful of such informa-tion. Remember that, when it comes to informa-tion, “secret” is nearly a synonym for “gossip,” and “exclusive” means roughly “unsubstantiated” or “unconfirmed.”

External Consistency

While the test of corroboration involves finding out whether other sources contain the same new informa-tion as the source being evaluated, the test of external consistency compares what is familiar in the new source with what is familiar in other sources. That is, information is usually a mixture of old and new, of some things you already know and some things you do not. The test of external consistency asks, Where this source discusses facts or ideas I already know something about, does the source agree or harmonize or does it conflict, exaggerate, or distort? The reasoning is that if a source is faulty where it discusses something you already know, it is likely to be faulty in areas where you do not yet know, and you should therefore be cautious and skeptical about trusting it.

Indicators of a Lack of Support

As you can readily guess, the lack of supporting evi-dence provides the best indication that there is indeed no available support.

Be careful then, when a source shows problems like these:

As you continue to work in the world of information, you should begin to devel-op an effective “baloney detector,” which will enable you to sense when a claim is too good, too bad, too excessive, or too weird to be true. Suppose, for example, you receive a chain letter e-mail claiming that you will be paid five thousand dollars if you for-ward it to all your friends. How likely is that to be true? And if it were true, wouldn’t you expect to hear about such an amaz-ing deal on the nightly news or in the paper? You can keep your baloney detector tuned up by read-ing widely and getting a good sense of what generally happens, and what truths and myths are spreading around the Web or another part of the information environment.

Summary of the CARS Checklist for Research Source Evaluation

Credibility

Trustworthy source, the quality of evidence and argument, author’s credentials, evidence of quality control, known or respected authority, organizational support. Goal: an authoritative source; a source that supplies some good evidence that allows you to trust it.

Accuracy

Up-to-date, factual, detailed, exact, comprehensive, audience and purpose reflect intentions of completeness and accuracy. Goal: a source that is correct today (not yesterday); a source that gives the whole truth.

Reasonableness

Fair, balanced, objective, reasoned, no conflict of interest, absence of fallacies or slanted tone. Goal: a source that engages the subject thoughtfully and reasonably; a source concerned with the truth.

Support

Listed sources, contact information, available corroboration, claims supported, documentation supplied. Goal: a source that provides convincing evidence for the claims made; a source you can triangu-late (find at least two other sources that support it).