Best UFO Cases” by Isaac Koi

 

PART 21:         Quantitative criteria : Vallee’s SVP ratings

 

Since the 1960s onwards, Jacques Vallee has written several discussions regarding classification and codification of UFO reports.

 

During 1963, he published one of the first articles on classifying UFO reports into various types (see Footnote 21.12).  That proposal including suggestions for the codification of certain indications of interest (e.g. the number of witnesses), but no codes evaluating the relative credibility of the reports.  (Even that basic proposal was met with some concern that it might give a misleading impression of the similarity between cases within the various categories proposed by Vallee – see Footnote 21.16).

 

A couple of years later, in his first book - “Anatomy of a Phenomenon : UFOs in Space” (1965) – Vallee referred to the assignment of a “reliability index” as part of the first step of an analysis (see Footnote 21.14). That concept was explained in an article he wrote later that year (see Footnote 21.13).  In that article, Vallee commented that all writers on the subject of UFOs agree on one point: “many reports refer to misinterpreted conventional objects” and asked “But exactly how many reports are significant? How do you go about finding them?”.  He commented (in an observation which remain almost as valid several decades later, as can be seen from the other Parts of this article) that: “Yet very little information is found in the literature on exactly how to select your sample. It seems that every UFO student uses his own judgment to make the choice … Most UFO studies thus generate confusion instead of clarification”.   He complained that no “reliability scale” was given in statistics relating to UFOs.  He expressed regret that UFO groups and magazines did not have “a set of simple tests ready for use when a report comes in, to weigh its degree of significance”.  He set out a flowchart as a “guide for the identification of obvious mistakes which have no place in a catalogue of UFO sighting”. However, as noted in the conclusion of that article, the proposed procedure “leaves the final estimate of the report to the investigator’s judgment”.

 

In his subsequent book “Challenge to Science – The UFO Enigma” (1966), Jacques Vallee included a detailed appendix entitled “An Analysis of UFO Activity” setting out proposed classifications and codifications of UFO reports.  That appendix included a section entitled “Reliability (Weight) of the Sightings” (see Footnote 21.15). Vallee explained that the “weight” to be given to a sighting within his system “is not only a measure of the reliability of the witness, it seeks to determine to what degree each report is important in a study of the phenomenon”, setting out the following categories represented by different characters:

 

“*” = “sightings that must by accounted for in any global theory of the phenomenon, either because of the strong evidence obtained or because of the large number or scientific competence of the witnesses (assuming favourable observing conditions)”.

 

“+” =  “significant cases where we feel that sincerity of the witnesses cannot be questioned, and where the reported phenomenon is representative of the problem under study”.

 

“=” = “doubtful cases where the report can be interpreted, on the basis of the data presented, by a borderline conventional phenomenon”

 

“-” = “nothing to do with the UFO phenomenon, but have to be catalogued because of the effect they have had on the general rumour, at least on a local scale”.

 

 

Since then Jacques Vallee has proposed a system of assigned three digits to indicate the credibility of a report.

 

Jacques Vallee’s “SVP” system involved assigning a value from zero to four for “S” (reliability of the Source), “V” (site Visit) and “P” (probability of natural explanations).

 

He has discussed this proposal in several of his books, including in “Confrontations” (1990) and “Revelations” (1991).

 

While this system is not as referred to in UFO books as frequently as Hynek’s Strangeness and Probability Ratings (see PART 20), it appears to me that Vallee’s system may have actually been implemented to a greater extent by several UFO databases and groups than Hynek’s better known proposal.

 

Vallee has made strident comments about the failure of other researchers to implement such a system.  For example, he has made the following comments in the books referred to above:

 

  1. “No classification system is complete without a way of assigning credibility or ‘weight’ to an oberservation.  While such a procedure is an integral part of any intelligence evaluation task, UFO researchers have rarely bothered to apply it in support of their own work” (see Footnote 21.02).

 

  1. “In the absence of such a rating [of the credibility of UFO reports], the UFO databases and catalogues that exist today are little more than large buckets filled with random rumours” (see Footnote 21.03).

 

Vallee has referred to a notable exception to his criticisms, i.e. the “quality index” proposed by Spanish researchers Vicente Juan Ballester-Olmos and Guasp, “but it is so detailed that I have found it difficult to apply in practice (see Footnote 21.02).  Vallee has stressed that a system needs to be simple enough to be applied quickly, and with enough mnemonic value to insure it does not require constant reference to a thick codebook.  The relevant “quality index” proposed by Ballester-Olmos and Guasp is discussed in PART 23.

 

Vallee proposal of a “very simple system a (‘the SVP rating’) to indicate the credibility of reports” relies on “only three questions” (see Footnote 21.01), i.e.:

 

1. “Do we know the source of the report?”

 

2. “Was a site visit made?”

 

3. “And what alternative explanations exist for the event?”

 

Each of the three digits assigned for S, V and P has a value from zero to four, as follows:

 

The first digit, “S” indicates the reliability of the source:

 

0 = unknown source, or an unreliable source

1 = a source of unknown reliability

2 = credible source, second hand

3 = credible source, first hand

4 = firsthand personal interview with the witness, by a source of proven reliability

 

 

The second digit, “V” indicates whether or not a site visit took place:

 

0 = no site visit, or the answer is unknown

1 = visit by a casual person unfamiliar with such phenomena

2 =  site visit by a personal familiar with the range of phenomena

3 = site visit by a reliable investigator with some experience

4 = site visit by a skilled analyst

 

 

The third digit, “P” indicates the probability of natural explanations:

 

0 = data is consistent with one or more natural causes

1 = natural explanation only requires slight alteration of the data

2 = natural explanation would demand gross alteration of one parameter

3 = natural explanation demands gross alteration of several parameters

4 = no natural explanation is possible, given the evidence

 

Thus, a rating of 222 or better (meaning that each of the three digits is 2 or higher) is supposed to indicate events reported through a reliable source, in which a site visit has been made, and where a natural explanation would require the gross alteration of at least one parameter.

 

 

Actual applications of Vallee’s SVP criteria

 

In an online article dated April 2007 on his website (see Footnote 21.08), Jacques Vallee has updated the discussion of his proposed classifications and SVP criteria that appears in the books referred to above.

 

In that online article, Vallee states that:

 

“This system is now in use in all our catalogues. It has also been used consistently by several major external studies, notably by CUFOS in their UFOCAT catalogue, by the National Institute for Discovery Science (NIDS) in its private database, and by the French study of pilot sightings conducted by M. Dominique Weinstein in connection with the GEIPAN (Groupe d’Etudes et d’Information sur les Phénomènes Aériens Non-identifiés) in connection with the CNES in Paris”.

 

Vallee concludes that “it is becoming possible to compare statistical data in cross-indexing among several databases, a significant first step towards international cooperation in the study of this puzzling phenomenon”.

 

It is not clear from the article which catalogues are included within Vallee’s reference to “all our catalogues” which now use his classifications and SVP criteria.  The same article refers to four databases developed by Vallee, so presumably it includes those databases.  They do not appear to be available on, or via, Vallee’s website.  I have made enquiries with various researchers but they were unable to enlighten me as to the nature or content of those databases. I have not troubled Jacques Vallee himself as yet.

 

As for the several major external studies that have, according to Vallee, “used consistently” the SVP criteria:

 

 

  1. (1) UFOCAT:

 

The biggest catalogue mentioned by Vallee appears to be CUFOS’s UFOCAT.

 

The “UFOCAT 2002 User’s Guide” states that Vallee SVP “system appears to have the advantage that it is simple enough to be applied quickly with enough mnemonic value that it does not require constant reference to a reference manual” (see Footnote 21.10).

 

However, the SVP rating does not appear in relation to most of the records within UFOCAT.  It seems to be present for only approximately 2,900 out of more than 65,000 sightings covered in the 2004 edition of the UFOCAT database.

 

I note that over 10 per cent of the rated sightings have the maximum SVP value, i.e. 444.  On the other hand, there are only two records rated 000 (both in relation to a hoax).  Given the number of records within UFOCAT labelled hoaxes (and there are quite a few, which should provide a very useful resource to anyone looking into hoaxed reports) that are not assigned any SVP, it may well be that there is a selection bias in that the researchers that coded entries teneded only to bother including SVP ratings for the more credible reports.

 

I have yet to locate any analysis of those SVP records. The researcher that has managed the UFOCAT project since about 1990 (Donald Johnson) is also unaware of any analysis of those records, commenting that there has not been any “in large part because the assignment of these SVP codes has been lopsided in favour of the superior cases”.  He commented that he principally used the codes to filter out cases when he was looking for quality cases and that “until there is a systematic assignment of a representative sample of cases they really can't be used for statistical analysis because the results would be meaningless” (see Footnote 21.17).

 

Nonethless, the inclusion of more than 2,900 SVP ratings in the UFOCAT makes this the second largest attempt to apply any quantitative criteria to assess the weight/credibility of UFO reports of which I am aware. (The largest such attempt is in Larry Hatch’s *U* database, which includes values in relation to Strangeness and Probability – as discussed in PART 20).

 

Due to the scale of that endeavour to add such ratings, I asked Donald Johnson how easy (or otherwise) the SVP criteria are to apply in practice when dealing with a large database. He responded that “The most difficult code to assign is the middle one, and the easiest is the last one. Most reports contain virtually no information about the quality of the investigation, yet I am reluctant to apply a zero code to an otherwise good case that lacks a followup investigation” (see Footnote 21.17).

 

Incidentally, although the SVP values are stated within the UFOCAT database as a single three digit number (e.g. 444) I have found it straightforward to convert UFOCAT’s data into a Microsoft Excel spreadsheet and within Excel it is possible to, for example, add the various digits in a number together (e.g. 4 + 4 + 4 = 12).  It would therefore be possible to create an additional column containing those totals and then rank the 2,900 rated sightings according to the total of their SVP values. (See Footnote 21.11).

 

As discussed in PART 20:  Quantitative criteria : Hynek – Strangeness and Probability, it is notable that UFOCAT, despite being sold by an organisation founded by J Allen Hynek (i.e. CUFOS), does not include Hynek’s Strangeness and Probability Ratings but has instead (to some extent) adopted Vallee’s SVP ratings.

 

 

 

  1. (2) NIDS Database

 

The private database of the National Institute for Discovery Science (NIDS) mentioned by Jacques Vallee appears to remain private.  This is unlikely to change, given that the National Institute for Discovery Science (NIDS) was disbanded back in about 2004. (The last substantive update to its website was in September 2004 – see Footnote 21.05).

 

However, an article was published on the NIDS website in April 2001 entitled  “The NIDS UFO Database: Classification and Credibility Indices” (see Footnote 21.06) which included an analysis of the SVP values assigned by NIDS to 660 cases received over the a 15 months period. That article stated that “a significantly higher level of our close encounter cases (71.7%) have high credibility according to the Vallee SVP index” than non-close encounter cases.

 

That article indicated that the numbers of cases in the database were “still very small” and referred to a hope “to add to the data in the coming months”, but no further similar article appears to have appeared on the NIDS website before NIDS was disbanded three years later.

 

  1. (3) Dominique Weinstein’s study

 

I have not found a copy of the French study of pilot sightings conducted by M. Dominique Weinstein in connection with the GEIPAN (Groupe d’Etudes et d’Information sur les Phénomènes Aériens Non-identifiés) in connection with the CNES in Paris.

 

I have seen various publically available catalogues of pilot sightings compiled by Dominique Weinstein (see Footnote 21.06 and Footnote 21.07).  However, those publically available catalogues do not appear to include SVP ratings or contain any analysis of SVP ratings.

 

 

 

 

Joseph Randall Murphy’s “Ufology Society International”

 

Vallee’s SVP ratings also appear to have been applied (although somewhat modified) by Joseph Randall Murphy’s “Ufology Society International”, also known as “USI” (see Footnote 21.09). The “USI Confidence Rating” scheme uses (or used, since the relevant group’s website does not appear to be available as more of June 2010) the same three categories as the Vallee SVP system plus one more for the type of memory a UFO report is gatherd from. The addition rating, the Mnemonic Rating (M), included the following:

 

0: Recalled via channeling dream or other altered state.

1: Hypnosis assisted with minimal corroboration.

2: Hypnosis assisted with independent corroboration.

3: Conscious recall of an event more than 5 years old.

4: Clear Conscious recall of a recent event.

 

When I asked J R Murphy about this additional factor by email in 2007, he kindly explained that “The USI Mnemonic Rating was developed for the purpose of providing a framework for addressing the memory state via which sighting report data is gathered. For the constructively skeptical and objective ufologist, this should be an important factor, but whether it will ever get established is another story”.  It appears that, thus far at least, Murphy’s suggseted additional factor has not been applied by any other groups or databases, nor have I been able to locate any analysis of the application of this additional factor to any files held by Murphy’s “Ufology Society International”.

 

Robert Moore (a British ufologist that has edited several UFO magazines, including one published by BUFORA) has commented that Vallee's SVP rating system "is a fair superior system to the Berliner scale" discussed in PART 20, commenting that Vallee's scale "very much represents the reality of Ufology as confronted by field investigators!" (see Footnote 21.18).

 

 

 

FOOTNOTES

 

[21.01] Jacques Vallee, “Revelations” (1991) at page 291 (in Appendix 2) of the Ballantine Books paperback edition.

 

[21.02] Jacques Vallee, “Confrontations” (1990) at page 218 (in the Appendix) of the Ballantine Books paperback edition.

 

[21.03] Jacques Vallee, “Revelations” (1991) at page 291 (in Appendix 2) of the Ballantine Books paperback edition.

 

[21.04] Archive of NIDS website, articles section:

http://web.archive.org/web/20070930043450/www.nidsci.org/whatsnew.php

 

[21.05] NIDS, “The NIDS UFO Database: Classification and Credibility Indices”, April 2001.

http://web.archive.org/web/20061210060109/www.nidsci.org/pdf/nids_ufo-database_0301.pdf

 

[21.06] Dominique Weinstein, “Military, Airline and Private Pilot UFO sightings from 1942 to 1996”, 1997.  Available online as at 1 June 2010 at:

http://www.project1947.com/acufoe.htm

 

[21.07] Dominique Weinstein, “Catalog of Military, Airliner, Private Pilots sightings from 1916 to 2000”, February 2001 edition.  Available online as at 1 June 2010 at:

http://www.narcap.org/reports/004/tr-4c.htm

 

[21.08] Jacques Vallee, “System of Classification and Reliability Indicators for the Analysis of the Behavior of Unidentified Aerial Phenomena”, April 2007.  Available online as at 1 June 2010 at:

http://www.jacquesvallee.net/bookdocs/classif.pdf

 

[21.09] Website of Joseph Randall Murphy’s “Ufology Society International” .  Available online as at 1 June 2007 at:

http://www.nucleus.com/~ufology/USI/Content/Classes-01.htm

 

[21.10] Donald Johnson, “The UFOCAT 2002 User’s Guide”, April 2002 at page 30.

 

[21.11] To add all the digits of a number within Excel, see (for example) the webpage at the link below which suggests using the following : “=SUM((LEN(A2)-LEN(SUBSTITUTE(A2,{1,2,3,4,5,6,7,8,9},"")))*{1,2,3,4,5,6,7,8,9})”:

http://www.mrexcel.com/board2/viewtopic.php?t=53620

 

[21.12] Jacques Vallee “How to classify and codify Saucer sightings”, FSR Volume 9 issue 5, September/October 1963, pages 9-12.

 

[21.13] Jacques Vallee “How to select significant UFO reports”, FSR Volume 11 issue 5, September/October 1965, pages 15-18.

 

[21.14] Jacques Vallee, “Anatomy of a Phenomenon” (1965) at pages 39-40 (in Chapter 3) of the Henry Regnery hardback edition (with the same page numbering in the Tandem paperback edition).

 

[21.15] Jacques Vallee and Janine Vallee, “Challenge to Science : The UFO Enigma” (1966) at pages 266-267 (in Appendix 4) of the Ballantine Books paperback edition, at page 222 of the Tandem paperback edition.

 

[21.16] William T Powers,“Some Preliminary Thoughts on Data Processing”, FSR Volume 12 issue 4, July/August 1966, pages 21-22

 

[21.17] Email from Donald Johnson to Isaac Koi, 21 June 2010.

 

[21.18] Email from Robert Moore to Isaac Koi, 24 June 2010.

Category: