October 6,2000

[Annex 1] Overview of the Evaluation Test


The certified enterprises are enterprises that submitted technologies that performed outstandingly in evaluation tests under the following conditions. JASRAC, which served as the project leader, has completed its technology evaluation certification based on the final report submitted by the NRI, which served as the project secretariat, and hereby announces its conclusions.

1. Musical Works Used for the Tests

The musical works used as the assignments were recorded on a CD-R as a WAV format file (44.1kHz/16bit/2ch) and distributed to participating enterprises. The robustness test was conducted on 2 popular music compositions, 1 classical composition, and 2 compositions with distinctive acoustics. The sound quality test was conducted on one of the compositions from each of the aforementioned categories, namely popular music, classical and distinctive acoustic. (The distinctive acoustic compositions used as assignments were traditional Japanese music and percussion compositions.)

2. Overview of the Robustness Tests

The robustness test evaluated whether the data watermarked by the participating enterprises could be extracted after the following processes, in the following environment, under equal conditions for all submitted technologies.

  • Processing in a mastering studio
  • Processing in a broadcasting studio (and a prospective broadcasting environment)
  • Processing for distribution through the Internet and other networks
  • Processing using commonly available consumer level equipment

The specific robustness testing items are as follows.

Robustness testing items
Testing Item Overview of Processing Involved
D/A, A/D transition Digital>>Analog>>Digital
Altered number of channels Stereo (2ch)>> mono
Down sampling 44.1kHz/16bit/2ch >>16kHz/16bit/2ch
Amplitude compression 44.1kHz/16bit/2ch >> 44.1kHz/8bit/2ch
Time and pitch compression and decompression
  • Time compression / decompression: ±10%
  • Pitch shift compression / decompression: ±10%
  • Linear data compression
  • MPEG 1 Audio Layer 3 (MP3): 128kbps
  • MPEG 2 AAC: 128kbps
  • ATRAC: Version 4.5
  • ATRAC 3: 105kbps
  • RealAudio: ISDN
  • Windows Media Audio: ISDN
  • Non-linear data compression
  • FM (FM multiple broadcast, terrestrial hertzian TV broadcast)
  • AM (AM broadcast)
  • PCM (Satellite TV broadcast: communications satellite, broadcasting satellite)
  • Characteristic transformation of frequency response
  • FM (FM multiple broadcast, terrestrial hertzian TV broadcast)
  • AM (AM broadcast)
  • PCM (Satellite TV broadcast: communications satellite, broadcasting satellite)
  • Noise White noise: S/N: - 40dB

    3. Overview of Audibility

    1. ABX test conducted
      • The listener listens to a sound recording with no watermark (A), a sound recording with watermarks embedded (B), and a sound recording which is one of the two (X).
      • The listener listens to A and B alternately twice for 40 seconds each, and then listens to X for 40 seconds. The listener decides whether X is A or B.
    2. Elimination of (correct) contingency responses
      • In order to eliminate (correct) contingency responses, the above tests were conducted 5 times for each participating enterprise.
      • The listener is defined to have detected the embedded watermark if the same listener correctly determines whether the watermark is embedded or not on each of the 5 tests. Under this definition, the significance of the responses are 95% or greater.
    3. 4 individuals each from the following occupation categories were selected as Golden Ears and Silver Ears from the recording industry.
      • Recording engineers
      • Mastering engineers
      • Synthesizer manipulators
      • Audio critics

    Inquiries: webmaster@nri.co.jp
    Copyright(c) 2000 Nomura Research Institute, Ltd. All rights reserved.