Performance Measurement and Analysis of H.323 Traffic

17 March 2005

Performance Measurement and Analysis of H.323 Traffic

Performance Measurement and Analysis of H.323 Traffic

Prasad Calyam ([email protected])

Mukundan Sridharan ([email protected])

Weiping Mandrawa ([email protected])

Paul Schopis ([email protected])

19th April 2004, PAM 2004, Juan-Les-Pins, France

Prasad Calyam ([email protected])

Mukundan Sridharan ([email protected])

Weiping Mandrawa ([email protected])

Paul Schopis ([email protected])

19th April 2004, PAM 2004, Juan-Les-Pins, France

2“ PAM 2004, April 19th 2004” www.oar.net

Background

Today H.323 Videoconferencing is a popular collaboration technology in both industry and academia

Demand for high-quality Video and Audio in H.323 applications is always on the rise even given the rapid advancements in the recent past

The “Network” is the critical variable that needs to be better understood in assuring satisfactory end-user experience while using H.323 applications

Our fundamental question:“What can any given network health diagnostic actually tell us about what the end-user experience might be when using an H.323 application such as Videoconferencing on that network?”


Topics of Discussion

Goals of our studyTerminologyTest Setup and MethodologyResults of our studyConclusions


Goals of our Study

To obtain “Good”, “Acceptable” and “Poor” performance bounds for network metrics such as delay, jitter and loss for H.323 applications, based on simultaneous subjective and objective quality assessment of H.323 audio and video streams

To identify the most dominating factor amongst delay, jitter and loss that affects end-user perception of audiovisual quality


H.323 Overview

H.323 is an umbrella standard that defines how real-time multimedia communications such as videoconferencing can be supported on packet switched networks (Internet)

Devices: Terminals, Gateways, Gatekeepers and MCUs

Codecs: H.261, H.263, G.711, G.723.1Signaling: H.225, H.245Transport Mechanisms: TCP, UDP, RTP and RTCPData collaboration: T.120Many others…


H.323 Protocol Stack

NETWORK

DATA LINK

PHYSICAL

TRANSPORT

SESSION

PRESENTATION

APPLICATION

Supplementary Services

Audio Signal

Video Signal Data

Control

G.711 G.728H.261 H.263 T.127

T.126

T.124

T.125/T.122

G.722 G.729

G.723.1

RTCP RAS RTP

H.450.3 H.450.2

H.450.1H.235

H.245 H.225UDP TCP

X.224.0


Understanding the factors that affect H.323 Traffic performance assessment…

Human FactorsIndividual perception of audiovisual quality, Lack of training to use the system effectively, …

Device FactorsVoIP endpoints, Gateways, MCUs, Routers, Firewalls, NATs, Modems, Operating System, Processor, memory, …

Network FactorsDelay, Jitter, Loss, throughput, BER, …


E2E Measurements of Audio and Video Applications

Two approaches to evaluating the performance of audiovisual quality

Subjective MeasurementsInvolve human participants to rate audiovisual quality Can you hear me now?We used the Mean Opinion Score (MOS) Ranking

technique (ITU-T P.800) Not just “Good”!

Objective MeasurementsAutomated techniques to rate audiovisual quality We used the “E-Model” (ITU-T G.107)


MOS Rankings


What constitutes a “Task”?

Any activity that takes place in a routine Videoconference

Casual conversation, intense discussion, a class lecture, movements involving minimal physical activity, …

Recommendations insist realistic scenarios, not just passive viewing

Guidelines can be found for task selection, participant training for scoring the audiovisual quality, task ordering, overall environment setup for quality assessments, …

H.323 Beacon loopback feature provided additional tasks for the various assessments scenarios


Understanding Delay…

Compression Delay

Transmission Delay

Electronic Delay

Propagation Delay

Processing Delay

Queuing Delay

Resynchronization Delay

Decompression Delay

Presentation Delay

SENDER SIDE NETWORK RECEIVER SIDE

Delay is the amount of time that a packet takes to travel from the sender’s application to reach the receiver’s destination application

Caused by codecs, router queuing delays, …One-way delay requirement is stringent for H.323

Videoconferencing to maintain good interaction between both endsGood (0ms-150ms), Acceptable (150ms-300ms), Poor

(> 300ms) Our Results!


Understanding Jitter…

Jitter is the variation in delay of the packets arriving at the receiving end

Caused by congestion, insufficient bandwidth, varying packet sizes in the network, out of order packets, …

Excessive jitter may cause packet loss in the receiver jitter buffers thus affecting the playback of the audio and video streamsGood (0-20ms), Acceptable (20ms-50ms), Poor (>50ms)

Our Results!


Understanding Loss…

Packet Loss is the packets discarded deliberately (RED, TTL=0) or non-deliberately by intermediate links, nodes and end-systems along a given transmission path

Caused by line properties (Layer 1), full buffers (Layer 3) or late arrivals (at the application)

Good(0%-0.5%), Acceptable (0.5%-1.5%), Poor (>1.5%) Our Results!


Overall Test Setup


World Sites involved in Testing…

More than 500 one-on-one subjective quality assessments from Videoconferencing end-users and corresponding traffic traces were obtained from the Testing!!!


Various Tools Used in the Testing…

OARnet H.323 Beacon – a tool to troubleshoot H.323 performance problems

Network path characteristics determination, Audio loopback, customized tests, …Ethereal – a network traffic sniffer tool

To capture traffic traces for Objective Ratings analysisTelchemy VQMon – a network traffic trace analyzer for VoIP

Used for obtaining Objective Ratings for various network settingsNISTnet – a network emulator tool

To create realistic network scenarios by introducing Delay, Jitter and LossSpirent SmartBits – a commercial network measurement suite

Used to qualify NISTnetNLANR Iperf – a TCP/UDP bandwidth measurement tool

Network path characteristics determinationjaalaM appareNet – a commercial network measurement suite

Network path characteristics determinationPolycom FX Videoconferencing Station – Audio/Video clips source


H.323 Beacon Screenshots…


H.323 Beacon Screenshots…(Contd.)


H.323 Beacon Screenshots… (Contd.)

http://www.itecohio.org/beacon


Internet Tests Methodology…

Design of Experiments → 2 Phase ApproachPhase I: LAN with WAN Emulation Tests

Full factorial (n**3 =27; n=3) Tasks with various network health scenario configurations on NISTnet

Phase II: Internet Tests 9 equivalent tasks out of possible 27 tasks + 3 H.323 Beacon tasks using the loopback featureEnd-to-End delay, jitter and loss values configured on NISTnet for each Internet test site = (LAN NISTnet Settings) – (Inherent Path Characteristics for corresponding Internet test site)

VII. G G PVIII. G P GIX. P G G I. BG BG BGII. BA BA BAIII. BP BP BP

I. G G GII. A A AIII. P P PIV. G G AV. G A GVI. A G G

G → Good; A → Acceptable; P → Poor; B(G/A/P) → Beacon Task


Results: Subjective and ObjectiveMOS Vs Delay

Subjective and Objective MOS Vs Delay

Pearson correlation co-efficient for Subjective and Objective Scores for Delay = 0.827


Results: Subjective and ObjectiveMOS Vs Jitter

Subjective and Objective MOS Vs Jitter

Pearson correlation co-efficient for Subjective and Objective Scores for Jitter = 0.737


Results: Subjective and ObjectiveMOS Vs Loss

Subjective and Objective MOS Vs Loss

Pearson correlation co-efficient for Subjective and Objective Scores for Loss = 0.712


Overall Results of Quality Grade Assessments for LAN/Internet Tests

GA*/PAG*/AA/PA/PPA/PGResultGGAGGPPAGLossGAGGPGPAGJitterAGGPGGPAGDelayS9S8S7S6S5S4S3S2S1

G → Good; A → Acceptable; P → Poor; S1 - S9 → Scenario 1-9

* next to a grade → end-user will more often perceive that Grade of quality in that particular scenario


Results: Effects of Normalized Delay, Jitter and Loss variations on Subjective MOS

0 1 2 3 4 5 61

1.5

2

2.5

3

3.5

4

4.5

5

5.5

Su

bje

ctiv

e M

OS

Normalized Scale

DelayJitterLoss

Normalized Scale: 1 Unit → 150ms Delay; 20ms Jitter; 0.5% Loss


Results: Effects of Normalized Delay, Jitter and Loss variations on Objective MOS

0 1 2 3 4 5 61

1.5

2

2.5

3

3.5

4

4.5

Ob

ject

ive

MO

S

Normalized Scale

DelayJitterLoss

End-user perception of audiovisual quality is more sensitive tochanges in jitter than to changes in delay and loss


Conclusions

We have determined performance bounds for latency, jitter and loss by conducting exhaustive testing in a LAN environment with WAN Emulation

By using many incremental values of each of the 3 metrics meticulously, we determined what ranges of these values in isolation and in combination affect the user perception of audiovisual quality as "good", "acceptable" and "poor" grades

We have demonstrated our theory of bounds developed in a LAN with WAN emulation, actually holds good in the “Internet”

By using literally every type of last mile connection and by testing with sites all over the world, we proved our LAN results scale consistently to the Internet


Conclusions (Contd.)

We have observed the H.323 Beacon loopback assessments (again both subjective and objective) to be in accord with the audiovisual tasks test results

This demonstrates the handy utility of the H.323 Beacon in determining user perceived audiovisual quality in H.323 production systems without remote end-user intervention

Our normalized results indicate that end-user perception of audiovisual quality is more sensitive to changes in jitter than to changes in delay and loss


Questions

Documents

Performance Measurement and Analysis of H.323 Traffic