Performance Comparison ofDTN Bundle Protocol Implementations

Wolf-Bastian Pöttner Johannes Morgenroth Sebastian Schildt Lars WolfIBR, Technische Universität Braunschweig

Mühlenpfordstraße 23, Braunschweig, Germany[poettner|morgenroth|schildt|wolf]@ibr.cs.tu-bs.de

ABSTRACT

In recent years, Delay Tolerant Networking (DTN) has re-ceived a lot of interest from the networking community. To-day the Bundle Protocol (RFC 5050) is the standard com-munication protocol in DTNs and several implementationsfor various platforms are available.

As of now, no quantitative analysis of the different imple-mentation’s performance or a structured evaluation of in-teroperability has been undertaken. We performed interop-erability checks and an extensive quantitative performanceanalysis of three Bundle Protocol implementations for Linuxsystems. In this paper we summarize our main findings.

While the overall results show, that all implementationsprovide some baseline compatibility with each other, thestability and achieved performance under stress situationsvaries widely between implementations.

Categories and Subject Descriptors

C.2.1 [Computer Systems Organization]: Network Ar-chitecture and Design—Store and forward networks

General Terms

Experimentation, Measurement, Performance

Keywords

Bundle Protocol, Evaluation, Performance, DTN

1. INTRODUCTION

Implementing Delay Tolerant Networking (DTN) as a wayto cope with intermittently connected networks has gaineda lot of interest in the research community lately. The stan-dard DTN protocol is the Bundle Protocol (BP) defined inRFC 5050. Today several Bundle Protocol implementationsfor different use cases exist, see Section 1.1.

As of now, no quantitative analysis of the different imple-mentation’s performance or a structured evaluation of inter-

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.CHANTS’11, September 23, 2011, Las Vegas, Nevada, USA.Copyright 2011 ACM 978-1-4503-0870-0/11/09 ...$10.00.

operability has been undertaken. We measured and com-pared the performance of three Bundle Protocol implemen-tations for Linux systems, namely DTN2, IBR-DTN andION, while focussing on the omnipresent TCP convergencelayer [3] (TCPCL). In this paper we give an overview of theresults. For additional results and a more in-depth discus-sion we refer the reader to [7].

We measured throughput and interoperability. As theBundle Protocol is a relatively new standard, interoper-ability problems can point out ambiguities in the specifi-cations. The maximum application layer throughput is oneof the most important quantities to determine when evaluat-ing network protocol implementations. Especially in DTNs,with short contacts between the nodes, the utilization of theavailable bandwidth determines the efficiency of the wholenetwork.

1.1 Bundle Protocol Implementations

DTN2 [2] is the reference implementation of the BundleProtocol by the Delay Tolerant Networking Research Group(DTNRG) featuring a flexible TCL console and XML inter-faces to external components. IBR-DTN [8] is a lightweight,modular and highly portable Bundle Protocol implementa-tion designed for embedded systems running OpenWRT. In-terplanetary Overlay Network (ION) [1] is a Bundle Protocolimplementation from the Jet Propulsion Laboratory (JPL)specifically developed to run in robotic spacecrafts on a real-time operating systems (RTOS).

2. EVALUATION

As already mentioned in Section 1, among the importantmetrics to evaluate Bundle Protocol implementations arethe throughput between two nodes as well as the interop-erability between different implementations. In this sectionwe first measured the throughput between two nodes of thesame implementation for different bundle payload sizes andtwo storage backends. In addition, we took a look at thebasic interoperability between different implementations fora fixed storage backend and measured the throughput thatcan be achieved in a mixed environment. Finally, we havemeasured the throughput in a simple opportunistic scenario.

The following measurements all concentrate on applica-tion layer throughput that is achievable when transportingdata via the Bundle Protocol. Consequently, all sizes arethe payload size of the respective bundles. Measurementsreferring to disk-based storage used a persistent bundle stor-age on a hard drive while measurements with memory-basedstorage stored all bundles in RAM.

61

Throughput is calculated based on the time at which thefirst and the last bundle are received. This ensures that in-fluences of a discovery mechanism that may delay the trans-mission of the first bundle are excluded.

2.1 Experimental Setup

All experiments were conducted in a controlled envi-ronment using computers equipped with an Athlon II X42.8GHz including 4GiB RAM running Ubuntu Linux 11.04.The computers are connected using 1GBit Ethernet. On thePCs we used different physical Ethernet ports for the BundleProtocol traffic and the traffic used for controlling and mon-itoring the experiments. The raw TCP throughput achievedby this setup is ∼940MBit/s which has been measured usingiperf1. In this paper we used DTN2 version 2.7, IBR-DTNversion 0.6.3 and ION version 2.4.0. All implementationshave been measured using their respective default configu-ration. For ION, we have removed the artificial rate limitfor the TCPCL that otherwise prevents ION from exceed-ing 90MBit/s. To prevent missing bundles, we had to enablereversible transactions in IONs storage system.

2.2 Network Throughput

0

200

400

600

800

1000

1 10 100 1000 10000 100000 1e+06 1e+07

Bundle

Thro

ughput

[MB

it/s

]

Bundle Payload Size [bytes]

DTN2 DiskDTN2 Mem

IBR-DTN DiskIBR-DTN Mem

ION Disk (T)ION Mem (T)

Figure 1: Network throughput comparison using disk- andmemory-based bundle storage.

In this experiment we measured the network performanceof the different implementations. We measured throughputbetween two nodes that are connected via GBit Ethernet.While a GBit link might be uncommon for typical DTNapplications it shows the daemons ability to saturate a givenlink. Failing to reach high bandwidths in this experimentindicates that bundle processing overhead in a given daemonmight be too high. This will not only pose a problem witha high bandwidth link, but it could also preclude a resourceconstrained node to saturate a lower bandwidth link.

On the sender node we injected 1000 bundles for the re-ceiver. After the bundles had been received by the sendingdaemon we opened the connection and measured the timeuntil all bundles have arrived at the receiver. For each pay-load size we performed 10 runs and we restarted the daemonsafter completing all runs for a given payload size. Figure 1shows the average of all 10 runs as well as the standard devi-ation. We have used disk-based storage backends, which arethe ones that will likely be used in practice. For comparison,we have also measured memory-based storage backends.1http://iperf.sourceforge.net/

Please note that the high standard deviation of thememory-based storage for DTN2 (which can also be seenin Table 1) is caused by the fact, that the throughput de-creases significantly from run to run.

2.3 Interoperability

For the interoperability tests we check whether the dae-mons can discover each other, which is important for ap-plications relying on opportunistic contacts. However, themost important thing is Bundle Protocol compatibility: Ifall daemons adhere to the Bundle Protocol and TCPCLspecification they should be able to exchange bundles. Forthe working combinations we measure the data throughputto give an estimate of the performance that can be expectedin heterogeneous environments.

2.3.1 Discovery

In order to discover other Bundle Protocol nodes withinthe same subnet, DTN2 uses a proprietary discovery packetformat. IBR-DTN supports the use of IPND [5] and DTN2’sdiscovery packet format. ION focusses solely on scheduledcontacts and cannot discover other nodes. We have per-formed a simple test in which two daemons had to discovereach other and then transfer a bundle. We found, thatDTN2 and IBR-DTN are able to discover each other andexchange bundles.

2.3.2 Throughput

�������TX

RXDTN2 IBR-DTN ION

DTN2 197.231 193.202 71.574±210.016 ±198.240 ±9.456

IBR-DTN 677.153 542.337 76.161±173.756 ±21.533 ±0.623

ION 454.513 420.743 267.832±53.763 ±11.640 ±2.028

(a) 100 KBytes payload size (in MBit/s)

�������TX

RXDTN2 IBR-DTN ION

DTN2 687.329 635.088 93.116±139.590 ±49.902 ±0.568

IBR-DTN 881.463 679.45 89.915±72.149 ±64.362 ±4.561

ION 871.677 926.005 448.833±76.428 ±4.009 ±3.592

(b) 1 MBytes payload size (in MBit/s)

Table 1: Average Interoperability Throughputs usingmemory-based Storage

In this experiment we want to determine how the interac-tion between different implementations impacts the achiev-able throughput. We created 1000 bundles on the sendernode and then measured the time it takes to transfer thesebundles to the receiver node. We used the TCPCL andmemory-based storage to preclude any performance impactsdue to disk caching and similar issues. We set up staticrouting and opened and closed connection between daemonsusing either built-in methods or iptables. We ran these mea-surements for all 9 possible pairs of implementations and theresults are stated in Table 1a for 100KByte payload size andin Table 1b for 1MByte payload. Each test was performed

62

10 times and the tables show the average throughput andthe standard deviation of these runs. The involved daemonswere restarted after completing all runs for a given payloadsize.

2.4 Opportunistic Throughput

Based on a scenario for public transport systems publishedin [6], we have two fixed nodes which are out of range of eachother and a mobile node that commutes between them. Thetiming is based on measurements published in [4].

In each cycle, the mobile node first makes contact withthe sender for 72 s, then has no contact for 30 s to make con-tact with the receiver for 72 s afterwards. After another 30 swithout contact, the cycle starts again with the whole exper-iment lasting 30minutes. The contacts are simulated usingiptables. We limited the TCP throughput to 12.7MBit/sbased on the results presented in [4] using a Token BucketFilter (tbf) based on the Linux Classless Queuing Disciplines(qdisc). The bundle creation frequency is fixed and ensuresthat always enough bundles for the next contact are presenton the sender node. Under optimal conditions this setupwould allow transmitting a total of 1000.59MByte of rawTCP data in 30minutes.

The hardware used is the same as described in section2.1. Each experiment is repeated five times for each bundlesize with DTN2 and IBR-DTN using the memory storagebackends. ION is not able to compete in this test, since itdoes not support opportunistic contacts. We used floodingas routing module for DTN2 and epidemic routing for IBR-DTN. The total amount of payload that has been transferredis listed in table 2, showing the average and the standarddeviation of 5 measurement runs. In this experiment we onlymeasured setups using the same implementation as senderand receiver and did not consider mixed environments.

250 kB 500 kB 1 MB 2 MB

DTN2 918.2 917.3 917.6 908±4.18 ±2.28 ±2.30 ±6.63

IBR-DTN 841.3 913.1 912.6 902±2.96 ±0.65 ±1.52 ±0.00

Table 2: Transmitted data and standard deviation in theopportunistic scenario (in MB)

3. CONCLUSIONS

To the best of our knowledge, we have performed the firstextensive study of performance and interoperability of threewidely used Bundle Protocol implementations.

The results show, that IBR-DTN and DTN2 can trans-fer bundles at a rate well above 600MBit/s when using thesame Bundle Protocol implementation as sender and receiverwhile ION reaches up to 449MBit/s for memory-based stor-age. Furthermore we can see, that the three implementa-tions are generally interoperable. However, the achievableperformance depends on a significant number of factors andcannot be predicted easily. Although intuitively one wouldhave expected to find the highest throughput between twoimplementations of the same type, the fastest way of trans-mitting bundles is to use ION as sender and IBR-DTN asreceiver.

The opportunistic scenario shows that DTN2 and IBR-DTN both are able to use the available bandwidth and time

with an efficiency between 84.08 and 91.77 %. Interest-ingly, the daemons perform almost identical for three pay-load sizes, despite the significant differences in the previousmeasurements. However, for a payload size of 250 kB DTN2significantly outperforms IBR-DTN.

One of the major influencing factors is the underlying stor-age system that effectively limits the achievable throughputfor links with high bandwidth. Disk storage can be consid-ered the default option for DTNs and is not only influencedby the throughput of the disk but also by the concrete imple-mentation and caching mechanisms of the underlying oper-ating system. Especially when the total amount of bundlesexceeds the available RAM, this can be a bottleneck.

We have conducted extensive experimental studies andthis paper presents only a fraction of the results due to spaceconstraints. Please refer to our technical report in [7] foradditional information and interpretation of the results. Inthere, we have extensive evaluation of the impact of differentstorage backends onto the throughput performance. More-over, we have measured the performance of the APIs of thedifferent implementations and we have investigated the im-pact of factors such as TCP segment length or the BundleSecurity Protocol on performance.

Acknowledgements

This work has been partially supported by EFRE (Europeanfund for regional development) project OPTraCom (W2-800-28895) and by the NTH School for IT Ecosystems.

4. REFERENCES

[1] S. Burleigh. Interplanetary Overlay Network: AnImplementation of the DTN Bundle Protocol. In 4th

IEEE Consumer Communications and Networking

Conference, 2007. (CCNC 2007), pages 222–226, Jan.2007.

[2] M. Demmer, E. Brewer, K. Fall, S. Jain, M. Ho, andR. Patra. Implementing Delay Tolerant Networking.Technical report, IRB-TR-04-020, Dec. 2004.

[3] M. Demmer and J. Ott. Delay Tolerant NetworkingTCP Convergence Layer Protocol. IETF Draft, Nov.2008.

[4] M. Doering, W.-B. Pottner, T. Pogel, and L. Wolf.Impact of Radio Range on Contact Characteristics inBus-based Delay Tolerant Networks. In Eighth

International Conference on Wireless On-Demand

Network Systems and Services (WONS 2011), pages195–202, Bardonecchia, Italy, Jan. 2011.

[5] D. Ellard and D. Brown. DTN IP Neighbor Discovery(IPND). IETF Draft, Mar. 2010.

[6] S. Lahde, M. Doering, W.-B. Pottner, G. Lammert,and L. Wolf. A practical analysis of communicationcharacteristics for mobile and distributed pollutionmeasurements on the road. Wireless Communications

and Mobile Computing, 7(10):1209–1218, Jan. 2007.[7] W.-B. Pottner, J. Morgenroth, S. Schildt, and L. Wolf.

An Empirical Performance Comparison of DTN BundleProtocol Implementations. Informatikbericht 2011-08,Technische Universitat Braunschweig, 2011.

[8] S. Schildt, J. Morgenroth, W.-B. Pottner, and L. Wolf.IBR-DTN: A lightweight, modular and highly portableBundle Protocol implementation. ElectronicCommunications of the EASST, 37:1–11, Jan. 2011.

63