FREE ELECTRONIC LIBRARY - Dissertations, online materials

Pages:   || 2 | 3 | 4 | 5 |   ...   | 17 |

«INTEGRATED HARDWARE/SOFTWARE DESIGN OF A HIGH PERFORMANCE NETWORK INTERFACE by Zubin Dittia Prepared under the direction of Professor Gurudatta M. ...»

-- [ Page 1 ] --






Zubin Dittia

Prepared under the direction of Professor Gurudatta M. Parulkar

A dissertation presented to the Sever Institute of

Washington University in partial fulfillment

of the requirements for the degree of


May 2001 Saint Louis, Missouri






by Zubin Dittia ADVISOR: Professor Gurudatta M. Parulkar May, 2001 Saint Louis, Missouri This thesis describes the design and implementation of a high performance network interface chip called the APIC (ATM Port Interconnect Controller). It also describes architectural enhance- ments to operating system (OS) software that are necessary to exploit some of the novel features that have been integrated into this chip.

High-performance network interface design has received significant interest from the research community in recent years because traditional design methodologies have not been successful in translating high network bandwidths and low network latencies to improved performance for appli- cations. This can be attributed to several factors: in the past network interfaces have been designed without careful consideration of the operating system software environment in which they get used;

main memory bandwidths have not scaled at the same rate as network bandwidths; and network interfaces and protocols have not been designed to support quality of service for applications. These are the problems addressed by this thesis, the objective being to develop new mechanisms which can result in significant improvements in application performance. In addition to incorporating these innovative features, the APIC design borrows proven and useful ideas from a number of com- mercial and research prototypes.

One of the ways in which the APIC addresses the memory bottleneck alluded to above is to function in a desk-area environment where different memories can be used to spread the load. The idea here is to dedicate one APIC chip and one memory bank to each high-bandwidth device in the system, thereby shedding the load from a host system’s main memory. Several such APIC-memory- device combinations can be daisy chained to form a desk-area network with high bandwidth and low latency characteristics.

There are several well-known operating system overheads associated with in-kernel implementations of network interface device drivers. These include context switch latency, system call overhead, and interrupt overhead. It is possible to remove a number of these inefficiencies and allow for increased performance for end applications if the data path of the device driver can be implemented as a library in user-space. While this idea has been proposed in the recent past, the APIC introduces two new mechanisms, Protected DMA and Protected I/O, which together provide for an efficient method for the implementation of user-space drivers.

Another problem which plagues high-speed network adapters is called receive livelock; this term is used to describe the situation in which, under heavy load, an operating system servicing a device might end up spending all its time in the interrupt service routine, and no useful work gets done. The APIC introduces a novel concept called Interrupt Demultiplexing, which taken alone can alleviate the effects of interrupt livelock, but in conjunction with user-space drivers can solve the problem entirely.

Network interfaces, except for ATM interfaces, have traditionally not provided special mechanisms for supporting quality-of-service (QoS) guarantees. Even ATM interfaces have traditionally supported QoS only to a limited extent. By providing pacing support independently for large numbers of connections, the APIC is able to efficiently and reliably support QoS guarantees simultaneously for large numbers of multimedia streams. This can be especially useful in the context of large multimedia-on-demand servers. This feature was made possible through a novel pacer design which uses a hardware d-heap data structure.

The APIC has been successfully implemented in 0.35 micron technology, and is currently in use in several projects both at Washington University and elsewhere, as part of the NSF-sponsored gigabit kits project.

to my parents Contents Tables



1 Introduction

1.1 Goals

1.2 Features of the APIC Chip

1.3 Contributions

1.4 Outline

2 Background and Motivation

2.1 DMA Subsystems

2.1.1 Why is the DMA Subsystem so important?

2.1.2 NIC design choices that affect the DMA Subsystem................ 11

2.2 Architectural Impact on Latency

2.2.1 Impact of interrupts on latency

2.3 Receive Livelock

2.4 QoS Support in Network Interfaces

–  –  –

3 Related Work

3.1 Related work in network interface design

3.2 Network interfaces supporting user-space control

3.3 Desk-Area Networks

3.4 Reducing Interrupt Overhead

3.5 Receive Livelock Elimination

3.6 QoS support for network interfaces

4 Contributions

4.1 Problem Statement

4.2 Overview of Solutions

5 Architecture Overview

5.1 APIC as a Network Interface Device

5.2 APIC-based DANs

5.3 Ports and Connections

5.3.1 The ATM Ports

5.3.2 The Bus Port

5.3.3 Virtual Connections (VCs)

5.4 Basic Operation

5.4.1 Segmentation and Reassembly

5.4.2 Packets and Frames

5.4.3 Cut-through Behavior

5.4.4 Channels and Connections (VCs)

5.5 Summary of Features

5.5.1 Multipoint and Loopback

5.5.2 AAL-0

5.5.3 AAL-5

v 5.5.4 Traffic Types

5.5.5 Batching

5.5.6 Remote Control

5.6 User-Space Control

5.6.1 Protected I/O

5.7 DMA Modes

5.7.1 Simple DMA

5.7.2 Pool DMA

5.7.3 Protected DMA

5.7.4 Packet Splitting

5.8 Interrupt Mechanisms

5.8.1 Interrupt Demultiplexing

5.8.2 Orchestrated Interrupts

5.8.3 Notification Lists

5.9 Miscellaneous Features

5.9.1 TCP Checksum Assist

5.9.2 Flow Control

5.9.3 Cache Coherent Bus Transfers

6 Internal Design of the APIC Chip

6.1 Clock Regimes

6.2 Module Functions and Paths Taken by Cells Through the Chip.......... 83 6.2.1 Synchronization Modules

6.2.2 Input and Output Ports

6.2.3 BusInterface

6.2.4 RegisterManager

6.2.5 VCXT

6.2.6 CellStore

6.2.7 RxSync

6.2.8 Requestor

–  –  –

7 APIC Software

7.1 Overall Software Framework

7.2 Kernel Driver Structure

7.2.1 Interaction with IP

7.2.2 Interaction with RATM

8 Experimental Results

8.1 Best-effort TCP Throughput

8.2 Pacing Test for UDP Traffic

8.3 Pacing Test for TCP Traffic

8.4 End-to-end Delay and Driver Performance

8.5 Protected DMA Throughput and Delay Performance

9 Conclusions

9.1 Contributions

9.2 Future Work

9.3 Closing Remarks



–  –  –

4.1 Comparison with Other Network Interfaces

8.1 Performance metrics for NetBSD on PCs used in experiments

8.2 Probe points in the protocol stack

8.3 Results of Ping-Pong Test

8.4 Results of User-Space Throughput Test


2.1 Typical Host Architecture

2.2 Data Touch Overhead in a Typical Protocol Stack

2.3 Impact of On-board Memory on Data Touches

2.4 Illustration of Receive Livelock

2.5 Behavior of a paced channel

2.6 Traditional versus User-space Control Model

2.7 Protection concerns in the user-space control model

3.1 Network Adapter Board (NAB) Architecture

3.2 Providing protected access to registers using VM overloading

3.3 Turner’s Pacing Algorithm

5.1 Location of an ATM NIC in a Computer System

5.2 An APIC Interconnect as a Desk Area Network

5.3 Perfect Shuffle Topology

5.4 Instances of Multipoint and Loopback Connections

5.5 An Example Multipoint Application

5.6 AAL-0 Frames and SAR

5.7 Memory-Mapped I/O Address Space of the APIC

5.8 Providing Protected Access to Registers using VM Overloading


5.9 Fine Grain Access Control Using Protected I/O

5.10 A Descriptor Chain

5.11 Transmitting Data Using a Descriptor Chain

5.12 FIFO Queue Model for a Transmit Descriptor Chain

5.13 FIFO Queue Model for a Receive Descriptor Chain

5.14 Receiving Data Using a Descriptor Chain

5.15 Illustration of Simple DMA

5.16 FIFO Queue Model for Simple DMA

5.17 Illustration of Pool DMA

5.18 FIFO Queue Model for Pool DMA

5.19 Illustration of Protected DMA

5.20 Notarization for Protected DMA

5.21 Pool DMA with Packet Splitting

5.22 Zero-Copy Using Packet Splitting and Page Remapping

5.23 A Different Way of Structuring a NIC Driver

6.1 Functional Block Diagram of APIC Internals

6.2 APIC Clock Regimes

6.3 Operation of the RegisterManager Module

6.4 Transit Path Forwarding

6.5 VC Translation Process in the VCXT Module

6.6 FIFO Queues in the Cell Store (Port 2 queues not included)

6.7 The Receive Path

6.8 FIFO Queues in the CellStore

6.9 The Transmit Path

6.10 Control and Response Cell Path

6.11 A Multipoint Receive Path

6.12 A Multipoint Transmit Path

6.13 Loopback Path

6.14 A Multipoint Loopback Path

x6.15 d-Heap Based Pacing

7.1 Software Framework for the APIC

7.2 APIC Kernel Driver Structure

7.3 Example Code to Illustrate RATM Access to the APIC

8.1 Experimental Setup

8.2 Throughput vs. Specified Pacing Rate for UDP Traffic

8.3 Throughput vs. Specified Pacing Rate for TCP Traffic

8.4 Measuring APIC delay and round-trip time performance

9.1 APIC Internal Layout

9.2 The APIC Network Interface Card


First and foremost, I would like to thank my advisor Guru Parulkar, to whose help and encouragement I owe everything. This research would not have been possible without him, and I thank him for his patience in dealing with my sometimes difficult work habits, and for always providing the right mix of freedom, encouragement, and perspective that few advisors can provide for their students.

Second, I would like to thank Dr. Jerry Cox, who has been involved with this project from the onset, and who has made significant contributions to the design effort. He was like a second advisor to me in this undertaking. In particular, a large portion of the internal design of the APIC chip is due to him, including many of the details of implementation of the pacing algorithm described here.

I would also like to thank Rex Hill and Will Eatherton, both of whom were responsible for the VHDL coding effort for most of this chip. Although they joined the project late, both made significant contributions to the design as well as the implementation. In particular, I would like to acknowledge Rex for the paragonal memory layout used in the cell store, and for significant innovation in getting the pacing algorithm committed to silicon. I gave Rex many sleepless nights by insisting on a very clean design, and continually adding and removing features. He seldom complained, and always maintained good humor, which made the gruelling task bearable. I also acknowdge Will for his contributions in coding the chips internal clock domain, and the UTOPIA ports. He has left his mark on the chip in the form of an intelligent flow control algorithm which allows the chip to exert flow control signals over optical links.

xii I owe a lot to John DeHart in this undertaking; he was instrumental in getting the APIC device driver debugged and in a shape in which it could be distributed to Gigabit Kits participants. I admire his abilities and am grateful for his assistance in a very difficult situation, which arose when he had to undertake working with the APIC when all of the chip’s developers, myself included, had left the University.

My thanks also go to Dr. Dave Richards, who spent considerable time and effort on implementing and debugging the MBus and PCI prototype cards, and on the UTOPIA port implementation.

He also contributed his considerable expertise in tracking down and fixing APIC physical level problems once the chips were back from the foundry. His constant speculations about the schedule of the project were, I am sorry to say, right on mark.

I would also like to thank Margaret Flucke for undertaking the massive layout task for the chip, and Tom Chaney and Fred Rosenberger for spending long hours poring over a printout of the chip’s layout to ensure signal integrity and dealing with power and clock distribution issues.

Pages:   || 2 | 3 | 4 | 5 |   ...   | 17 |

Similar works:

«The National Academy of Needlearts Presents The Exemplary 2016 The Somerset Inn Troy, Michigan About the National Academy of Needlearts The National Academy of Needlearts (The Academy) was founded in 1985 to further embroidery education. This goal is accomplished directly through classes and an exhibition held during the annual Assembly for Embroiderers. The Academy indirectly accomplishes its goals through the certification of embroidery teachers, judges, and exhibit administrator’s...»

«Otitis Media and Aboriginal Children A HANDBOOK FOR TEACHERS AND COMMUNITIES Design Rationale On the left is an ear represented in stencil form. The circles lead into the ear. Lines to the circles show that hearing loss can be partial, shown by dotted ends. Wavy lines leading into the ear indicate the different influences on hearing. At the bottom is the family who keep the health of the community through their children. In the middle are message sticks representing the importance of culturally...»

«SIMON & SCHUSTER’S PIMSLEUR ® haitian creole reading booklet Travelers should always check with their nation's State Department for current advisories on local conditions before traveling abroad. Graphic Design: Maia Kennedy © and ‰ Recorded Program 2010 Simon & Schuster, Inc. © Reading Booklet 2010 Simon & Schuster, Inc. Pimsleur® is an imprint of Simon & Schuster Audio, a division of Simon & Schuster, Inc. Mfg. in USA. All rights reserved. aCKnoWleDGMentS Haitian Creole Voices...»

«1 Voorwoord Met trots presenteren wij het locatiespecifieke pedagogisch werkplan van KDV De Globetrotters. Dit is een uitwerking van het werkplan kinderdagverblijf welke van toepassing is op al onze kinderdagverblijven en onderdeel is van het algemeen pedagogisch beleid van SKS Alles Kids. In dit werkplan laten wij zien hoe wij bij KDV De Globetrotters te werk gaan. SKS Alles Kids geeft kinderen de “vijf”. Wij bedoelen hiermee, dat wij werken vanuit vijf speerpunten en dit in praktijk...»

«HPM 472 Program Evaluation in Health Care (Three Credit Hours) Department of Health Policy and Management Gillings School of Global Public Health Spring 2016 Syllabus 2304 McGavran Greenberg Time: Monday – Wednesday (11:15 – 12:30pm) Instructor: Becky Slifkin, PhD, MHA TA: Office: 1116 Rosenau Office: Office hours upon request Email: becky_slifkin@unc.edu Email: Phone: 919-843-1219 Course Overview This course provides an overview of the key concepts, methods, and approaches in the field of...»

«Investigating gender and sexuality in the ESL classroom: Raising publishers’, teachers’ and student’s awareness Jane Sunderland (Principal Investigator) Joanna Pawelczyk (Project Leader) Łukasz Pakuła (Project Leader) This document is an output from the ELT Research Award scheme funded by the British Council to promote innovation in English language teaching research. The views expressed are not necessarily those of the British Council. Outline Introduction! • Gender and sexuality in...»

«Theatre Studies: GA 2 Monologue Performance examination GENERAL COMMENTS In 2002, there was a broad and satisfactory range of results and interpretations and there has been a discernible improvement in the standard of monologue portrayal. The results were indicative of teachers being increasingly knowledgeable regarding the demands of the task. Teachers play an important and positive role in assisting students to choose the most appropriate monologue as well as playing a strong directorial role...»

«To Hell in a Handcart Educational realities, teachers' work and neo-liberal restructuring in NSW TAFE Judith Clark B.A., Dip. Ed., M. Ed. A thesis submitted in fulfillment of the requirements for the degree of Doctor of Philosophy The University of Sydney December 2003 Abstract This thesis examines the impact of neo-liberal economic restructuring on teachers, specifically teachers in technical and further education. Historically, there has been limited research undertaken on teachers as...»

«The Role of Japanese Teachers in English Classes in Japanese Elementary School 1 The Role of Japanese Teachers in English Classes in Japanese Elementary School Yuka Shimizu Key words: English education policy in Japan, elementary schools, an activity type and a subject type, Japanese homeroom teachers, ALTs Introduction: New proposal for English education in Japan, 2013 The Japanese Ministry of Education, Culture, Sports, Science and Technology (henceforth, MEXT) has decided to introduce...»


«Participatory Educational Research (PER) Vol. 1(1), pp. 65-78, June, 2014 Available online at http://www.partedres.com ISSN: 2148-6123 http://dx.doi.org/10.17275/per. Experiences of pre-service English teachers gained through community service learning in kindergartens Gülşah Külekçi* Department of English Language and Teaching, Dokuz Eylül University Education Faculty, İzmir, Turkey Article history Many authors agree that community service learning should be Received: integrated...»

«Rabbi Norman Patz Erev Rosh Hashanah 5775 September 24, 2014 / Tishri 1, 5775 Temple Beth Shalom of Puerto Rico UNDERSTANDING THE NEW LANDSCAPE As we gather to welcome in 5775, I wish you all a good year, a year of health and happiness, of spiritual and moral growth, a year of satisfaction, of fulfillment and peace – the richest of God’s blessings – in a world that sends us so many mixed signals. Our prayer today is y’hee ratzohn mil’fa-nekha Adonai eh-lo-hei-nu vei’lo-hei avo-teinu...»

<<  HOME   |    CONTACTS
2016 www.dissertation.xlibx.info - Dissertations, online materials

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.