As our reliance on the Internet and networking applications grows, effective monitoring of network infrastructure becomes crucial for reliable operation. Current systems for monitoring large networks suffer from significant lack of visibility into problems, or do not scale well to processing large amounts of network data.
This talk is about the design of network infrastructure to provide high-accuracy diagnostics while reducing the overhead of data processing. First, I will discuss Marple, a performance query system, which allows network operators to query fine-grained flow-level statistics over network traffic. Marple leverages a novel one-way cache to generate accurate query results from a hardware key-value store running on switches. Then, I will discuss a path query system that enables network operators to monitor exactly those packets that follow paths of interest through the network. These two systems, supported by declarative query languages, turn a network into a database that provides insights into its own operation.