Just-in-time analytics on large file systems

US9244975B2 · kind B2 · utility

Assignees

• The George Washington University · US
• The Board of Regents of the University of Texas System · US
• THE JOHNS HOPKINS UNIVERSITY · US

Inventors

• Gautam Das · Irving, US
• Hao Huang · Kenmore, US
• Sandor Szalay · Baltimore, US
• Nan Zhang · San Diego, US

Key dates

• Filing date: Dec 16, 2011
• Grant date: Jan 26, 2016
• Expiry date: Dec 16, 2031

Classification

• Technology area (CPC G): Physics
• CPC primary: G06F16/2453
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

As file systems reach the petabytes scale, users and administrators are increasingly interested in acquiring high-level analytical information for file management and analysis. Two particularly important tasks are the processing of aggregate and top-k queries which, unfortunately, cannot be quickly answered by hierarchical file systems such as ext3 and NTFS. Existing pre-processing based solutions, e.g., file system crawling and index building, consume a significant amount of time and space (for generating and maintaining the indexes) which in many cases cannot be justified by the infrequent usage of such solutions. User interests can often be sufficiently satisfied by approximate (i.e., statistically accurate) answers. A just-in-time sampling-based system can, after consuming a small number of disk accesses, produce extremely accurate answers for a broad class of aggregate and top-k queries over a file system without the requirement of any prior knowledge. The system is efficient, accurate and scalable.