Minimizing metadata size for file synchronization using variable-length chunking

Ho Min Jung; Jae Min Park; Chang Geun Song; Sun Jeong Kim; Se Hak Chun; Young Woong Ko

doi:10.1007/978-94-007-4516-2_35

Minimizing metadata size for file synchronization using variable-length chunking

Ho Min Jung, Jae Min Park, Chang Geun Song, Sun Jeong Kim, Se Hak Chun, Young Woong Ko

Dept. of Business Administration

Hallym University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

A file synchronization program has to exchange lots of metadata information between a client and a server. As data have been growing rapidly in file synchronization systems, the metadata exchanging overhead is more critical now than before. In this paper, we propose a practical approach for minimizing metadata exchange overhead on the file synchronization system using variable-length chunking. The key idea is to minimize metadata by adapting file similarity information and generating only one hash key for several metadata. In the experiment, traditional file synchronization program sends 4.3 Mbyte over 500 MByte data file. However, the proposed algorithm only sends 37 Kbyte metadata.

Original language	English
Title of host publication	Future Information Technology, Application, and Service, FutureTech 2012
Pages	349-357
Number of pages	9
Edition	VOL. 1
DOIs	https://doi.org/10.1007/978-94-007-4516-2_35
State	Published - 2012
Event	7th FTRA International Conference on Future Information Technology, FutureTech 2012 - Vancouver, BC, Canada Duration: 26 Jun 2012 → 28 Jun 2012

Publication series

Name	Lecture Notes in Electrical Engineering
Number	VOL. 1
Volume	164 LNEE
ISSN (Print)	1876-1100
ISSN (Electronic)	1876-1119

Conference

Conference	7th FTRA International Conference on Future Information Technology, FutureTech 2012
Country/Territory	Canada
City	Vancouver, BC
Period	26/06/12 → 28/06/12

Keywords

Deduplication
File synchronization
Metadata
Variable-length chunking

Access to Document

10.1007/978-94-007-4516-2_35

Cite this

Jung, H. M., Park, J. M., Song, C. G., Kim, S. J., Chun, S. H., & Ko, Y. W. (2012). Minimizing metadata size for file synchronization using variable-length chunking. In Future Information Technology, Application, and Service, FutureTech 2012 (VOL. 1 ed., pp. 349-357). (Lecture Notes in Electrical Engineering; Vol. 164 LNEE, No. VOL. 1). https://doi.org/10.1007/978-94-007-4516-2_35

@inproceedings{58679149672c4586b19857b959c3e157,

title = "Minimizing metadata size for file synchronization using variable-length chunking",

abstract = "A file synchronization program has to exchange lots of metadata information between a client and a server. As data have been growing rapidly in file synchronization systems, the metadata exchanging overhead is more critical now than before. In this paper, we propose a practical approach for minimizing metadata exchange overhead on the file synchronization system using variable-length chunking. The key idea is to minimize metadata by adapting file similarity information and generating only one hash key for several metadata. In the experiment, traditional file synchronization program sends 4.3 Mbyte over 500 MByte data file. However, the proposed algorithm only sends 37 Kbyte metadata.",

keywords = "Deduplication, File synchronization, Metadata, Variable-length chunking",

author = "Jung, \{Ho Min\} and Park, \{Jae Min\} and Song, \{Chang Geun\} and Kim, \{Sun Jeong\} and Chun, \{Se Hak\} and Ko, \{Young Woong\}",

year = "2012",

doi = "10.1007/978-94-007-4516-2\_35",

language = "English",

isbn = "9789400745155",

series = "Lecture Notes in Electrical Engineering",

number = "VOL. 1",

pages = "349--357",

booktitle = "Future Information Technology, Application, and Service, FutureTech 2012",

edition = "VOL. 1",

note = "7th FTRA International Conference on Future Information Technology, FutureTech 2012 ; Conference date: 26-06-2012 Through 28-06-2012",

}

Jung, HM, Park, JM, Song, CG, Kim, SJ, Chun, SH & Ko, YW 2012, Minimizing metadata size for file synchronization using variable-length chunking. in Future Information Technology, Application, and Service, FutureTech 2012. VOL. 1 edn, Lecture Notes in Electrical Engineering, no. VOL. 1, vol. 164 LNEE, pp. 349-357, 7th FTRA International Conference on Future Information Technology, FutureTech 2012, Vancouver, BC, Canada, 26/06/12. https://doi.org/10.1007/978-94-007-4516-2_35

Minimizing metadata size for file synchronization using variable-length chunking. / Jung, Ho Min; Park, Jae Min; Song, Chang Geun et al.
Future Information Technology, Application, and Service, FutureTech 2012. VOL. 1. ed. 2012. p. 349-357 (Lecture Notes in Electrical Engineering; Vol. 164 LNEE, No. VOL. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Minimizing metadata size for file synchronization using variable-length chunking

AU - Jung, Ho Min

AU - Park, Jae Min

AU - Song, Chang Geun

AU - Kim, Sun Jeong

AU - Chun, Se Hak

AU - Ko, Young Woong

PY - 2012

Y1 - 2012

N2 - A file synchronization program has to exchange lots of metadata information between a client and a server. As data have been growing rapidly in file synchronization systems, the metadata exchanging overhead is more critical now than before. In this paper, we propose a practical approach for minimizing metadata exchange overhead on the file synchronization system using variable-length chunking. The key idea is to minimize metadata by adapting file similarity information and generating only one hash key for several metadata. In the experiment, traditional file synchronization program sends 4.3 Mbyte over 500 MByte data file. However, the proposed algorithm only sends 37 Kbyte metadata.

AB - A file synchronization program has to exchange lots of metadata information between a client and a server. As data have been growing rapidly in file synchronization systems, the metadata exchanging overhead is more critical now than before. In this paper, we propose a practical approach for minimizing metadata exchange overhead on the file synchronization system using variable-length chunking. The key idea is to minimize metadata by adapting file similarity information and generating only one hash key for several metadata. In the experiment, traditional file synchronization program sends 4.3 Mbyte over 500 MByte data file. However, the proposed algorithm only sends 37 Kbyte metadata.

KW - Deduplication

KW - File synchronization

KW - Metadata

KW - Variable-length chunking

UR - https://www.scopus.com/pages/publications/84867061999

U2 - 10.1007/978-94-007-4516-2_35

DO - 10.1007/978-94-007-4516-2_35

M3 - Conference contribution

AN - SCOPUS:84867061999

SN - 9789400745155

T3 - Lecture Notes in Electrical Engineering

SP - 349

EP - 357

BT - Future Information Technology, Application, and Service, FutureTech 2012

T2 - 7th FTRA International Conference on Future Information Technology, FutureTech 2012

Y2 - 26 June 2012 through 28 June 2012

ER -

Minimizing metadata size for file synchronization using variable-length chunking

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this