Google File System Review
Application driven design
Issue:
Current distributed file system cannot satisfy the need of Google workload.
Solution & Characteristics
-very large scale
-inexpensive components
-workload dominated by read
-support multiple concurrent append (for map reduce)
-large chunk 64MB, chunk replicated (-anti-crash, balance workload)
-directories are an illusion, /a/b/foo take the whole string, the string does not matter, to like hashtable to find the file, this only useful for small amounts of files, large amounts of files require large hashtable which may not be able to fit into memory so Unix FS does not use it.
-Architecture -master (all the requests start from master); -chunk server(using Unix FS, every it just creates a file); clients
-Atomic Record Append
Advantages,
-Reduce client's need to interact with server (master)
-Reduce network overhead, many operations on the same chunk
-Reduce the size of metadata stored on master (it could be stored memory for faster access)
Disadvantages:
-If a small file has only one chunk stored on one chunkserver, it may become hot spot due to multiple access to the same file
-Another big issue for large block size is internal fragmentation (write a block which is less the block size to the disk will cause fragmentation inside block ), the solution is lazy space allocation which will first reserve the space for the block and perform writes afterwards(I think it's waiting for data to occupy the entire block size)
- master may become a bottleneck, though it only handles query, not data transfer. Later they user multiple masters to handle this problem/bottleneck
I think Google File System is for google crawler which crawls large amount of data to the file system, then another application will index all the content for user search. It's not completed in one step. My thinking before class is very close, they indeed use this to do analysis but for like Map Reduce, the crawled data may also be stored there.
In-Class Note
Hadoop Distributed File System is the open source counterpart of GFS.
Map Reduce: Used to do analysis, map reduce put data into system like GFS/HDFS
"The trend is NOSQL", database is not able to do that much data. Facebook is using HDFS to do analysis, but the main profiles are kept in database.
Big table:
"Database is heavyweight compared to Big Table" You can do query on top of big table. "Only have Key-Value mapping while database has extra things such as schema, profile photo,id,name,age"
GFS,HDFS do not do query, they only do computation.
(Deign file system for multi-media?)
Consistence: all replica have the same data
Defined: reflects consistence and mutation (do exactly as what clients ask for for example, user asks to write 1, all replica will write 1). It's stronger condition.
Issue:
Current distributed file system cannot satisfy the need of Google workload.
Solution & Characteristics
-very large scale
-inexpensive components
-workload dominated by read
-support multiple concurrent append (for map reduce)
-large chunk 64MB, chunk replicated (-anti-crash, balance workload)
-directories are an illusion, /a/b/foo take the whole string, the string does not matter, to like hashtable to find the file, this only useful for small amounts of files, large amounts of files require large hashtable which may not be able to fit into memory so Unix FS does not use it.
-Architecture -master (all the requests start from master); -chunk server(using Unix FS, every it just creates a file); clients
-Atomic Record Append
Advantages,
-Reduce client's need to interact with server (master)
-Reduce network overhead, many operations on the same chunk
-Reduce the size of metadata stored on master (it could be stored memory for faster access)
Disadvantages:
-If a small file has only one chunk stored on one chunkserver, it may become hot spot due to multiple access to the same file
-Another big issue for large block size is internal fragmentation (write a block which is less the block size to the disk will cause fragmentation inside block ), the solution is lazy space allocation which will first reserve the space for the block and perform writes afterwards(I think it's waiting for data to occupy the entire block size)
- master may become a bottleneck, though it only handles query, not data transfer. Later they user multiple masters to handle this problem/bottleneck
I think Google File System is for google crawler which crawls large amount of data to the file system, then another application will index all the content for user search. It's not completed in one step. My thinking before class is very close, they indeed use this to do analysis but for like Map Reduce, the crawled data may also be stored there.
In-Class Note
Hadoop Distributed File System is the open source counterpart of GFS.
Map Reduce: Used to do analysis, map reduce put data into system like GFS/HDFS
"The trend is NOSQL", database is not able to do that much data. Facebook is using HDFS to do analysis, but the main profiles are kept in database.
Big table:
"Database is heavyweight compared to Big Table" You can do query on top of big table. "Only have Key-Value mapping while database has extra things such as schema, profile photo,id,name,age"
GFS,HDFS do not do query, they only do computation.
(Deign file system for multi-media?)
Consistence: all replica have the same data
Defined: reflects consistence and mutation (do exactly as what clients ask for for example, user asks to write 1, all replica will write 1). It's stronger condition.
评论