System Design Interview

Wrap Up

Wrap up 1 min readLesson 5 of 5

In this chapter, we described the high-level design for S3-like object storage. We compared the differences between block storage, file storage, and object storage.

The focus of this interview is on the design of object storage, so we listed how the uploading, downloading, listing objects in a bucket, and versioning of objects are typically done in object storage.

Then we dived deeper into the design. Object storage is composed of a data store and a metadata store. We explained how the data is persisted into the data store and discussed two methods for increasing reliability and durability: replication and erasure coding. For the metadata store, we explained how the multipart upload is executed and how to design the database schema to support typical use cases. Lastly, we explained how to shard the metadata store to support even larger data volume.

Congratulations on getting this far! Now give yourself a pat on the back. Good job!

Chapter Summary

Reference Material

  1. Fibre channel: https://en.wikipedia.org/wiki/Fibre_Channel
  2. iSCSI: https://en.wikipedia.org/wiki/ISCSI
  3. Server Message Block: https://en.wikipedia.org/wiki/Server_Message_Block
  4. Network File System: https://en.wikipedia.org/wiki/Network_File_System
  5. Amazon S3 Strong Consistency: https://aws.amazon.com/s3/consistency/
  6. Serial Attached SCSI: https://en.wikipedia.org/wiki/Serial_Attached_SCSI
  7. AWS CLI ls command: https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html
  8. Amazon S3 Service Level Agreement: https://aws.amazon.com/s3/sla/
  9. Ambry: LinkedIn’s Scalable Geo-Distributed Object Store: https://assured-cloud-computing.illinois.edu/files/2014/03/Ambry-LinkedIns-Scalable-GeoDistributed-Object-Store.pdf
  10. inode: https://en.wikipedia.org/wiki/Inode
  11. Ceph’s Rados Gateway: https://docs.ceph.com/en/pacific/radosgw/index.html
  12. grpc: https://grpc.io/
  13. Paxos: https://en.wikipedia.org/wiki/Paxos_(computer_science)
  14. Raft: https://raft.github.io/
  15. Consistent hashing: https://www.toptal.com/big-data/consistent-hashing
  16. RocksDB: https://github.com/facebook/rocksdb
  17. SSTable: https://www.igvita.com/2012/02/06/sstable-and-log-structured-storage-leveldb/
  18. B+ tree: https://en.wikipedia.org/wiki/B%2B_tree
  19. SQLite: https://www.sqlite.org/index.html
  20. Data Durability Calculation: https://www.backblaze.com/blog/cloud-storage-durability/
  21. Rack: https://en.wikipedia.org/wiki/19-inch_rack
  22. Erasure Coding: https://en.wikipedia.org/wiki/Erasure_code
  23. Reed–Solomon error correction: https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction
  24. Erasure Coding Demystified: https://www.youtube.com/watch?v=Q5kVuM7zEUI
  25. Checksum:https://en.wikipedia.org/wiki/Checksum
  26. Md5: https://en.wikipedia.org/wiki/MD5
  27. Sha1: https://en.wikipedia.org/wiki/SHA-1
  28. Hmac: https://en.wikipedia.org/wiki/HMAC
  29. TIMEUUID: https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/timeuuid_functions_r.html

Finished reading?

Mark it complete to track your progress.

PreviousDesign Deep DiveS3-like Object StorageNext Understand the ProblemReal-time Gaming Leaderboard