P2P File Sharing

What is a peer-to-peer file sharing system?

A P2P file sharing system distributes large files directly between participants, with no central server holding the content. Every downloader can also become an uploader, so aggregate bandwidth can grow with the number of participants instead of depending on one origin. The design centers on three questions: how peers find one another without a central directory, how upload incentives affect swarm health, and how a receiver verifies data arriving from an untrusted peer. The resulting system exercises DHT design, piece selection, choking algorithms, content addressing, and Merkle-tree integrity.

TL;DR

Create a content descriptor whose infohash identifies the exact metadata, discover peers through a tracker and optionally a Kademlia-style DHT, then download verified pieces from many peers over persistent wire connections. Rarest-first selection improves swarm availability, choking/unchoking provides an incentive to upload, and piece hashes or Merkle proofs reject corrupted data before assembly.

The tracker and DHT coordinate; they do not serve file bytes. The swarm is the data plane, so a single seeder can leave without destroying availability if other peers hold a complete copy. The detailed sections retain the tracker API, wire messages, DHT operations, bandwidth calculations, incentive alternatives, and v1/v2 integrity trade-offs.

Scope and assumptions

The main path covers public torrent-style sharing, a tracker, optional DHT discovery, direct peer transfers, piece verification, rarest-first selection, and basic upload incentives. Authentication/private trackers, traffic shaping, legal enforcement, and streaming playback are follow-up designs. The file sizes, peer counts, connection counts, block sizes, discovery times, and throughput figures below are illustrative protocol assumptions or benchmark targets; actual behavior depends on client implementation, network conditions, and the deployed protocol version.

Functional Requirements

Core Requirements

1. A user can add a file to the network and receive a shareable identifier (a magnet link or torrent file) that others can use to download it.
2. Any peer can download a file using only its identifier, without contacting the original uploader.
3. Files remain downloadable as long as at least one other peer holds a complete copy.
4. Every downloaded piece is verified for integrity; corrupted or tampered pieces are rejected and re-fetched.

Below the Line (out of scope)

• User authentication or private sharing (all shared content is treated as public).
• Rate limiting, traffic shaping, or ISP-level optimization (e.g., BitTorrent's LT-seeding and LEDBAT).
• Streaming playback before a download completes.
• Legal enforcement or content takedown mechanisms.

Authentication is below the line because the public protocol path is modeled without access control. An infohash identifies metadata and content; it does not prove that a user is authorized to download it. A private tracker would add authentication, authorization, and usually a per-user passkey at the tracker boundary.

Streaming is below the line because piece selection changes dramatically for video (sequential instead of rarest-first). Adding streaming support means running a parallel sequential-priority policy on the top N pieces while the rest of the torrent uses rarest-first. The tradeoff is covered in the piece selection deep dive.

Non-Functional Requirements

Core Requirements

• Availability: A torrent remains downloadable as long as at least one seeder is online. Target: 99.9% torrent availability for any torrent with 3+ seeders. There is no single point of failure in the content delivery path.
• Throughput: A peer downloading a popular file should saturate its uplink by receiving pieces from multiple peers simultaneously. Target is full bandwidth utilization (a 100 Mbps client downloads at 100 Mbps).
• Scale: The system must support millions of active torrents and tens of millions of simultaneous peers. The tracker or DHT must handle a sustained announce rate of hundreds of thousands of requests per second.
• Integrity: A corrupted or maliciously altered piece must be detected and discarded before assembly. Target: zero corrupted bytes written to disk; SHA-1 per-piece verification (BitTorrent v1 baseline) completes under 5 ms per 256 KB piece; SHA-256 per-block verification (BitTorrent v2) completes under 1 ms per 16 KB block on commodity hardware.
• Latency (peer discovery): A new peer should find its first set of peers for a torrent within 5 seconds of announcing.

Below the Line

• Sub-second peer discovery (requires centralized index or pre-seeded local DHT nodes).
• Global deduplication across torrents with identical content but different infohashes.

Full bandwidth utilization across millions of peers means a central content server becomes the bottleneck quickly. Serving 10 million concurrent peers at 100 Mbps each would require about 125 TB/s of outbound data. A tracker's only job is coordination (returning peer lists); the data transfer happens directly between peers.

Use this multiplication early to show why a central-server-only design misses the stated scale target and why the swarm must carry the data path.

Integrity at petabyte scale means we cannot rely on TLS or a trusted origin. Data arrives from anonymous strangers who may be misconfigured, malicious, or silently corrupting packets. Each piece must carry a self-certifying hash so any receiver can verify it independently.

30-second answer / outline

1. Hash a file into pieces and publish a torrent descriptor or magnet link whose infohash identifies the metadata.
2. Discover peers through a tracker, then use a DHT and peer exchange as additional discovery paths when decentralization is required.
3. Open several persistent peer connections, exchange bitfields, request blocks, and verify each completed piece before writing it as trusted data.
4. Use rarest-first selection to protect swarm availability and choking/optimistic unchoking to make upload capacity useful to other peers.
5. Make tracker/DHT state replaceable and keep the file itself in the swarm; handle NAT, churn, malicious peers, hash-version differences, and endgame retries explicitly.

5-minute explanation

The seeding client computes piece hashes and an infohash locally, so peers can agree on the exact content descriptor before exchanging bytes. A tracker returns a bounded peer sample; a DHT maps the infohash to peer contacts without requiring one central directory. Neither discovery mechanism is a content origin.

The downloader maintains a bitfield for each connection, requests blocks in parallel, and assembles a piece only after its expected hash or Merkle proof passes. Rarest-first requests protect pieces that have few remaining copies. Choking limits who receives upload capacity, while optimistic unchoking probes peers that might contribute useful bandwidth.

Reliability comes from many independent sources, resumable piece state, periodic announces, and fallback discovery. Integrity comes from self-certifying metadata and per-piece or per-block verification, not from trusting a peer or assuming transport encryption is enough. The deep dives compare tracker, PEX, and DHT discovery; piece-selection policies; incentive models; and flat versus Merkle-based integrity metadata.

45-minute interview approach

This is a time-boxed interview plan, not a promise that the article should be read in 45 minutes.

• 0–5 minutes — Clarify the contract: Confirm public versus private sharing, file-size range, streaming, anonymity, retention, seeding expectations, and whether tracker-only discovery is acceptable.
• 5–10 minutes — Establish scale: Use the illustrative torrent, peer, announce-rate, piece-size, discovery-latency, and client-bandwidth assumptions; ask how skewed the swarm popularity distribution is.
• 10–15 minutes — Define the protocol: Specify torrent/magnet metadata, infohash, tracker announce/scrape, peer handshake, bitfield, request/piece, and resume semantics.
• 15–22 minutes — Draw the data plane: Show seeding client, tracker/DHT, peer swarm, persistent connections, piece scheduler, disk, and verification gate.
• 22–30 minutes — Deep dive on discovery: Compare tracker redundancy, PEX, Kademlia lookup, bootstrap, reannounce, churn, NAT, rate limiting, and Sybil resistance.
• 30–35 minutes — Deep dive on transfer behavior: Explain rarest-first, endgame, choking/unchoking, parallel requests, backpressure, and the sequential-priority exception for streaming.
• 35–41 minutes — Deep dive on trust and reliability: Cover SHA-1 versus SHA-256/Merkle metadata, corrupted peers, resume after restart, incomplete swarms, privacy, and operational monitoring.
• 41–45 minutes — Trade-offs and close: Revisit centralized tracker simplicity versus DHT resilience, metadata size, seed retention policy, protocol compatibility, and the fact that discovery is not authorization.

Core Entities

• Torrent: The metadata bundle for a shared file, containing the infohash (SHA-1 of its info dictionary), file name, total size, piece length (fixed, typically 256 KB to 1 MB), and an ordered list of SHA-1 hashes for each piece.
• Piece: A fixed-size chunk of the file data, identified by its zero-based index within the torrent. Pieces are downloaded independently and verified against the torrent's piece hash list before assembly.
• Block: A sub-unit of a piece (typically 16 KB) used in the wire protocol. Peers request blocks, not full pieces. Once all blocks of a piece arrive, the piece is assembled and verified.
• Peer: A network participant identified by a 20-byte peer ID, IP address, and port. A peer is a leecher if its download is incomplete and a seeder if it holds the complete file.
• Swarm: The complete set of peers (seeders and leechers) currently sharing a particular torrent.
• Tracker: A centralized coordination server that maps an infohash to a list of peer IP/port pairs. Peers announce their presence to the tracker periodically.

The primary relationships: a Torrent has many Pieces. A Swarm belongs to one Torrent and contains many Peers. Each Peer maintains a bitfield (a bit vector, one bit per piece) indicating which pieces it currently holds.

Schema detail and DHT storage layout are deferred to the deep dives.

API Design

The system has two distinct protocol surfaces: the tracker HTTP protocol (used by peers to discover each other) and the peer wire protocol (used by peers to exchange pieces directly).

FR 1 and FR 2: Announce presence and discover peers (tracker)

GET /announce
  ?info_hash=<20-byte-urlencoded-SHA1>
  &peer_id=<20-byte-urlencoded-random>
  &port=6881
  &uploaded=0
  &downloaded=0
  &left=<bytes-remaining>
  &event=started | completed | stopped

Response (compact format):
{
  "interval": 1800,
  "peers": "<binary blob: 6 bytes per peer (4-byte IP + 2-byte port)>"
}

The tracker returns a bounded sample of peer addresses; 50 and an interval of 1800 seconds are illustrative protocol values in this article. The peer opens connections to those addresses and begins exchanging pieces. Use event=completed when the download finishes to tell the tracker the peer has become a seeder.

Supplemental (FR 1): Scrape (current swarm stats)

GET /scrape?info_hash=<20-byte-urlencoded-SHA1>

Response:
{
  "files": {
    "<info_hash>": {
      "complete": 142,
      "incomplete": 37,
      "downloaded": 9821
    }
  }
}

Scrape lets a client check swarm health before committing to a download. A swarm with zero seeders may still complete if its leechers collectively hold every piece, but at least one complete copy is the simplest availability invariant.

FR 3 and FR 4: Peer wire protocol (direct peer-to-peer)

The wire protocol is length-prefixed binary, not HTTP. Key message types:

// Peer wire protocol messages (simplified)

HANDSHAKE:
  <pstrlen=19><"BitTorrent protocol"><reserved 8 bytes>
  <info_hash 20 bytes><peer_id 20 bytes>

BITFIELD (after handshake):
  <length><0x05><bitfield>  // one bit per piece; 1 = peer has it

INTERESTED / NOT_INTERESTED:
  <length><0x02>  // "I want pieces you have"
  <length><0x03>  // "I no longer want pieces from you"

CHOKE / UNCHOKE:
  <length><0x00>  // "I will not upload to you"
  <length><0x01>  // "I will now upload to you"

REQUEST (download block from peer):
  <length><0x06><piece_index><block_offset><block_length>

PIECE (upload block to peer):
  <length><0x07><piece_index><block_offset><block_data>

HAVE (announce newly completed piece):
  <length><0x04><piece_index>