Listing and copying data

Edit online

Listing and copying data uses a producer/consumer model: list operations produce content that the copy operations use to initiate copies.

It prevents delays in copying files that could occur while waiting for either a list or a copy to complete first before starting a new copy or list.

  1. A List operation generates a list of the first 1,500 slice names that are found in the smaller namespaces that are stored in the tracking file.
    image
  2. This slice list is put into a queue in memory.
    image
  3. Each copy operation pulls a single slice from the first slice list in the queue.
    image
  4. If slices are left in the list, the slice list is reinserted at the end of the queue.
    image
  5. The slice read and remote-write operations happen asynchronously.
  6. After an entire slice list is completed copying successfully, the range in the namespace housing the slice list is truncated and the tracking file is updated.
    image
  7. After an entire small namespace is completed, it is removed from the tracking file.
    image

The number of bytes transferred is also maintained in the tracking file for reporting purposes. A progress bar is visible on the Source Slicestor device and the Destination Slicestor device pages on the Manager Web Interface.

The combination of randomized namespace ranges and jumping between list requests results in highly parallel and uniform reading from all data disks.