Data distribution phases for grid synchronization

Push phase

During the push phase, the goal is to distribute data to all targeted compute hosts.

The repository server generates a network-aware host list of target hosts that require the data set, and transfers data to the first host in the list. Each repository server agent is responsible for transferring the data to the next host within the list, until all hosts have received the data set. In the case of errors in the pipeline, repository servers also redirect repository server agents during the push phase.

The repository server agents listen for incoming data set push requests from the repository server or peer repository server agents. Each agent persists the data locally and sends the data to the next host in the list.

Pull phase

During the pull phase, individual compute hosts that are missing the data set (for example, because of errors, or resource group changes) can retrieve the data after the push phase ends.

The repository server sends instructions to an individual host specifying the data set, source host, and port. In turn, the repository server agents connect to peers to retrieve the data set. The repository server agents also listen for incoming data set pull requests from peer repository server agents. Pull actions are always between repository server agents, never from the repository server directly.

Delete phase

During the delete phase, the repository server sends delete instructions to repository server agents to delete an entire data set, or specific files or directories within the data set. Repository server agents remove the local copy of the data set (or files or directories within the data set). There is no chaining at this phase.