Copying files from DropBox to IBM Cloud Object Storage on schedule

Creating the flow from a template

If you use a template, you can create the flow immediately, then select each node in the flow to validate the node and select the application accounts that you want to use. Search for the template name on the Templates tab.

The template name is "Weekly copy Dropbox files to new Cloud Object Storage S3 bucket named with date".

To continue with the template, see the template instructions in Github (opens in a new window)

After you create a flow from the template, you can continue to test your flow.

Creating the flow from scratch

1. On the catalog tab, connect App Connect to your Dropbox and IBM Cloud Object Storage accounts. (You can also connect to applications in the flow editor.)

   The following steps assume that you used the catalog to connect App Connect to your Eventbrite, Sharepoint, and Slack accounts.

2. From the Designer dashboard, click Create > Event-driven flow.

   Complete the following steps. As you progress, App Connect automatically saves your changes. If you move away from the flow at any stage, the flow is saved as a draft flow that you can complete at another time.

3. Enter a name that identifies the purpose of your flow; for example: Weekly copy Dropbox files to new COS S3 bucket named with date.
4. Configure a scheduler as the event that is to trigger the flow once a week.
   In this example, the flow runs every week, on Friday at 8 PM GMT. To test the flow, you can set the flow to run when it is first started.

   1. Click the Toolbox tab to open the Toolbox.
   2. Click Scheduler.
   3. Click Select schedule type and select the Calendar option.
   4. Set the Run every options to Week on FRI at 20:00.
   5. Set Timezone to Etc/GMT.
   6. Select Also run the flow when it's first switched on.

   

5. Click the (+) and then scroll down to Dropbox.
6. Select Folders > Retrieve all folders as the action.

   Actions on folders or files in Dropbox are by ID, which is equivalent to the file name. The "Retrieve all folders" action returns details of all folders (as a JSON object) so that App Connect can map from a folder name to its ID.

   If your Dropbox account has more than 10 folders, configure the Maximum number of items to retrieve option. You can also change the options if the maximum number of items (folders) is exceeded, or if no items are found.

7. Use a variable of path_lower"="/shared to handle the ID of the one folder that you want to process. (You don't need to use a variable, but a variable with a meaningful name can be useful instead of using a property reference and expression in multiple places.)
   1. Click Add node (+), then open the Toolbox tab.
   2. Select Set variable.
   3. Click Add property and enter the name FolderID.
   4. Click Edit mappings, then map the ID field from the Dropbox "Retrieve folders" action. For example, type id, then select ID from under foldercollection.

      The "Retrieve folders" action returns an array of data for one or more folders (as indicated by the message under the FolderID field). Therefore, use a JSONata function to get the ID of the folder with the path_lower"="/shared variable.

   5. In the FolderID field, click the ID mapping, and from the menu, click Edit expression. This field shows the JSONata expression that is used to get ID values for all folders as an array.

      {{$DropboxRetrieveallfolders.id}}

      To restrict the data to the folder with the path_lower"="/shared variable, change the expression to the following example.

      {{$DropboxRetrieveallfolders[path_lower="/shared"].id}}

   

8. Click Add node (+), then scroll down to Dropbox, to add the action to retrieve all the files in the folder to be processed.
9. Select Files > Retrieve all files as the action.
10. Filter the action to get files from the folder with the ID that you identified previously.
    1. Click Add condition.
    2. Click in the field after equals, then type folderif.
    3. From the menu that opens, select the variable FolderID.
11. Click Add node (+), then scroll down to IBM Cloud Object Storage S3.
12. Select IBM Cloud Object Storage S3 > Create bucket as the action to create a bucket that is based on the date when the action is completed.
    Configure the action as follows.

    1. Click in the Bucket name field.
    2. Click the Insert a function icon.
    3. From under Date/time functions, select $now().
    4. Configure the $now() function and add text or field mappings for the name that you want to use for the Cloud Object Storage S3 bucket. The following example adds text and edits the $now() function to change the format of the date in the bucket name.

       
       dearliza-{{$now('[Y0001]-[M01]-[D01]') }}
       

       For more information, see the JSONata documentation.

    

13. Add a For each loop to process each file from the Dropbox Retrieve all files action.
    1. Click Add node (+), then open the Toolbox tab.
    2. Select For each.
    3. Click in the Select the collection of items to process field.
    4. Type filecoll, then from the menu that opens, click filecollection from under the Retrieve files action.

       Use the default values of the options to process all items (files) in the collection sequentially and to continue processing other files if an error occurs during file processing.

       

    5. In the For each node, configure what happens for each Dropbox file.
       This example adds an If node to take different actions that depend on the suffix of the Dropbox file. (Dropbox doesn't provide different Content-Type values for different types of files.)

       • If a file name contains .mp3 or .pdf, handle this file as a binary file.
         1. Use the Dropbox Download file content action to get the content of this file by using its file path and a file type of binary.
         2. Use the IBM Cloud Object Storage S3 Create object action to create an object in the selected bucket with the file name of this file and the content from Dropbox.
       • If a file name doesn't contain .mp3 or .pdf, handle the file as a text file.
         1. Use the Dropbox Download file content action to get the content of this file by using its file path and a file type of text.
         2. Use the IBM Cloud Object Storage S3 Create object action to create an object in the selected bucket with the file name of this file and the content from Dropbox. Set the content-type to text/plain.

       

       1. In the For each node, click Add node (+), click the Toolbox tab, and then click If (conditional).
       2. In the if field, type name, then select Name from under filecollection.
       3. Click Name, then select Apply a function.
       4. From under String functions, select $contains.
       5. In the $contains expression, change the missing chars with .mp3. This value completes the expression as follows.

          {{$contains($Foreachitem.name, ".mp3")}}

          (You can paste this string into the if field.)

       6. Change the equals condition to an is true condition.
       7. Add another condition for .pdf files. Click Add condition, then use the same steps as for .mp3 files. For example, add the following expression.

          {{$contains($Foreachitem.name, ".pdf")}}

       8. Next to the if label, change the condition to Any of the following are true.
14. Configure the actions for the If branch of the For each node.
    1. In the If branch of the For each node, click Add node (+).
    2. Scroll down to Dropbox and select Files > Download file content.
    3. Click in the Path field, type path, then select Path lower from under For each … filecollection.
    4. Set File Type to Binary.
    5. In the If branch of the For each node, click the next Add node (+) icon.
    6. Scroll down to IBM Cloud Object Storage S3 and select Objects > Create object.
    7. Configure the Create object action.
       • Bucket: Typically, you select an existing bucket from the list, but for this flow, you need to specify a bucket that doesn't yet exist. To specify this bucket, use advanced mode.
         1. Click the Map options menu (⋮) and select Switch to advanced mode.
         2. Click in the Bucket field and enter the same name that you used on the Create bucket action. (You can copy the value from the Bucket name field of that action).

            dearliza-{{$now('[Y0001]-[M01]-[D01]') }}

       • Object name: Click in the field and insert a mapping to the For each > filecollection > Name value.
       • Object content: Click in the field and insert a mapping to the Dropbox > Download file content > File content value.
       • Leave the Content type field empty.
15. Configure the actions for the Else branch of the For each node.
    1. In the If branch of the For each node, click Add node (+).
    2. Scroll down to Dropbox and select Download file content.
    3. Click in the Path field, type path, then select Path lower from under For each … filecollection.
    4. Set the File Type to Text.
    5. Click the next Add node (+) icon in the If branch of the For each node.
    6. Scroll down to IBM Cloud Object Storage S3 and select Objects > Create object.
    7. Configure the Create object action.
       • Bucket: Typically, you select an existing bucket from the list, but for this flow, you need to specify a bucket that doesn't yet exist. To specify this bucket, use advanced mode.
         1. Click the Map options menu (⋮) and select Switch to advanced mode.
         2. Click in the Bucket field and enter the same name that you used on the Create bucket action. (You can copy the value from the Bucket name field of that action.

            
            dearliza-{{$now('[Y0001]-[M01]-[D01]') }}

       • Object name: Click in the field and insert a mapping to the For each > filecollection > Name value.
       • Object content: Click in the field and insert a mapping to the Dropbox > Download file content 2 > File content' value. (Make sure that you select File content for the Download file content action on the Else branch.)
       • Set the Content type to text/plain.