Knowledgebase: Advisories
HotFolder Progressive and Delta Features Cause MD5 Failures
Posted by Aly Essa, Last modified by Aly Essa on 20 December 2016 07:30 AM

Overview

There are two transfer features in the FileCatalyst HotFolder which have shown some issues when used simultaneously. The Progressive feature allows the HotFolder to transfer the file as it grows. The Incremental feature will limit file transfers to files that are new or have been modified. The Delta of a file is defined as portions of the original file that have been modified or changed. Depending on the Incremental setting selected, either the whole file or File Deltas will be sent to the receiver. More information on the Incremental and Advanced Progressive features are available in the HotFolder documentation.

By default, the Basic Progressive feature is enabled in HotFolder tasks. When Incremental and Progressive options are enabled, the transfer will generate an index of what chunks of the file have been modified. As the size of the original and changed file increases the process of calculating the File Delta takes longer to complete. Factors that can increase the time it takes to process the file include low read speeds from the source drive and low write speeds to the destination drive and insufficient system resources such as CPU and memory allocation.

The problem that we have identified occurs when there are large sized Delta files that need to be transferred and the amount of time that is required to move them and reconstruct the file on the receivers end is not sufficient. In most of these cases, only a portion of the file gets sent and the Delta file is not created fast enough on senders side.

This guide provides a solution for Progressive file transfers without the use of Incremental (Deltas) and Progressive Transfers with the use of Basic Progressive. There are signals sent from the receiver to trigger the next batch of files and by that time this request is processed the read timeout kicks in and terminates the transfer.

Environment

FileCatalyst HotFolder v3.5 to v3.7

Resolution

  1. Dynamic Files Options

    This solution has to be deployed by using the MultiClient option in the HotFolder Task Settings. The Advanced Progressive feature will wait for 5-seconds and this value can be raised in 5-second intervals by modifying step f outlined below. This value can not be predetermined and will be different from one network to another. Use the following steps to deploy a 5-second delay in the transfer:


    1. Open FileCatalyst HotFolder.
    2. Click on the Scheduler Tab on the left and side and select your task. Hit the Edit button above it to access the task settings.
    3. From the Transfer Tab at the top make sure Enable Multi-Client and select the number of MultiClient Connections you would like to use.
    4. Hit the Apply button and then navigate to the Dynamic Files tab.
    5. Select the Advanced Progressive radial.
    6. Check the option that will Wait for file growth to cease. Enter 5 (seconds) for the Wait value.
    7. Hit Ok to save your changes.

  2. HotFolder System Property

    The System Property forces the transfer to delay to 5-seconds after each incremental transfer of the Growing Delta File, the receiver immediately checks if the file has grown while the transfer was taking place. Without the System Property, the transfer would stop and fail if the Delta File hadn't grown (would fail because the Delta File wasn't actually done yet). With the System Property, instead of terminating the transfer, the receiver will wait for 5-seconds and check again. The 5-second delay will automatically add at least 5 seconds to every progressive transfer, and more if the incremental transfers lead to mid transfer delays. So choosing the delay time can have impacts on performance, but choosing a delay that is too short will cause the delta transfers to fail.

    A very important note to mention if you choose to deploy this solution: this feature impacts all progress transfers by delaying their completion by the defined number of seconds. When using this System Property the HotFolder application will accept two types of input, one is an integer which will reflect the Wait Time in Seconds or a boolean value of True which will default the Wait Time to 5-seconds. You can only use one of these two inputs for the System Property. 

    Follow these steps to properly deploy the System Property in the FileCatalyst HotFolder:

    1. Open the HotFolder Control Panel.
    2. Click on Settings on the left side and navigate to the System Properties tab at the top.
    3. Select the Add Property button.
    4. Here are some examples of delay values you can use:

      1. Default 5-Second Delay:

        Property Name: unlimited.fc.progressiveDelayBeforeAppending
        Value: True

      2. 10-Second Delay:

        Property Name: unlimited.fc.progressiveDelayBeforeAppending
        Value: 10

      3. 15-Second Delay:

        Property Name: unlimited.fc.progressiveDelayBeforeAppending
        Value: 15 

      4. After you have entered your delay value click on OK to close the dialog and hit the Apply button to save these changes to your configuration file.

Comments (0)