Optimizing Large Fileset Transfers in the HotFolder
Posted by Aly Essa, Last modified by Aly Essa on 05 March 2019 02:11 PM


FileCatalyst HotFolder tasks take several minutes to begin transferring files when large filesets (100K+ files and folders) are used.

FileCatalyst HotFolder maintains a file attribute cache with a default maximum capacity of 100000. The attribute cache alleviates the need for repeated expensive calls to retrieve file attributes from the file system. These calls are particularly expensive when the files are located on a network drive (i.e. SMB).

The attribute cache is populated with values from all files FileCatalyst HotFolder touches, from all tasks, including their associated transfer cache index files. When dealing with file sets that exceed the default maximum size of 100000, the attribute cache will quickly reach capacity, and previously cached values will be discarded using an LRU (least recently used) algorithm. This can result in very slow scanning of files (2-3 seconds per 1000 files), and add delays to task startup and process further filtering options before the actual file transfer can begin.


FileCatalyst HotFolder v3.7.3 and earlier


To accommodate the attributes of the entire file set, the eventual transfer cache size, and application specific files, the maximum size of the attribute cache should be tuned to roughly 2-2.5X the number of files to be transferred (for all tasks). This can be accomplished by adding a System Property in the HotFolder.

  1. From the HotFolder Remote Admin navigate to the Settings tab and look for System Properties on the top row.
  2. Select the Add Property button and enter the following:

    Property Name: unlimited.fc.fastio.max_entry_count
    Value: <max cache size>

    As mentioned above the max cache value should be at least 2x - 2.5x the total size of the fileset. For example, if the fileset to be transferred contains 100000 files/folders, this system property should be set to at least 200000.

  3. If symlinks are not needed, an additional System Property must be added:

    Property Name: unlimited.fc.follow_symlinks
    Value: false 

    When running HotFolder on Windows, this can significantly increase performance beyond simply increasing the attribute cache size.

  4. Once all the properties have been added hit Apply and restart the HotFolder Service.