Download

Overview

DownloadTask downloads files from remote FTP or SFTP servers to local directories. It supports parallel downloading, file archiving on the remote server, deletion after transfer, and checked transfers to ensure files are not actively being written before download.

Use this task when: You need to retrieve data files from external partners, vendors, or remote servers via FTP/SFTP protocols.

Don't use this task when:

• Downloading files via HTTP/HTTPS (use a web download client instead)
• Accessing cloud storage like S3 or Azure Blob Storage (use cloud-specific clients)
• Files are already on the local filesystem (use file copy operations instead)

Parameters

Execution Flow

Processing Algorithm

The download process uses a multi-threaded approach to efficiently transfer multiple files:

1. File Discovery: Connects to the remote server and searches the RemoteDirectory for files matching RemoteFileName (supports wildcards)
2. Queue Creation: All matching files are added to a download queue
3. Concurrent Download: Up to MaxDownloaders threads run in parallel, each pulling files from the queue
4. Checked Transfer (optional): If UseCheckedTransfer = true, each file is checked multiple times to ensure its size remains constant (indicating the write is complete) before downloading
5. Post-Transfer Actions: After successful download, files can be deleted or moved to an archive directory on the remote server
6. Result Tracking: The task tracks succeeded and failed transfers, logging detailed results

Example Execution

Scenario: Downloading 100 CSV files from a vendor SFTP server with 10 concurrent downloaders:

Configuration:
- MaxDownloaders = 10
- RemoteFileName = "*.csv"
- RemoteDirectory = "/outbound/claims"
- UseCheckedTransfer = true

Execution:
1. Connect to SFTP server
2. Find 100 files matching "*.csv" in "/outbound/claims"
3. Add all 100 files to download queue
4. Start 10 downloader threads
5. Each thread:
   - Pull next file from queue
   - Check file size (wait if still changing)
   - Download file to LocalDirectory
   - Move file to RemoteArchiveDirectory
   - Process next file from queue
6. Complete when queue is empty
7. Log: "Completed. 100 Succeeded. 0 Failed."

Timeline: With 3KB files and stable network, approximately 10-30 seconds total

Usage Examples

Example 1: Basic SFTP download with archiving

// Download all CSV files from vendor and archive them on remote server
var task = new DownloadTask
{
    Host = "vendor.sftp.example.com",
    Port = 22,
    Username = "mdclarity_user",
    Password = "secure_password",
    Protocol = ProtocolSubset.Sftp,
    RemoteDirectory = "/outbound/data",
    RemoteFileName = "*.csv",
    LocalDirectory = @"C:\MDClarity\Downloads\Vendor",
    RemoteArchiveDirectory = "/archive/processed",
    // Files will be moved to /archive/processed after download
};

Example 2: Download specific file and delete after success

var task = new DownloadTask
{
    Host = "partner.ftp.example.com",
    Port = 21,
    Username = "integration_user",
    Password = "password123",
    Protocol = ProtocolSubset.Ftp,
    RemoteDirectory = "/reports",
    RemoteFileName = "daily_claims_[id].txt",  // Specific filename
    LocalDirectory = @"C:\MDClarity\Reports",
    DeleteOnSuccess = true,  // Remove from remote server after download
    ConnectionTimeoutInSeconds = 60
};

Example 3: High-volume download with checked transfer

// Download many files with parallel transfers, ensuring files are complete before downloading
var task = new DownloadTask
{
    Host = "data.sftp.example.com",
    Username = "mdclarity_prod",
    Password = "prod_password",
    Protocol = ProtocolSubset.Sftp,
    RemoteDirectory = "/exports/visits",
    RemoteFileName = "visit_*.dat",  // Pattern matching multiple files
    LocalDirectory = @"C:\MDClarity\Imports\Visits",
    MaxDownloaders = 15,  // Allow up to 15 concurrent downloads
    UseCheckedTransfer = true,  // Wait for files to finish writing before download
    RemoteArchiveDirectory = "/exports/archive"
};

Example 4: Download all files from a directory

var task = new DownloadTask
{
    Host = "backup.sftp.internal",
    Username = "backup_user",
    Password = "backup_pass",
    Protocol = ProtocolSubset.Sftp,
    RemoteDirectory = "/daily_exports",
    RemoteFileName = null,  // null or omitted = download all files in directory
    LocalDirectory = @"C:\MDClarity\Backups\Daily",
    MaxDownloaders = 5  // Conservative for potentially large files
};

Related Tasks Comparison

Key Differences:

• Download vs Upload: Download retrieves files from remote to local; Upload sends files from local to remote
• Download vs LoadDataFile: Download retrieves files; LoadDataFile imports them into database tables (often used sequentially)
• Download vs AddFilesToQueue: Download transfers files; AddFilesToQueue creates processing queue entries for existing files

Performance Considerations

Network Impact

• Bandwidth: Downloads are limited by network bandwidth and server capacity
• Concurrent Connections: MaxDownloaders controls parallelism (more threads = faster for many small files, but may overwhelm network or remote server)
• Connection Timeout: ConnectionTimeoutInSeconds should be increased for slow or unreliable connections
• Checked Transfer: UseCheckedTransfer = true adds latency (rechecks every 10ms up to 3 times) but prevents downloading incomplete files

Optimization Tips