Performance Comparison of C#, Java, and Python for Processing Compressed UniProt Data Files

Efficient processing of large compressed datasets, such as UniProt’s Swiss-Prot (uniprot_sprot.dat.gz) and TrEMBL (uniprot_trembl.dat.gz), is critical in bioinformatics. These files differ significantly in size, making performance optimization essential when reading or processing them.

---

Data Files

1. Swiss-Prot (uniprot_sprot.dat.gz):
   • Compressed size: 641 MB
2. TrEMBL (uniprot_trembl.dat.gz):
   • Compressed size: 174 GB

The TrEMBL file is approximately 271 times larger than Swiss-Prot, emphasizing the critical importance of choosing the right tools and languages to handle this scale efficiently.

---

What Was Tested

The benchmark focused solely on reading the files line by line without performing any data processing. This means the actual computational cost of transforming, analyzing, or extracting data from each line was not included. Real-world processing—such as parsing, filtering, or applying algorithms—will undoubtedly slow down the performance further, potentially by a significant but unknown factor, depending on the complexity of the task.

---

Performance Results for Swiss-Prot File

Language	Elapsed Time	Compressed File Size Processed	Processing Speed (MB/sec)
Python	20.19 seconds	641 MB	~31.7 MB/sec
Java	11.21 seconds	641 MB	~57.2 MB/sec
C#	5.99 seconds	641 MB	~107.1 MB/sec

---

Extrapolated Results for TrEMBL File

Based on the observed performance for Swiss-Prot, the estimated times for processing the 174 GB TrEMBL file (compressed) are:

Language	Estimated Time	Compressed File Size
Python	~5,490 seconds (~91.5 minutes)	174 GB
Java	~3,040 seconds (~50.7 minutes)	174 GB
C#	1,626 seconds (~27.1 minutes)	174 GB

---

Critical Observations

1. Reading vs. Processing:
   • These results measure the raw reading speed—line-by-line reading without additional processing.
   • In real-world use cases, actual data processing or transformation tasks (e.g., parsing, filtering, and aggregating) will slow down execution by an unknown factor, potentially making the differences between languages even more pronounced.
2. C# Dominates:
   • Completing the TrEMBL file read in just ~27 minutes, C# demonstrates superior performance.
   • When processing tasks are added, C#’s performance advantage will likely amplify due to its efficient runtime and superior handling of computational workloads.
3. Java is Respectable but Lags:
   • At ~51 minutes, Java performs well for reading, but when complex processing is introduced, it may lag further behind C# due to its higher resource demands and slower runtime optimizations.
4. Python is a Bottleneck:
   • Python’s 91-minute read time makes it an impractical choice for massive datasets. When processing tasks are included, Python’s inefficiency will likely become a limitation, making it unsuitable for serious bioinformatics workflows at scale.

---

Conclusion

This analysis highlights the significant differences in performance between C#, Java, and Python when processing large compressed datasets like UniProt’s Swiss-Prot and TrEMBL files. While the benchmark focused solely on line-by-line reading without additional processing, it’s important to note that actual data processing tasks will inevitably slow down execution by an unknown factor, further emphasizing the need for efficient file handling.

• C# demonstrates the fastest performance, making it the most suitable choice for handling massive files like TrEMBL, with its efficient I/O handling and optimized runtime.
• Java offers good performance and remains a strong contender, particularly for cross-platform environments, though it lags behind C#.
• Python, while easy to use, is significantly slower and becomes impractical for large-scale datasets like TrEMBL, especially when additional processing is involved.

Code Implementations

Python

import gzip
from timestopper import TimeStopper

gz_file_path = "path/to/uniprot_sprot.dat.gz"

try:
    with gzip.open(gz_file_path, "rt", encoding="utf-8") as gz_file:
        ts = TimeStopper()
        ts.start()
        line_count = 0
        for line in gz_file:
            line_count += 1  # No processing, just reading
        elapsed_time = ts.get_elapsed_time_seconds()
        print(f"Read {line_count} lines in {elapsed_time:.2f} seconds.")
except Exception as e:
    print(f"An error occurred: {e}")

Java

import java.io.*;
import java.util.zip.GZIPInputStream;

public class GZFileReaderBuffered {
    public static void main(String[] args) {
        String gzFilePath = "path/to/uniprot_sprot.dat.gz";

        try (
            FileInputStream fileInputStream = new FileInputStream(gzFilePath);
            GZIPInputStream gzipInputStream = new GZIPInputStream(fileInputStream);
            BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(gzipInputStream), 8192 * 1000)
        ) {
            long startTime = System.nanoTime();
            int lineCount = 0;
            while (bufferedReader.readLine() != null) {
                lineCount++;  // No processing, just reading
            }
            long endTime = System.nanoTime();
            double elapsedTime = (endTime - startTime) / 1e9;
            System.out.println("Elapsed time: " + elapsedTime + " seconds. Lines read: " + lineCount);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

C#

using System;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;

namespace UniProtReader {
    class Program {
        static void Main(string[] args) {
            string gzFilePath = "path/to/uniprot_sprot.dat.gz";

            try {
                using (FileStream fileStream = new FileStream(gzFilePath, FileMode.Open, FileAccess.Read, FileShare.Read, 4096 * 1000, FileOptions.SequentialScan))
                using (BufferedStream bufferedStream = new BufferedStream(fileStream, 8192 * 1000))
                using (GZipStream gzipStream = new GZipStream(bufferedStream, CompressionMode.Decompress))
                using (StreamReader reader = new StreamReader(gzipStream)) {
                    Stopwatch stopwatch = new Stopwatch();
                    stopwatch.Start();

                    int lineCount = 0;
                    while (reader.ReadLine() != null) {
                        lineCount++;  // No processing, just reading
                    }

                    stopwatch.Stop();
                    Console.WriteLine($"Elapsed time: {stopwatch.ElapsedMilliseconds / 1000.0} seconds. Lines read: {lineCount}");
                }
            } catch (Exception e) {
                Console.WriteLine($"An error occurred: {e.Message}");
            }
        }
    }
}

---

System

OS Name Microsoft Windows 11 Pro
Version 10.0.26100 Build 26100
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name DESKTOP-ELN6IOH
System Manufacturer Gigabyte Technology Co., Ltd.
System Model B550 GAMING X V2
System Type x64-based PC
System SKU Default string
Processor AMD Ryzen 9 5900X 12-Core Processor, 3701 Mhz, 12 Core(s), 24 Logical Processor(s)
BIOS Version/Date American Megatrends International, LLC. FFa, 11/04/2024
SMBIOS Version 3.3
Embedded Controller Version 255.255
BIOS Mode UEFI
BaseBoard Manufacturer Gigabyte Technology Co., Ltd.
BaseBoard Product B550 GAMING X V2
BaseBoard Version x.x
Platform Role Desktop
Secure Boot State Off
PCR7 Configuration Elevation Required to View
Windows Directory C:\WINDOWS
System Directory C:\WINDOWS\system32
Boot Device \Device\HarddiskVolume4
Locale United Kingdom
Hardware Abstraction Layer Version = "10.0.26100.1"
Username DESKTOP-ELN6IOH\Merlin
Time Zone W. Europe Standard Time
Installed Physical Memory (RAM) 64.0 GB
Total Physical Memory 63.9 GB
Available Physical Memory 53.1 GB
Total Virtual Memory 63.9 GB
Available Virtual Memory 51.3 GB
Page File Space 0 bytes
Kernel DMA Protection Off
Virtualisation-based security Not enabled
App Control for Business policy Enforced
App Control for Business user mode policy Audit
Automatic Device Encryption Support Elevation Required to View
Hyper-V - VM Monitor Mode Extensions Yes
Hyper-V - Second Level Address Translation Extensions Yes
Hyper-V - Virtualisation Enabled in Firmware No
Hyper-V - Data Execution Protection Yes