Home PS3 Xbox Wii PSP DS Blog Register Contact

How Research Teams Choose Infrastructure for Long-Term Data Projects

Most infrastructure decisions in research are made once, quietly, at the start of a project and then lived with for five to seven years. A grant is awarded, someone spins up a machine or buys a box for the lab, the first dataset lands, and the arrangement calcifies. By the time anyone questions it, three PhD students have built pipelines against it and the original administrator has moved institutions.

That is the real design constraint. A long-term data project is not judged on how fast it starts but on whether the data is still intact, still readable, still attributable and still affordable in year six. The teams that get this right tend to ask a different set of questions from the ones that get it wrong.

Key Takeaways

• Long-term projects fail on durability, custody and cost drift far more often than on raw compute performance.
• Separate the three tiers early: active compute, working storage, and archival storage. They have different lifespans and different economics.
• Decide who administers the environment in year four, before you choose the environment.
• Data residency, retention rules and audit evidence are cheaper to design in than to retrofit under review.
• Predictable, fixed-cost infrastructure suits sustained research workloads better than consumption billing that fluctuates with activity.

Start with the lifespan, not the workload

Commercial infrastructure planning usually starts from current load. Research planning has to start from the calendar, because the funding period, the retention obligation and the publication timeline are all fixed in advance and none of them bend.

A five-year longitudinal study has a predictable shape: ingest and instrumentation in year one, heavy processing in years two and three, analysis and writing in year four, and a long tail of archival access and replication requests afterwards often extending years past the end of the grant. Compute demand peaks in the middle. Storage demand only ever rises. Treating both as a single line item is the most common planning error we see, and it produces environments that are simultaneously over-provisioned for compute and under-provisioned for storage.

The practical move is to size the compute tier for the busiest eighteen months, and to size the storage tier for the day the project formally ends plus the retention period that follows it.

The three tiers, and why they should not share a machine

Active compute. Where processing, model training, simulation and statistical work happen. Characterised by bursts, high CPU and memory demand, and tolerance of interruption a failed job is re-run, not lost.

Working storage. The datasets currently being processed. Needs low latency and high throughput, sits next to compute, and is genuinely small compared to the archive.

Archival storage. Everything the project has produced, retained for the full obligation period. Needs durability, versioning, integrity checks and immutability far more than it needs speed.

Combining these on one server is convenient in month one and expensive in year three, because the archive grows until it constrains the compute tier, and enlarging the compute tier drags the archive along with it. Keeping them separate means each can be scaled on its own schedule.

For the archive, S3-compatible object storage has become the sensible default in research. It gives a flat namespace that scales without repartitioning, an API that every analysis toolchain already speaks, and critically object versioning and object locking. Our own object storage platform holds data in triple replication across locations and supports object lock in both GOVERNANCE and COMPLIANCE modes, which is the mechanism that turns "we promised not to alter the raw data" into something you can demonstrate rather than assert.

Where dedicated hardware earns its place

Virtualised cloud instances are the right default for elastic, interruptible work. Sustained research computing frequently is not that.

Workloads that run for weeks at full CPU, genomics pipelines and simulation work with large memory footprints, and anything constrained by sustained disk throughput all behave more predictably on dedicated hardware, where there is no contention for the underlying resource and performance does not vary with what other tenants are doing. This is a large part of why sustained-compute projects gravitate toward dedicated servers in Bulgaria and comparable single-tenant options within the EU.

Fixed monthly cost is also easier to defend in a grant budget than a consumption line that rises whenever a student runs an expensive job. Single and dual-processor platforms, Intel or AMD, Linux or Windows, with unmetered traffic and no bandwidth cap the specification follows the workload instead of a tier list.

The pragmatic pattern that has emerged is hybrid: dedicated hardware for the sustained core, virtual instances for elastic overflow and for the services around the pipeline, and object storage underneath both.

Reproducibility is an infrastructure property

The FAIR principles findable, accessible, interoperable, reusable are usually discussed as a data management concern. In practice most of the failures are infrastructural. Data that lived on a departmental machine that was decommissioned is not findable. Data behind a bespoke access mechanism nobody documented is not accessible. Data whose provenance depends on a filesystem timestamp that a migration silently rewrote is not reusable.

Three things do most of the work here. Immutable raw data, so the original inputs cannot be overwritten by a well-meaning script. Versioned outputs, so a result published in 2027 can still be traced to the exact inputs and code that produced it. And a stable, documented access path a fixed endpoint and credential model rather than "ask whoever has the SSH key".

Who administers this in year four?

This is the question that decides more architectures than any benchmark, and the one most often deferred.

Research teams are rarely staffed with dedicated system administrators. The person who set the environment up is frequently a postdoc whose contract ends before the project does. Any design that assumes continuous in-house administration is making an assumption about staffing that the grant does not fund.

There are three honest answers, and each implies a different setup. If the team genuinely has an ongoing systems person, self-managed infrastructure gives maximum control at minimum cost. If administration is occasional and shared, a managed layer where the provider handles the operating system, patching, monitoring and backups removes the single point of human failure. If the institution runs central IT that will adopt the environment, the requirement is compatibility with their standards rather than independence from them.

Choosing honestly here is more valuable than choosing ambitiously. An unpatched server with no owner is a data loss event waiting for a date.

Residency, retention and the paperwork tier

Research data brings obligations that ordinary web workloads do not. Funder mandates specify retention periods. Ethics approvals specify handling conditions. Where the data concerns people, the General Data Protection Regulation specifies a good deal more, and collaboration agreements frequently specify jurisdiction outright.

Three requirements are worth settling before anything is provisioned:

Where the data physically sits. Not the provider's headquarters the data centre. For EU-funded and health-adjacent work this is often contractual rather than preferential.

What evidence you can produce. Certifications such as ISO 9001, ISO/IEC 27001 and ISO/IEC 20000-1 exist precisely so that you can point to an audited management system instead of describing your provider's good intentions. Delta.BG holds all three, and our infrastructure sits in an Equinix Tier 3+ facility in Sofia.

What happens at the end. Deletion obligations, handover format and export cost. Egress fees that look trivial per gigabyte become a real obstacle when a project needs to move twenty terabytes to a national repository at the end of its life.

A checklist worth running before you provision

1. State the retention period and the end-of-project date in writing, then size storage for that horizon rather than for today.
2. Separate archive from working storage from compute, even if all three start small.
3. Name the administrator for year four. If you cannot, choose a managed option.
4. Turn on versioning and immutability for raw data from the first upload, not after the first accident.
5. Test a restore. A backup that has never been restored is a hypothesis.
6. Write down the exit path, including export volume and cost.

Conclusion

Long-term research infrastructure rewards conservatism. The teams that reach year six with intact, citable, reproducible data are rarely the ones that chose the most sophisticated platform they are the ones that separated their storage tiers, wrote down who was responsible, made raw data immutable on day one, and picked a cost model that did not surprise them halfway through.

The technology matters less than the fact that someone thought about year four while it was still year one.

Planning infrastructure for a multi-year project? Delta.BG provides dedicated servers, Cloud VPS, S3-compatible Object Storage with triple replication and object locking, Block Storage and Managed Kubernetes from an Equinix Tier 3+ data centre in Sofia, with 24/7 monitoring and administration and ISO 9001, ISO/IEC 27001 and ISO/IEC 20000-1 certification. Talk to our team on +359 2 4 288 288 or at [email protected].