Alternative Compute, Data-Lake Storage: How to Evaluate a Workload Move

Alternative compute and data-lake storage can change the economics of recurring data jobs when the complete execution path fits the workload. The query engine performs the processing; the storage architecture determines where inputs and outputs live. Both decisions need to preserve correctness, freshness, access and downstream usability.

Anavsan routes recurring workloads with compute and storage considered together. That includes daily transformations and scheduled jobs whose current resource consumption may no longer match their needs—governed through APEX so the proposed path stays attached to evidence, ownership and a verified outcome.

A useful review follows the full path: source data, transformation, persisted output and the consumers that depend on it. Lower execution cost only becomes a convincing benefit when the rest of that path works economically too. For how to choose candidates and compare avoidable cost, start with Do Your Daily Data Workloads Still Belong on Their Current Engine?

A data lake is part of the architecture, not the execution engine

“Move it to a data lake” leaves an important question unanswered: what will execute the workload?

An architecture using object storage still needs a compatible query engine, a way to discover and interpret the data, and appropriate access controls. Table format, catalog, file layout and maintenance choices can all affect the resulting service.

For a daily transformation, separate three decisions:

A design may change the engine while leaving shared data in place. Another may create a new lake-based representation. A third may leave the source untouched and change where the resulting dataset is persisted. The workload and its dependencies determine which is appropriate.

These are architecture patterns for evaluation, not a list of engine destinations.

The routing diagram usually leaves something out

A simple diagram shows a workload moving from engine A to engine B. It suggests a neat replacement: one execution cost disappears and another takes its place.

Real systems have more relationships. An ingestion job updates a table. A transformation builds a serving dataset. A dashboard queries it. A downstream export reads the dashboard’s source. An operational team depends on the export’s arrival time.

Moving one query can affect that chain in several ways. The destination might need access to additional tables. A replica may refresh on a different schedule. A policy may need an equivalent implementation. An old output may still be consumed after the visible dashboard moves.

Lineage makes these relationships easier to investigate. It does not automatically answer whether a dependency is still required, whether the observed map is complete or whether a change is authorized. Those questions need evidence and ownership as well.

First distinguish shared access from another copy

There are two broad data-access patterns to evaluate. Neither is universally better.

Shared formats can expand the available choices. Snowflake documents external-engine access to supported Iceberg tables through its catalog capabilities.^[1] That provides a concrete interoperability mechanism; it does not mean every existing native table or workload can move unchanged.

Open formats also do not settle SQL semantics, application behavior or operational responsibility. The target still needs to support the table features and operations used by the workload. Validate the exact combination instead of treating “supports Iceberg” as a complete compatibility test.

For a synchronized copy, ask what maintains it. That mechanism becomes part of the workload’s cost and service contract. If nobody owns the update process, the new route is not ready for production.

Build a dependency record around the proposed move

Start with a specific workload rather than attempting to document the entire estate. Capture what is needed to make the decision and operate the pilot safely.

A missing edge should trigger investigation. It should not automatically be interpreted as the absence of a consumer. Some dependencies may sit outside the metadata or history available to the review.

This also improves the migration scope. Teams sometimes discover that a small query depends on a wide set of upstream objects. Other times, a narrow, well-understood dataset provides a reasonable starting point for a pilot.

Freshness belongs in the measurement

Suppose an internal report must show data no more than 30 minutes old. Its new engine returns results in two seconds, but the destination copy is an hour behind. The query is fast and the workload still fails its requirement.

Measure freshness from the business event or agreed source watermark through to the result that the consumer receives. Include upstream processing, synchronization and any destination caching that affects the answer.

Avoid relying exclusively on the time at which a copy job completed. Completion does not necessarily prove that all expected records arrived or that every upstream input was current.

For a pilot, record the relevant watermarks with the comparison results. This makes it possible to distinguish a SQL correctness issue from a mismatch caused by different input versions. The team can then decide whether to change the data path, adjust the design or stop the move.

Storage deserves a lifecycle, not just a price estimate

A new copy is easy to justify during an experiment. It is harder to remove after another report begins using it.

Give each additional dataset a purpose, owner, refresh policy, retention rule and retirement condition. Record which consumers are allowed to depend on it while it is experimental.

For example, a pilot copy might exist only to compare a specific report over a representative period. If the pilot fails, the team should know how to retire it. If it succeeds, the copy should transition into an explicitly owned production asset rather than surviving under an old test label.

Do not assume that moving a consumer makes its source disposable. The source may serve other workloads, support recovery or remain the authoritative dataset. A retirement decision requires a separate dependency review.

This is where lineage and routing reinforce each other: a placement decision creates a lifecycle question, and the dependency record helps determine what can safely change.

Count the data path in the business case

The cost comparison should include any incremental execution, data retrieval, synchronization, storage, network transfer, monitoring and operating effort. Some amounts will appear on invoices; others represent internal work. Label the categories rather than presenting them as identical cash savings.

Remote access can also affect performance. DuckDB’s tuning documentation discusses the impact of remote I/O and tools for observing requests and data transferred.^[2] A fast test against local or warm data may not represent a production path that repeatedly reads remote objects.

The practical lesson is to measure the actual architecture. Test where the data resides, with representative file layout, demand and cache conditions. Include the original environment when it remains involved in refresh, recovery or fallback.

If the source capacity stays paid for, report that fact. A move can create useful headroom without reducing current invoices. That may still be worthwhile, but the business case should say which benefit it expects and over what period.

A hypothetical pilot: one report, two data paths

Illustrative scenario, not a customer result. A team wants to evaluate another engine for a recurring operations report. The report reads a daily transaction dataset and a frequently updated reference table. Its consumers expect corrections to appear during the working day.

An initial test copies only the transaction dataset. The query runs quickly, but the reference table is an older snapshot. Some results differ, and the apparent performance improvement is not yet meaningful.

The dependency review exposes the missing freshness requirement. The team now has a choice: provide a suitable shared-access path, maintain a correctly synchronized reference dataset or keep the report on its current engine.

Each choice has a different operating cost. Synchronization might eliminate the apparent benefit. Shared access might be feasible but require additional compatibility and permission checks. Staying might be the best outcome after a small improvement to the original query.

The pilot has still succeeded as a decision process. It identified the requirement before a broader rollout introduced unreliable results.

Use lineage to guide a controlled change

For the proposed route, create a review record with five items:

1. The workload and the reason for evaluating a different path.
2. The dependency map, including unresolved coverage gaps.
3. Acceptance criteria for results, freshness, access and performance.
4. A complete cost comparison and a rollback owner.
5. The conditions for expanding, stopping or retiring pilot assets.

Revisit that record when inputs, consumers or demand materially change. A route that was appropriate for one report may need reassessment after it becomes the source for several others.

How Anavsan approaches compute and storage together

Anavsan routes recurring workloads to suitable alternative query engines and storage, including data-lake architectures. The objective is to address compute and storage together, with the job’s requirements shaping the execution path.

For a daily dbt transformation or scheduled aggregation, the relevant unit is the complete flow: what it reads, how it runs, what it writes and which consumers depend on the result. Lineage provides context for that decision, while cost and service measurements establish whether a change is worthwhile.

Talk to Anavsan about a recurring workload whose compute or storage deserves a fresh look. Start with its inputs, outputs and daily operating requirements.

Continue reading

• Do Your Daily Data Workloads Still Belong on Their Current Engine?
• Anavsan APEX
• Snowflake Data Lineage: See Connections, Check Usage
• How Fresh Does Your Data Really Need to Be?

References

1. Snowflake, Use an external query engine with Iceberg tables
2. DuckDB, Tuning workloads
3. Anavsan APEX