C# Interview Question #195

How would you process a large number of records without exhausting memory?

Senior .NET Practical Scenarios Senior Advanced

Quick Interview Answer

Avoid loading the complete dataset into memory with an unrestricted ToList.

Detailed Explanation

Avoid loading the complete dataset into memory with an unrestricted ToList.

Use pagination, batching, streaming, or asynchronous enumeration depending on the source. Select only required columns and avoid tracking when entities do not need to be updated.

For bulk data modifications, process manageable batches and consider database-side set-based operations such as ExecuteUpdate or ExecuteDelete where applicable.

If operations call external services, concurrency should be bounded so memory, sockets, database connections, and downstream rate limits are not exhausted.

Code Example

const int batchSize = 500;

var batch = await _context.Products
    .AsNoTracking()
    .OrderBy(p => p.Id)
    .Take(batchSize)
    .Select(p => new
    {
        p.Id,
        p.Name
    })
    .ToListAsync();