Backend Interview: How Do You Store Passwords Securely and Migrate Legacy Hashes?

Question and Scope

A SaaS product with 10,000,000 accounts is rebuilding password authentication. About 7,000,000 database records use bcrypt with cost 10, 2,000,000 use PBKDF2-HMAC-SHA256 with 210000 iterations, and 1,000,000 use SHA-256 without a per-record salt. The existing formats are inconsistent, and some rows do not directly identify their algorithm and parameters. At peak, the authentication service must handle 1500 password verifications per second. Each instance permits at most 24 concurrent expensive hashes.

Design a complete migration to Argon2id. Explain the threat model, parameter selection and capacity measurement, evolvable storage format, mixed-algorithm verification, login-time upgrade, concurrent password changes, long-inactive accounts, pepper management, response to database or key exposure, account-enumeration and resource-exhaustion defenses, and rollout and completion criteria. The account counts, algorithm mix, legacy parameters, and concurrency limit are interview constraints, not universal security thresholds.

This is a backend security and authentication-engineering question. The credential verifier and migration state machine are the focus. A full registration, password-recovery, MFA, or session design is out of scope except where those flows affect credential migration or breach response.

What the Interviewer Is Evaluating

First, can the candidate distinguish online guessing from offline cracking? Rate limits constrain a login endpoint, but they do not constrain an attacker who has obtained the hash database. Passwords are low-entropy human secrets. A fast digest such as SHA-256 lets an attacker enumerate candidates quickly; a password-specific hash raises the cost of each guess through a per-record salt, tunable computation, and memory cost.

Second, does the candidate understand that parameters must be measured against real capacity? “Use Argon2id” chooses an algorithm but does not finish the design. A complete answer selects memory m, iterations t, parallelism p, salt length, and output length, then calculates how concurrent verification affects memory, CPU, latency, and denial-of-service exposure. Higher parameters are not automatically safer. If attack traffic exhausts service memory, authentication availability fails first.

Third, can the candidate migrate irreversible data? The system does not know each user's plaintext password, so it cannot decrypt bcrypt results offline and convert them to Argon2id. The usual path is to verify a legacy record successfully and use the plaintext supplied in that request to create a current record. Long-inactive accounts require a controlled legacy verifier, temporary wrapping, or a reset, depending on the legacy algorithm's risk and exposure history.

Finally, a strong answer handles races and operational boundaries. A login upgrade must not overwrite a concurrent password reset. A pepper should not live beside the hash database. Algorithm versions, parameters, and migration state need observability, but logs must not contain passwords, complete hashes, or peppers. “The new code is deployed” is not a completion criterion.

Clarifications Before Answering

Can every existing format be identified reliably? Establish the algorithm, parameters, character encoding, salt location, and historical library version. Do not guess a format and try several verifiers for an unidentifiable row.
How did legacy bcrypt handle inputs beyond 72 bytes? Reproduce the historical implementation's encoding and truncation semantics so migration does not silently change the user's effective credential.
Has the database, a backup, or a legacy hash set ever been exposed? Wrapping the current database cannot retract an already exposed unsalted fast digest. Forced resets and session action may be necessary.
What is the authentication instance's real resource budget? Obtain baseline memory, CPU quota, hash-concurrency limit, autoscaling speed, target p95 and p99 latency, and acceptable failure rate.
Do FIPS or other compliance constraints apply? They can constrain algorithms and implementations, but a compliance label does not replace parameter measurement or migration design.
Does a pepper already exist? Confirm its storage, call dependency, versions, rotation capability, audit boundary, and behavior when the key service is unavailable.
What are the transaction boundaries for password changes and session issuance? The order of migration, concurrent reset, and session issuance must be explicit, or an old password may obtain a new session after a reset.
How should business value and risk divide long-inactive accounts? Privileged, recently active, and years-inactive accounts can have different deadlines and reverification requirements.

30-Second Answer Framework

“I would define the primary goal as resistance to offline guessing after a hash-database breach and protect online authentication capacity separately. New passwords use Argon2id with an independent random salt. The record carries the algorithm, version, m/t/p, salt, and output. If we use a pepper, it lives in a key system outside the database. I would use OWASP's current minimum as a candidate starting point, then load-test on production-equivalent instances for concurrent memory, CPU, and p99 latency instead of copying one value.

At login, the declared record version selects exactly one legacy verifier. After successful verification, I calculate the current Argon2id record and compare-and-swap against the old hash. If a concurrent change makes that fail, I reload and verify the latest record rather than overwrite a password reset. New and reset passwords use the current format immediately. Unsalted SHA-256 accounts get a shorter migration deadline; when safe wrapping is unavailable or exposure has occurred, they must reset.

The endpoint uses uniform errors, a dummy current hash, independent account and source rate limits, and bounded hash concurrency to resist enumeration and memory exhaustion. After release, I track migration by algorithm version, latency, memory, failures, CAS conflicts, and reset completion. I declare completion only after high-risk legacy formats reach zero, race and peak-load tests pass, and a key-rotation exercise succeeds.”

Step-by-Step Explanation

Step 1: Separate the online and offline attack surfaces

Normal login is the online path. Endpoint throughput, per-account controls, source controls, risk systems, and monitoring constrain the attacker. Guessing after a database breach is the offline path: the attacker runs the verifier on their own GPUs, ASICs, or cloud instances without the application's rate limits. Password storage primarily raises the cost of each candidate attempt on that second path.

Unsalted SHA-256 has two problems. It is fast, and the same password produces the same digest, so an attacker can reuse precomputed work and identify groups that reuse a password. An independent random salt for every record makes equal passwords produce different outputs and forces per-record work. The salt can be stored with the hash and need not be secret. A salt does not turn fast SHA-256 into a suitable password hash; the meaningful resistance comes from a dedicated, tunable, preferably memory-hard algorithm.

A pepper is a separate control. It is a server-side secret shared across records or managed by version, and it must be separated from the password database, typically in a key-management service, HSM, or protected execution environment. It can reduce the impact of a database-only breach, but it replaces neither per-record salts nor slow hashing. If the database and pepper are both exposed, database batch processing cannot safely rekey the affected records because the service does not possess the users' plaintext passwords.

Step 2: Select parameters and calculate authentication capacity

For new records, use a maintained Argon2id implementation. One current OWASP minimum candidate is m=19456 KiB, t=2, and p=1. RFC 9106 gives higher-memory general recommendations, including 64 MiB, three passes, and four lanes for a memory-constrained option. The documents target different operational constraints, so no one tuple is a universal Web-service constant.

A more defensible selection process is:

Start from a reviewed candidate on the same CPU, memory limit, and runtime used in production.
Measure single-verification median, p95, p99, actual resident memory, and CPU time.
Load-test a mix of normal peak traffic, login bursts, wrong passwords, and nonexistent accounts.
Bound concurrency and observe queue time, container OOMs, CPU throttling, and upstream timeouts.
Choose the highest sustainable attacker cost within the availability budget, and record the hardware, library version, and measurement date.

At 19 MiB, the theoretical hash working memory for 24 simultaneous operations is already 456 MiB, before process baseline, library overhead, request objects, and safety margin. If one verification takes 250 milliseconds on the target instance, its rough maximum is about 96 completions per second. Serving 1500 per second requires at least about 16 continuously available equivalent instances, with additional headroom for tail latency, failures, and autoscaling lag. These calculations expose the capacity order of magnitude; load tests determine the final values.

Use a fresh independent 128-bit random salt per record and, for example, a 256-bit output. Let a mature library generate and parse the standard encoding instead of assembling cryptographic fields manually. Password input needs a stable, documented byte encoding before hashing. If Unicode is accepted, define normalization and apply it consistently during registration, login, and migration.

Step 3: Make credential records self-describing and evolvable

The record must carry the non-secret information needed for verification. An Argon2 encoding can look like this:

text

$argon2id$v=19$m=19456,t=2,p=1$SALT_BASE64$TAG_BASE64

Legacy formats also need deterministic mappings such as {bcrypt}, {pbkdf2-sha256}, and {sha256-legacy}. A label is a parsing protocol, not a security endorsement. Reject an unknown label, missing field, invalid Base64 value, or out-of-policy parameter and send the account to a controlled recovery queue. Do not try algorithms in sequence until one happens to match.

Parameter parsing needs upper bounds. If an attacker can tamper with a record, they could set extreme memory or iteration values and consume service resources through a login request. The verifier accepts only deployed algorithms and allowed parameter ranges. An out-of-range record produces a security event without sensitive data and requires account recovery.

The credential table needs at least the current encoding, credential-update time, and security-disposition state. Migration reports can aggregate from the encoding prefix or a low-cardinality scheme_id. Never place a full hash, salt, candidate password, pepper secret, or verification intermediate in logs, metric labels, or analytics systems.

Step 4: Upgrade safely after a successful login

The migration function parses the record and invokes the one matching verifier. Only after a legacy credential succeeds does the service have the correct plaintext from this request and can calculate a current Argon2id record. Keep expensive work outside the short database transaction and condition the write on the old record:

text

record = loadCredential(userId)
ok = verifyByDeclaredScheme(record.hash, submittedPassword)
if !ok: rejectWithGenericError()

if needsRehash(record.hash):
  upgraded = hashWithCurrentPolicy(submittedPassword)
  changed = compareAndSwap(userId, expected=record.hash, replacement=upgraded)
  if !changed:
    latest = loadCredential(userId)
    if !verifyByDeclaredScheme(latest.hash, submittedPassword):
      rejectAndAskForFreshLogin()

issueSessionAfterCredentialStateIsConfirmed()

Compare-and-swap prevents a login upgrade from overwriting a concurrent password reset. A failed conditional update can mean another login completed the same migration or that the user just selected a different password. Reload and verify the current input against the latest record; do not issue a session merely because the old record once matched. Registration, voluntary password changes, and password recovery write the current format directly and never create another legacy record.

Risk policy determines whether login can continue when the upgrade write fails. A retryable database error can temporarily leave an acceptable legacy record and emit a migration-failure event. A high-risk format can require a successful upgrade before session issuance. In either case, cap retries so one login does not execute several expensive hashes indefinitely.

Step 5: Treat acceptable and high-risk legacy formats differently

Bcrypt and sufficiently strong PBKDF2 can retain read-only verification during a controlled migration period and upgrade after successful login. Minimize the legacy verifiers' exposure: they serve existing records only and cannot create new credentials. Track remaining accounts, activity, and migration velocity for every format. Before removing a verifier at its deadline, resolve every account that still depends on it.

Unsalted SHA-256 has higher risk. If no exposure has occurred and an immediate universal reset is infeasible, the service can treat the existing digest as an input to a newly salted slow outer algorithm as temporary database hardening. Verification first reproduces the historical SHA-256 operation and then verifies the outer layer. After a successful login, the service still replaces it with a standard Argon2id record computed from the submitted password.

That wrapper is not equivalent to Argon2id(password). It cannot retract an inner digest that has already leaked, and it does not repair historical encoding, truncation, or weak-password problems. If a legacy hash or backup was exposed, the account is privileged, or the format semantics are uncertain, require a reset, revoke relevant sessions, and recover through an independent verification flow. Ordinary accounts inactive for years can also have password login frozen at the deadline and use recovery on return instead of keeping the weakest verifier forever.

Step 6: Design pepper versions and breach response

If a pepper is used, an reviewed keyed preprocessing step can precede the password hash, or an HMAC can protect its output. A mature construction and security review should decide the exact composition. The database stores only a non-secret pepper version identifier; the actual key remains outside the database and its backups. The authentication service uses least-privilege access, with explicit cache lifetime and behavior during key-service failure.

Routine rotation can keep the current and previous keys briefly. After successful verification with the old version, recompute the full record from the submitted plaintext and current pepper. Old keys cannot remain indefinitely. A rotation plan counts accounts on the previous version and assigns a reset or freeze path to those that do not migrate.

Layer the breach response according to evidence:

Database only: preserve evidence, close the entry point, increase monitoring, and decide on resets after evaluating the algorithms and parameters. A secret pepper buys additional defense but does not make weak passwords safe.
Pepper only: rotate the key, investigate access logs, and determine whether the attacker might also have obtained the hash database.
Database and matching pepper: handle this as password-hash exposure, force affected accounts to reset, revoke or shorten relevant sessions, and warn users to change reused passwords.

A successful rotation does not erase historical exposure. The incident record should cover affected versions, account scope, backups, sessions, notification, completion percentage, and residual exceptions.

Step 7: Resist enumeration and resource exhaustion together

Nonexistent accounts, wrong passwords, disabled accounts, and migration failures return the same external error semantics, including consistent status and response shape. For a nonexistent account, run one controlled dummy hash under the current policy to reduce the obvious gap between an immediate return and expensive verification. Mixed legacy algorithms can still have different timing. Migration, runtime hardening, and statistical measurement should reduce exploitable differences; do not promise that every network response is perfectly constant-time.

Expensive hashing itself creates a denial-of-service surface. Before the hash queue, perform coarse source throttling and request-validity checks that do not reveal whether an account exists. For admitted work, enforce independent account and source quotas, a global bounded queue, and 24 concurrency permits per instance. A single bucket keyed by an IP-and-username pair lets an attacker change one dimension repeatedly and evade aggregate limits.

When the queue is full, fail fast with a uniform temporary response instead of accumulating work without bound. Scaling signals should include queue time, active hashes, memory headroom, and CPU throttling, not only request count. Logs contain an irreversible internal account identifier, low-cardinality scheme version, result class, throttling dimension, and latency bucket. They contain no submitted password, complete credential record, or key material.

Step 8: Roll out in stages and accept with evidence

Begin with a read-only inventory. Confirm that every legacy record parses, and verify historical implementations with known test vectors in an isolated environment. Then deploy code that can read every supported legacy format but writes only the current format. Make new registrations and password changes write Argon2id first. Enable login upgrades for a small cohort, observe CAS conflicts, verification failures, and resource curves, and expand gradually.

Before full release, verify at least:

Correct-password, wrong-password, boundary-length, Unicode, and malformed-record behavior for every historical format.
Races between login upgrade and concurrent password reset, proving that an old login cannot overwrite the new password or issue an invalid session.
Rotation and failure exercises for current, previous, and unknown pepper versions.
Latency, memory, CPU, queueing, and scaling at the target 1500 verifications per second with mixed attack traffic.
Message, status, size, and latency distributions for nonexistent accounts and every failure class.
Scans for sensitive fields in the database, backups, logs, metrics, and error tracking.

The migration dashboard groups by algorithm and risk tier: total count, recently active count, daily successful upgrades, failure reasons, mandatory-reset completion, and deadline. Technical completion means every new write uses the current policy; unsalted SHA-256 and exposed versions are at zero; unneeded legacy verifiers are removed; any remaining acceptable legacy formats have documented exceptions; peak-load and race tests pass; pepper rotation and recovery exercises have evidence; and the authentication SLO remains within agreement.

High-Quality Sample Answer

“I would first separate the two attack surfaces. Endpoint controls constrain online attacks; they do not constrain offline attacks after a hash-database breach. Passwords therefore need a dedicated memory-hard hash with an independent random salt and tunable cost. New records use a mature Argon2id implementation and self-describe the algorithm version, m/t/p, salt, and output. If enabled, peppers are versioned in a key system outside the database and its backups.

I would not copy parameters directly from a blog. I would use OWASP's current 19 MiB, t=2, p=1 as one minimum candidate and measure single and concurrent p50, p95, p99, CPU, and real memory on production-equivalent instances, then load-test normal peaks and attack traffic. Nineteen MiB times 24 concurrent operations is already 456 MiB of working memory, so the service needs a bounded queue, concurrency permits, and sufficient instance headroom. A mature library generates a random salt per record, and the record keeps parameters for future upgrades.

At login, the declared format selects exactly one verifier. After a legacy password succeeds, I calculate the current Argon2id record outside the database transaction and update only if the old hash still matches. If CAS fails, I reload the latest credential and verify this input again. Concurrent logins can then converge, while an old login cannot overwrite a concurrent password reset. I issue a session only after confirming current credential state. Registration, voluntary change, and recovery write only the new format from day one.

Bcrypt and still-acceptable PBKDF2 get a defined migration period with verification-only support. Unsalted SHA-256 enters a higher-risk queue. If it has never leaked, a newly salted slow outer wrapper can provide short-term containment, but it does not replace rehashing the actual password. Exposed, privileged, and long-inactive accounts reset or freeze by the deadline. Legacy verifiers do not remain forever.

The endpoint uses uniform errors and a dummy current hash to reduce account-enumeration differences, plus independent source and account limits, a global queue, and memory-concurrency bounds to resist hash-amplification denial of service. Completion evidence includes version distribution, zero high-risk legacy records, migration failures and CAS conflicts, peak-load and race tests, sensitive-log scans, pepper rotation, and breach exercises. I would declare completion only after those checks pass and the planned legacy verifiers are retired.”

Common Mistakes and Improvements

Storing passwords as salted SHA-256 → salt prevents cross-account work reuse but not fast guessing of one record → use a dedicated, tunable, memory-hard password hash.
Claiming hashes cannot be cracked → candidate guessing still finds weak passwords → state the goal as raising offline cost, alongside compromised-password blocking and MFA.
Treating OWASP parameters as permanently optimal → hardware, libraries, instance resources, and traffic change → record the benchmark environment, remeasure, and upgrade by version.
Converting every bcrypt record to Argon2id offline → a standard new hash cannot be derived without plaintext → rehash after successful verification and wrap, reset, or freeze the remainder by risk.
Updating migration without an old-value condition → login can overwrite a completed password reset → use compare-and-swap, then reload and reverify after failure.
Holding a user-row lock during hashing → expensive work amplifies lock duration and pool occupancy → compute outside the transaction and perform a short conditional write.
Keeping the pepper in the same database configuration table → one database breach obtains both layers → store the key in an independently protected system and audit by version.
Rotating a pepper by changing one environment variable → old records cannot be verified directly with the new key → keep two versions briefly, then recompute after successful login or require reset.
Returning immediately for a nonexistent account → response time reveals account existence → use uniform errors, a dummy hash, and empirical latency-distribution tests.
Running Argon2id without a concurrency bound → an attacker can amplify memory and CPU consumption → apply two-dimensional limits, then a bounded queue and concurrency permits.
Ignoring bcrypt's input boundary → historical 72-byte behavior can change the effective credential during migration → reproduce the legacy verifier and require an explicit password change for affected accounts.
Checking only that new registrations use Argon2id → active or high-risk legacy records remain exposed → track inventory by algorithm, risk, and activity, with zero and exception criteria.

Follow-Up Questions

Follow-up 1: Why can a salt be stored in plaintext while a pepper must remain secret?

A salt makes each record's input unique, preventing equal passwords from sharing outputs and precomputed work. An attacker who knows the salt must still guess each record separately. A pepper adds value because a database-only attacker lacks a server-side secret, so it must remain separated from the hash database. The controls have different jobs; a pepper does not replace an independent per-record salt.

Follow-up 2: Can the database directly increase an Argon2id iteration count?

No standard higher-parameter password hash can be derived from the old output without the user's plaintext. An outer wrapper can temporarily harden the old output, but it changes record semantics and cannot erase the risk of a previously exposed output. A standard upgrade still recomputes after successful password verification or requires a reset.

Follow-up 3: Why not permit login immediately after CAS fails?

The failure can mean another request completed the same migration, or it can mean the user simultaneously set a different password through recovery. The old password is invalid in the second case. Reloading the latest record and verifying the submitted input distinguishes the cases; a failed verification cannot receive a session.

Follow-up 4: Are higher Argon2id parameters always better?

Higher memory or time cost increases offline attack cost and legitimate-login resource use. Excessive parameters create tail latency, queue buildup, instance OOMs, and a denial-of-service amplifier for cheap requests. The right policy is the highest sustainable cost measured under actual hardware, concurrency, scaling, and SLO constraints, and it must be reassessed as hardware and threats change.

Follow-up 5: What should happen to an account that has been inactive for five years and still uses bcrypt?

Classify it by privilege, exposure history, and legacy parameters. An ordinary low-risk account can have password login frozen after the migration deadline and set a new password through controlled recovery when the user returns. Privileged or affected accounts should reset earlier and have sessions revoked. Keeping the legacy verifier forever prevents the migration from ever completing.