Stored profiles

A stored profile is one identity on disk: the cookies and localStorage of a signed-in session, saved to a JSON file, plus a small file that says how that identity looks (user agent, viewport, proxy, locale). You sign in once, save, and every later run starts from the file, already signed in. One file per account lets you scrape as many identities as you have, each isolated from the others.

This page uses the Playwright library (@povio/pyro/wright) and its storageState format, which is the same JSON Playwright writes. For profiles that the server boots and signs in once and then forks per job, see Profiles booted once (fork).

There is no persistent profile directory

Pyro keeps all browser storage in memory: cookies, localStorage, sessionStorage, IndexedDB and the HTTP cache live in a browser context and are gone when it closes. There is no --user-data-dir flag and no launchPersistentContext. A profile persists only as the JSON file you save with context.storageState().

What a profile file holds

context.storageState() returns, and with { path } writes, this shape:

{
  "cookies": [
    {
      "name": "sid",
      "value": "f77fa7d6-eafe-4071-90c0-d85ac2ebcf81",
      "domain": "127.0.0.1",
      "path": "/",
      "expires": 1794509897.088918,
      "httpOnly": true,
      "secure": false,
      "sameSite": "Lax"
    }
  ],
  "origins": [
    {
      "origin": "http://127.0.0.1:4500",
      "localStorage": [{ "name": "apiToken", "value": "1137c7b4-1c1a-438b-b9e1-3f8c8f3fe3a0" }]
    }
  ]
}

Saved	Not saved
Every cookie in the context's jar, httpOnly ones included, with expiry	sessionStorage
localStorage of the origins of the context's open pages	IndexedDB (the indexedDB: true option is accepted and ignored)
	The HTTP cache, service workers, permissions

Two consequences:

• Save while a page of each origin you care about is open. localStorage is read from the open pages, one per origin. If the app keeps its token on app.example.com and your page is on www.example.com, open a page on app.example.com before you save.
• Apps that keep their session in IndexedDB or sessionStorage do not round-trip. For those, sign in again per run, or use a fork profile, which keeps the whole signed-in page in memory.

Loading works the other way. browser.newContext({ storageState }) takes a path or the object. Cookies go into the new context's jar before its first page loads. localStorage entries are written into each origin once, before the page's own scripts run, so the app finds its token at boot.

Directory layout

Keep one directory with two files per identity:

profiles/                     mode 700
  alice.json                  storageState: credentials, mode 600
  alice.identity.json         how alice looks: user agent, proxy, locale...
  bob.json
  bob.identity.json

Passwords do not go into this directory. Read them from the environment or a secret manager at sign-in time; the state file holds only what the site gave the browser after sign-in.

An identity file holds the context options for that identity, plus the settings the library applies another way:

{
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
  "viewport": { "width": 1440, "height": 900 },
  "extraHTTPHeaders": { "accept-language": "de-DE,de;q=0.9" },
  "locale": "de-DE",
  "timezoneId": "Europe/Berlin",
  "proxy": "http://user:pass@proxy-de.example.net:3128",
  "proxyBypass": "*.internal"
}

Field	How it is applied
userAgent, viewport, extraHTTPHeaders	Context options; the library honors them.
locale	CDP Emulation.setLocaleOverride: navigator.language(s) and Accept-Language. Intl and Date formatting keep the process locale. The library's own locale option has no effect.
timezoneId	CDP Emulation.setTimezoneOverride: Date, Intl, Temporal. It applies to the whole connection, so two contexts on one connection cannot have different zones. The library's own timezoneId option has no effect.
proxy, proxyBypass	pyro serve --proxy and --proxy-bypass: one browser server per distinct proxy. The library's proxy context option has no effect. Loopback hosts (localhost, 127.0.0.1) skip the proxy unless proxyBypass contains <-loopback>.

Proxies, user agents, headers explains each setting and what else is available.

A profile store

This module opens a profile by name. It reuses the stored state when it still works, signs in again when it does not, and saves the result. It runs one identity at a time; for many identities running jobs in parallel, with a limit per identity and re-sign-in in the middle of a run, use the identity pool recipe, which builds on the same ideas. Save it as profiles.ts:

// profiles.ts: one identity per name, stored on disk, reused across runs.
import { chromium, type Browser, type BrowserContext, type Page } from '@povio/pyro/wright';
import { existsSync } from 'node:fs';
import { mkdir, readFile, rename, writeFile } from 'node:fs/promises';

export const DIR = process.env.PROFILE_DIR ?? 'profiles';

export type Identity = {
  userAgent?: string;
  viewport?: { width: number; height: number };
  extraHTTPHeaders?: Record<string, string>;
  locale?: string;
  timezoneId?: string;
  proxy?: string;
  proxyBypass?: string;
};
export type SignIn = (page: Page, name: string) => Promise<void>;

const paths = (name: string) => {
  if (!/^[\w.-]+$/.test(name)) throw new Error(`bad profile name ${name}`);
  return { state: `${DIR}/${name}.json`, identity: `${DIR}/${name}.identity.json` };
};

export async function readIdentity(name: string): Promise<Identity> {
  const { identity } = paths(name);
  return existsSync(identity) ? JSON.parse(await readFile(identity, 'utf8')) : {};
}

// The proxy is a server flag, so each distinct proxy gets its own browser server.
const browsers = new Map<string, Promise<Browser>>();
function browserFor(id: Identity): Promise<Browser> {
  const key = `${id.proxy ?? ''}|${id.proxyBypass ?? ''}`;
  let browser = browsers.get(key);
  if (!browser) {
    const args = id.proxy ? ['--proxy', id.proxy, ...(id.proxyBypass ? ['--proxy-bypass', id.proxyBypass] : [])] : [];
    browser = chromium.launch({ args });
    browsers.set(key, browser);
  }
  return browser;
}
export async function closeAll() {
  for (const browser of browsers.values()) await (await browser).close();
  browsers.clear();
}

async function newContext(name: string, withState: boolean) {
  const identity = await readIdentity(name);
  const { locale, timezoneId, proxy, proxyBypass, ...options } = identity;
  const browser = await browserFor(identity);
  const context = await browser.newContext({ ...options, ...(withState ? { storageState: paths(name).state } : {}) });
  const page = await context.newPage();
  if (locale || timezoneId) {
    const cdp = await context.newCDPSession(page);
    if (locale) await cdp.send('Emulation.setLocaleOverride', { locale });
    if (timezoneId) await cdp.send('Emulation.setTimezoneOverride', { timezoneId });
  }
  return { context, page };
}

/** Saves cookies and localStorage, readable by the owner only, replaced atomically. */
export async function saveProfile(context: BrowserContext, name: string) {
  const { state } = paths(name);
  await mkdir(DIR, { recursive: true, mode: 0o700 });
  await writeFile(`${state}.tmp`, JSON.stringify(await context.storageState(), null, 2), { mode: 0o600 });
  await rename(`${state}.tmp`, state);
}

/** A signed-in context for `name`: the stored state if it still works, else a new sign-in. */
export async function openProfile(
  name: string,
  opts: { signIn: SignIn; isSignedIn: (page: Page) => Promise<boolean> },
): Promise<{ context: BrowserContext; page: Page; reused: boolean }> {
  if (existsSync(paths(name).state)) {
    const { context, page } = await newContext(name, true);
    if (await opts.isSignedIn(page)) return { context, page, reused: true };
    await context.close(); // expired or revoked
  }
  const { context, page } = await newContext(name, false);
  await opts.signIn(page, name);
  await saveProfile(context, name);
  return { context, page, reused: false };
}

And a script that uses it. The site-specific parts are signIn (fill the login form) and isSignedIn (open a page that needs a session and see whether you were sent to the login page). If the site keeps a token in localStorage and answers API calls with 401 once it expires, check that response too; the identity pool recipe shows both checks.

// scrape.ts: bun scrape.ts alice bob
import type { Page } from '@povio/pyro/wright';
import { closeAll, openProfile, type SignIn } from './profiles';

const BASE = 'http://127.0.0.1:4500';

const signIn: SignIn = async (page, name) => {
  const password = process.env[`PASSWORD_${name.toUpperCase()}`];
  if (!password) throw new Error(`no password for ${name}: set PASSWORD_${name.toUpperCase()}`);
  await page.goto(`${BASE}/login`);
  await page.getByLabel('User').fill(name);
  await page.getByLabel('Password').fill(password);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.waitForURL(`${BASE}/account`);
};

const isSignedIn = async (page: Page) => {
  await page.goto(`${BASE}/account`);
  return !page.url().startsWith(`${BASE}/login`);
};

try {
  for (const name of process.argv.slice(2)) {
    const { context, page, reused } = await openProfile(name, { signIn, isSignedIn });
    await page.goto(`${BASE}/account`);
    await page.locator('.order').first().waitFor();
    console.log(name, reused ? 'reused' : 'signed in', await page.locator('.order').allTextContents());
    await context.close();
  }
} finally {
  await closeAll();
}

First run, then a second run:

$ PASSWORD_ALICE=... PASSWORD_BOB=... bun scrape.ts alice bob
pyro: CDP server listening on ws://127.0.0.1:51214/
alice signed in [ "A-1: 12.5", "A-2: 99" ]
bob signed in [ "A-1: 12.5", "A-2: 99" ]
$ bun scrape.ts alice bob
pyro: CDP server listening on ws://127.0.0.1:51221/
alice reused [ "A-1: 12.5", "A-2: 99" ]
bob reused [ "A-1: 12.5", "A-2: 99" ]

The second run needs no passwords: both sessions came from the files. The pyro: line is the launched server's startup message on stderr; each distinct proxy in the identity files starts one more server and prints one more such line.

With identity files in place, each identity looks as its file says on every run, the reused ones included. In a test with the example identity file above for alice (pointing at a local proxy) and a second file for bob with another user agent, time zone and proxy, alice's pages reported navigator.language de-DE, the Europe/Berlin time zone, a 1440 px wide viewport and the Chrome user agent, and every request went through her proxy with her credentials. bob's went through his.

Only signing in

To create profiles without scraping, for example from a setup script or by hand once per account, the minimum is:

import { chromium } from '@povio/pyro/wright';

const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();
await page.goto('https://example.com/login');
// ...fill the form and submit...
await page.waitForURL('https://example.com/account');
await context.storageState({ path: 'profiles/alice.json' });
await browser.close();

Run chmod 600 profiles/alice.json afterwards; storageState({ path }) writes with your default permissions.

Sites with captchas, two-factor prompts or device checks at sign-in cannot always be signed in by a script. Pyro is headless only and has no window to sign in by hand. Sign in where you can (a script with the code from your authenticator app, or a one-time manual step in another tool that exports the same storageState JSON, such as Playwright with a headed Chrome), then load the file here.

Expiry and refresh

Sessions end. The site expires the cookie, rotates a token, or signs you out when it sees the session from a new IP. The store above handles this by checking: isSignedIn loads a page that needs a session, and a redirect to the login page triggers a new sign-in.

Some refinements:

• Check without a page. context.request sends the cookies of the context's storageState, outside the browser. A request that is redirected to the login page without a session tells you the state is stale without loading the app:

  const context = await browser.newContext({ storageState: 'profiles/alice.json' });
  const r = await context.request.get('https://example.com/account', { maxRedirects: 0 });
  const signedIn = r.status() === 200; // 302 to /login when the session is gone

  This sees cookies only. If the site's session lives in a localStorage token, check with a page.

• Look at the expiry first. Cookies carry expires (seconds since the epoch; -1 means a session cookie). If the session cookie expires within the next hour, sign in again before you start a long job:

  import { readFileSync } from 'node:fs';
  
  function expiresSoon(path: string, cookieName: string, marginSeconds = 3600): boolean {
    const { cookies } = JSON.parse(readFileSync(path, 'utf8'));
    const c = cookies.find((c: { name: string }) => c.name === cookieName);
    return !c || (c.expires > 0 && c.expires < Date.now() / 1000 + marginSeconds);
  }

• Save again after long runs. Sites that roll the session (a new cookie on each request, sliding expiry) leave the file behind. Call saveProfile(context, name) at the end of a job, while a page of the site is still open, so the next run starts from the newest state. storageState() reads localStorage only from the origins of the open pages: if the job ended on another origin, the site's token is missing from the new state. Merge the old file's origins into it before you save, as the identity pool does.

• Detect sign-outs mid-run. If a page you scrape suddenly shows the login form, stop that identity's jobs, sign in again, and retry the job, instead of saving the signed-out state over the good file.

• Keep one session per identity. Two processes signing in as the same account at the same time can invalidate each other's sessions. Run one job per identity at a time, or share one saved state between them and sign in from one place only.

Rotating identities

With several profiles, spread the work across them so no single account carries all of it. A simple round robin over the profile names:

const names = ['alice', 'bob', 'carol'];
let turn = 0;
const nextIdentity = () => names[turn++ % names.length];

For least-recently-used assignment, a limit of jobs per identity, and waiting when every identity is busy, use the pool in A pool of signed-in identities.

Rotate jobs over identities, not proxies under one identity. Keep each identity on the proxy in its file: a session that suddenly comes from another IP address is a common reason for a site to sign it out. See Mapping identities to proxies.

Rotation is for spreading load you are allowed to make. It is not a way around a site's rate limits or bans; see Scraping responsibly.

Running profiles in parallel

Each profile gets its own browser context, and contexts are isolated: two contexts on one connection have separate cookie jars and storage, so alice and bob can be open at the same time in one browser. For real parallelism, give each worker its own connection; each connection is a browser thread on the server. Concurrency at scale has a worker pool that assigns jobs to profiles, and A pool of signed-in identities adds a proxy per identity and re-sign-in.

The timezone override is the exception: it applies to a whole connection, and a second context on that connection asking for a different zone fails with Cannot set timezone. Keep identities with different time zones on different connections.

The same localStorage for every context: serve --local-storage

When every context needs the same localStorage entries, for example one API token that is not tied to a person, the server can seed them:

{ "https://app.example.com": { "token": "eyJhbGciOi...", "settings": { "theme": "dark" } } }

pyro serve --local-storage seed.json
# or: PYRO_LOCAL_STORAGE=seed.json pyro serve

Top-level keys are origins (or URLs; only the origin is used). Values are objects of entries. A non-string value is stored as its JSON text, so settings reads back as the string '{"theme":"dark"}'. Every connection and every new browser context starts with a fresh copy, and a page's writes stay in its own context. It seeds localStorage only, not cookies. With chromium.launch(), pass it as args: ['--local-storage', 'seed.json'].

Security: these files are credentials

A state file is as good as a password for as long as the session lives, and often better: it skips two-factor authentication. Treat it that way.

• Keep them out of version control. Add the directory to .gitignore:

  profiles/*.json
  !profiles/*.identity.json

  (Identity files may contain proxy passwords; if yours do, ignore them too.)

• Restrict permissions. chmod 700 profiles and chmod 600 profiles/*.json. The store above writes with mode 600. Run the scraper under its own system user.

• Encrypt at rest on shared machines and in backups. Options: keep the directory on an encrypted volume, or encrypt each file with age or sops and decrypt into memory at load time. newContext({ storageState }) accepts an object, so the plaintext never has to touch the disk:

  const state = JSON.parse(await decrypt(`profiles/${name}.json.age`)); // your decrypt function
  const context = await browser.newContext({ storageState: state });

  Moving profiles between machines is a complete version with node:crypto: sealed files, key rotation, and revocation.

• Do not log them. Cookie values and tokens end up in logs, crash reports and CI output easily. Log profile names, not states.

• Revoke when leaked. Signing out on the site (or changing the password) usually invalidates the session in the file. Rotating your encryption key does not: a copy that leaked with its key stays valid until the site ends the session.

• Mind the CDP port. Anyone who can connect to your pyro serve can read every cookie of every open context. Keep it on 127.0.0.1 or a private network. See Running as a service.

Next

• A pool of signed-in identities: many identities, a proxy and user agent each, a job limit per identity, re-sign-in on expiry. Tested end to end.
• Moving profiles between machines: encrypted profile files, key rotation, revocation.
• Profiles booted once (fork): the server signs in once and forks a fresh signed-in page per job.
• Profiles and pre-booted pages: the concept in general.
• Proxies, user agents, headers: every identity setting.