infinity-fetch - v1.2.0

infinity-fetch 🚀

Configurable recursive fetch for paginated APIs. Works in Node.js and browsers.

Automatically re-invokes a fetch function across pages until a stop condition is met — accumulating all results into a single array. Zero dependencies.

How it works

                    ┌─────────────────────────────────────────────────────┐
                    │                   infinityFetch                     │
                    └────────────────────────┬────────────────────────────┘
                                             │
                         initialParams = { start: 0, limit: 100 }
                                             │
                                             ▼
                    ┌────────────────────────────────────────┐
                    │          fetcher(params)               │
                    │   api.project('x').repo('y').commits() │
                    └────────────────┬───────────────────────┘
                                     │
                    ┌────────────────▼───────────────────────┐
                    │             Response                    │
                    │  { values, isLastPage, nextPageStart,  │
                    │    size, limit, start }                 │
                    └──────┬─────────────────────┬───────────┘
                           │                     │
                    isLastPage?                  no
                           │                     │
                          yes            getNextParams()
                           │          { start: nextPageStart }
                           │                     │
                           │                     ▼
                           │        ┌────────────────────────┐
                           │        │   fetcher(nextParams)  │  ← repeats
                           │        └────────────────────────┘
                           │
                           ▼
              ┌──────────────────────────────────────┐
              │   return { items[], pages: number }  │
              │   items  → all pages accumulated     │
              │   pages  → total iterations done     │
              └──────────────────────────────────────┘

Installation

npm install infinity-fetch

Usage

`pagedFetch` — for Bitbucket-style paginated APIs

If your API returns { values, isLastPage, nextPageStart, size, limit, start }, use the built-in helper:

import { pagedFetch } from 'infinity-fetch';

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100, // items per page, defaults to 100
});

console.log(`${items.length} commits fetched across ${pages} pages`);

With loading state, progress tracking, and a delay between pages:

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100,
  maxPages: 20,
  delay: 200, // wait 200ms between each page fetch
  onStart: () => setLoading(true),
  onEnd:   () => setLoading(false),
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} commits`);
  },
});

`infinityFetch` — generic, fully configurable

Use this when your API has a different response shape:

import { infinityFetch } from 'infinity-fetch';

const { items, pages } = await infinityFetch({
  // The function that performs a single page request
  fetcher: (params) => github.issues.list(params),

  // Parameters for the very first request
  initialParams: { page: 1, per_page: 50 },

  // When to stop — return true on the last page
  isLastPage: (response) => response.data.length < 50,

  // How to compute params for the next page
  getNextParams: (_response, currentParams) => ({
    ...currentParams,
    page: currentParams.page + 1,
  }),

  // Which field holds the items
  getItems: (response) => response.data,

  // Optional: safety cap on number of pages
  maxPages: 100,

  // Optional: milliseconds to wait between each page fetch
  delay: 200,

  // Optional: called once before the first fetch
  onStart: () => setLoading(true),

  // Optional: called once after all pages are done — receives the final result
  onEnd: ({ items, pages }) => {
    setLoading(false);
    console.log(`Done: ${items.length} items across ${pages} pages`);
  },

  // Optional: called after each individual page
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} items`);
  },
});

console.log(`${items.length} issues fetched across ${pages} pages`);

API Reference

`pagedFetch<TItem>(config)`

Option	Type	Default	Description
`fetcher`	`(params: PagedParams) => Promise<PagedResponse<TItem>>`	required	Function that fetches one page
`limit`	`number`	`100`	Items per page
`maxPages`	`number`	`Infinity`	Maximum pages to fetch (safety limit)
`delay`	`number`	—	Milliseconds to wait between each page fetch
`onStart`	`() => void`	—	Called once before the first fetch
`onEnd`	`(result: InfinityFetchResult<TItem>) => void`	—	Called once after all pages are done
`onPage`	`(items, response, pageIndex) => void`	—	Called after each individual page

PagedParams

{ start: number; limit: number }

PagedResponse<TItem> — expected response shape:

{
  values: TItem[];
  isLastPage: boolean;
  nextPageStart?: number;
  size: number;
  limit: number;
  start: number;
}

Returns: Promise<InfinityFetchResult<TItem>>

`infinityFetch<TResponse, TParams, TItem>(config)`

Option	Type	Default	Description
`fetcher`	`(params: TParams) => Promise<TResponse>`	required	Function that fetches one page
`initialParams`	`TParams`	required	Parameters for the first request
`isLastPage`	`(response: TResponse) => boolean`	required	Returns `true` to stop iteration
`getNextParams`	`(response: TResponse, currentParams: TParams) => TParams`	required	Computes params for the next page
`getItems`	`(response: TResponse) => TItem[]`	required	Extracts items from a response
`maxPages`	`number`	`Infinity`	Maximum pages to fetch (safety limit)
`delay`	`number`	—	Milliseconds to wait between each page fetch
`onStart`	`() => void`	—	Called once before the first fetch
`onEnd`	`(result: InfinityFetchResult<TItem>) => void`	—	Called once after all pages are done
`onPage`	`(items, response, pageIndex) => void`	—	Called after each individual page

Returns: Promise<InfinityFetchResult<TItem>>

type InfinityFetchResult<TItem> = {
  items: TItem[];  // all items collected across every page
  pages: number;   // total number of pages fetched
};

Compatibility

Environment	Support
Node.js 18+	✅
Node.js 20+	✅
Modern browsers	✅
Deno / Bun	✅
ESM	✅
TypeScript	✅ (types included)

Contributing

Commits must follow the Conventional Commits spec — this drives automatic versioning and changelog generation via semantic-release.

Commit prefix	Triggers
`fix:`	Patch release (`0.0.x`)
`feat:`	Minor release (`0.x.0`)
`feat!:` / `BREAKING CHANGE:`	Major release (`x.0.0`)
`chore:`, `docs:`, `test:`	No release

git commit -m "feat: add onPage callback to pagedFetch"
git commit -m "fix: handle missing nextPageStart gracefully"
git commit -m "feat!: rename items field to data"

Setting up repository secrets

For the release pipeline to work, add these secrets in Settings → Secrets and variables → Actions:

Secret	Description
`NPM_TOKEN`	npm automation token (`npm token create --type automation`)

GITHUB_TOKEN is provided automatically by GitHub Actions — no setup needed.

Enabling GitHub Pages

Go to Settings → Pages and set the source to GitHub Actions.

Changelog

See CHANGELOG.md for the full release history.

License

MIT