cacache

NOTE: This repository has moved to https://github.com/npm/cacache

cacache is a Node.js library for managinglocal key and content address caches. It's really fast, really good atconcurrency, and it will never give you corrupted data, even if cache filesget corrupted or manipulated.

It was originally written to be used as npm's local cache, butcan just as easily be used on its own.

Translations: español

Install

$ npm install --save cacache

Table of Contents

• Example
• Features
• Contributing
• API
  • Using localized APIs
  • Reading
    • ls
    • ls.stream
    • get
    • get.stream
    • get.info
    • get.hasContent
  • Writing
    • put
    • put.stream
    • put* opts
    • rm.all
    • rm.entry
    • rm.content
  • Utilities
    • setLocale
    • clearMemoized
    • tmp.mkdir
    • tmp.withTmp
  • Integrity

    • Subresource Integrity

    • verify

    • verify.lastRun

Example

const cacache = require('cacache/en')
const fs = require('fs')

const tarball = '/path/to/mytar.tgz'
const cachePath = '/tmp/my-toy-cache'
const key = 'my-unique-key-1234'

// Cache it! Use `cachePath` as the root of the content cache
cacache.put(cachePath, key, '10293801983029384').then(integrity => {
  console.log(`Saved content to ${cachePath}.`)
})

const destination = '/tmp/mytar.tgz'

// Copy the contents out of the cache and into their destination!
// But this time, use stream instead!
cacache.get.stream(
  cachePath, key
).pipe(
  fs.createWriteStream(destination)
).on('finish', () => {
  console.log('done extracting!')
})

// The same thing, but skip the key index.
cacache.get.byDigest(cachePath, integrityHash).then(data => {
  fs.writeFile(destination, data, err => {
    console.log('tarball data fetched based on its sha512sum and written out!')
  })
})

Features

• Extraction by key or by content address (shasum, etc)
• Subresource Integrity web standard support
• Multi-hash support - safely host sha1, sha512, etc, in a single cache
• Automatic content deduplication
• Fault tolerance (immune to corruption, partial writes, process races, etc)
• Consistency guarantees on read and write (full data verification)
• Lockless, high-concurrency cache access
• Streaming support
• Promise support
• Pretty darn fast -- sub-millisecond reads and writes including verification
• Arbitrary metadata storage
• Garbage collection and additional offline verification
• Thorough test coverage
• There's probably a bloom filter in there somewhere. Those are cool, right? ��

Contributing

The cacache team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The Contributor Guide has all the information you need for everything from reporting bugs to contributing entire new features. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.

All participants and maintainers in this project are expected to follow Code of Conduct, and just generally be excellent to each other.

Please refer to the Changelog for project history details, too.

Happy hacking!

API

Using localized APIs

cacache includes a complete API in English, with the same features as othertranslations. To use the English API as documented in this README, userequire('cacache/en'). This is also currently the default if you dorequire('cacache'), but may change in the future.

cacache also supports other languages! You can find the list of currentlysupported ones by looking in ./locales in the source directory. You can usethe API in that language with require('cacache/<lang>').

Want to add support for a new language? Please go ahead! You should be able tocopy ./locales/en.js and ./locales/en.json and fill them in. Translating theREADME.md is a bit more work, but also appreciated if you get around to it. ����

> cacache.ls(cache) -> Promise<Object>

Lists info for all entries currently in the cache as a single large object. Eachentry in the object will be keyed by the unique index key, with correspondingget.info objects as the values.

Example

cacache.ls(cachePath).then(console.log)
// Output
{
  'my-thing': {
    key: 'my-thing',
    integrity: 'sha512-BaSe64/EnCoDED+HAsh=='
    path: '.testcache/content/deadbeef', // joined with `cachePath`
    time: 12345698490,
    size: 4023948,
    metadata: {
      name: 'blah',
      version: '1.2.3',
      description: 'this was once a package but now it is my-thing'
    }
  },
  'other-thing': {
    key: 'other-thing',
    integrity: 'sha1-ANothER+hasH=',
    path: '.testcache/content/bada55',
    time: 11992309289,
    size: 111112
  }
}

> cacache.ls.stream(cache) -> Readable

Lists info for all entries currently in the cache as a single large object.

This works just like ls, except get.info entries arereturned as 'data' events on the returned stream.

Example

cacache.ls.stream(cachePath).on('data', console.log)
// Output
{
  key: 'my-thing',
  integrity: 'sha512-BaSe64HaSh',
  path: '.testcache/content/deadbeef', // joined with `cachePath`
  time: 12345698490,
  size: 13423,
  metadata: {
    name: 'blah',
    version: '1.2.3',
    description: 'this was once a package but now it is my-thing'
  }
}

{
  key: 'other-thing',
  integrity: 'whirlpool-WoWSoMuchSupport',
  path: '.testcache/content/bada55',
  time: 11992309289,
  size: 498023984029
}

{
  ...
}

> cacache.get(cache, key, [opts]) -> Promise({data, metadata, integrity})

Returns an object with the cached data, digest, and metadata identified bykey. The data property of this object will be a Buffer instance thatpresumably holds some data that means something to you. I'm sure you know whatto do with it! cacache just won't care.

integrity is a SubresourceIntegritystring. That is, a string that can be used to verify data, which looks like<hash-algorithm>-<base64-integrity-hash>.

If there is no content identified by key, or if the locally-stored data doesnot pass the validity checksum, the promise will be rejected.

A sub-function, get.byDigest may be used for identical behavior, except lookupwill happen by integrity hash, bypassing the index entirely. This version of thefunction only returns data itself, without any wrapper.

Note

This function loads the entire cache entry into memory before returning it. Ifyou're dealing with Very Large data, consider using get.streaminstead.

Example

// Look up by key
cache.get(cachePath, 'my-thing').then(console.log)
// Output:
{
  metadata: {
    thingName: 'my'
  },
  integrity: 'sha512-BaSe64HaSh',
  data: Buffer#<deadbeef>,
  size: 9320
}

// Look up by digest
cache.get.byDigest(cachePath, 'sha512-BaSe64HaSh').then(console.log)
// Output:
Buffer#<deadbeef>

> cacache.get.stream(cache, key, [opts]) -> Readable

Returns a Readable Stream of the cached data identified by key.

If there is no content identified by key, or if the locally-stored data doesnot pass the validity checksum, an error will be emitted.

metadata and integrity events will be emitted before the stream closes, ifyou need to collect that extra data about the cached entry.

A sub-function, get.stream.byDigest may be used for identical behavior,except lookup will happen by integrity hash, bypassing the index entirely. Thisversion does not emit the metadata and integrity events at all.

Example

// Look up by key
cache.get.stream(
  cachePath, 'my-thing'
).on('metadata', metadata => {
  console.log('metadata:', metadata)
}).on('integrity', integrity => {
  console.log('integrity:', integrity)
}).pipe(
  fs.createWriteStream('./x.tgz')
)
// Outputs:
metadata: { ... }
integrity: 'sha512-SoMeDIGest+64=='

// Look up by digest
cache.get.stream.byDigest(
  cachePath, 'sha512-SoMeDIGest+64=='
).pipe(
  fs.createWriteStream('./x.tgz')
)

> cacache.get.info(cache, key) -> Promise

Looks up key in the cache index, returning information about the entry ifone exists.

Fields

• key - Key the entry was looked up under. Matches the key argument.
• integrity - Subresource Integrity hash for the content this entry refers to.
• path - Filesystem path where content is stored, joined with cache argument.
• time - Timestamp the entry was first added on.
• metadata - User-assigned metadata associated with the entry/content.

Example

cacache.get.info(cachePath, 'my-thing').then(console.log)

// Output
{
  key: 'my-thing',
  integrity: 'sha256-MUSTVERIFY+ALL/THINGS=='
  path: '.testcache/content/deadbeef',
  time: 12345698490,
  size: 849234,
  metadata: {
    name: 'blah',
    version: '1.2.3',
    description: 'this was once a package but now it is my-thing'
  }
}

> cacache.get.hasContent(cache, integrity) -> Promise

Looks up a Subresource Integrity hash in the cache. If contentexists for this integrity, it will return an object, with the specific single integrity hashthat was found in sri key, and the size of the found content as size. If no content exists for this integrity, it will return false.

Example

cacache.get.hasContent(cachePath, 'sha256-MUSTVERIFY+ALL/THINGS==').then(console.log)

// Output
{
  sri: {
    source: 'sha256-MUSTVERIFY+ALL/THINGS==',
    algorithm: 'sha256',
    digest: 'MUSTVERIFY+ALL/THINGS==',
    options: []
  },
  size: 9001
}

cacache.get.hasContent(cachePath, 'sha521-NOT+IN/CACHE==').then(console.log)

// Output
false

> cacache.put(cache, key, data, [opts]) -> Promise

Inserts data passed to it into the cache. The returned Promise resolves with adigest (generated according to opts.algorithms) after thecache entry has been successfully written.

Example

fetch(
  'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'
).then(data => {
  return cacache.put(cachePath, 'registry.npmjs.org|cacache@1.0.0', data)
}).then(integrity => {
  console.log('integrity hash is', integrity)
})

> cacache.put.stream(cache, key, [opts]) -> Writable

Returns a WritableStream that insertsdata written to it into the cache. Emits an integrity event with the digest ofwritten contents when it succeeds.

Example

request.get(
  'https://registry.npmjs.org/cacache/-/cacache-1.0.0.tgz'
).pipe(
  cacache.put.stream(
    cachePath, 'registry.npmjs.org|cacache@1.0.0'
  ).on('integrity', d => console.log(`integrity digest is ${d}`))
)

> cacache.put options

cacache.put functions have a number of options in common.

opts.metadata

Arbitrary metadata to be attached to the inserted key.

opts.size

If provided, the data stream will be verified to check that enough data waspassed through. If there's more or less data than expected, insertion will failwith an EBADSIZE error.

opts.integrity

If present, the pre-calculated digest for the inserted content. If this optionif provided and does not match the post-insertion digest, insertion will failwith an EINTEGRITY error.

algorithms has no effect if this option is present.

opts.algorithms

Default: ['sha512']

Hashing algorithms to use when calculating the subresource integritydigestfor inserted data. Can use any algorithm listed in crypto.getHashes() or'omakase'/'お任せします' to pick a random hash algorithm on each insertion. Youmay also use any anagram of 'modnar' to use this feature.

Currently only supports one algorithm at a time (i.e., an array length ofexactly 1). Has no effect if opts.integrity is present.

opts.uid/opts.gid

If provided, cacache will do its best to make sure any new files added to thecache use this particular uid/gid combination. This can be used,for example, to drop permissions when someone uses sudo, but cacache makesno assumptions about your needs here.

opts.memoize

Default: null

If provided, cacache will memoize the given cache insertion in memory, bypassingany filesystem checks for that key or digest in future cache fetches. Nothingwill be written to the in-memory cache unless this option is explicitly truthy.

If opts.memoize is an object or a Map-like (that is, an object with getand set methods), it will be written to instead of the global memoizationcache.

Reading from disk data can be forced by explicitly passing memoize: false tothe reader functions, but their default will be to read from memory.

> cacache.rm.all(cache) -> Promise

Clears the entire cache. Mainly by blowing away the cache directory itself.

Example

cacache.rm.all(cachePath).then(() => {
  console.log('THE APOCALYPSE IS UPON US ��')
})

> cacache.rm.entry(cache, key) -> Promise

Alias: cacache.rm

Removes the index entry for key. Content will still be accessible ifrequested directly by content address (get.stream.byDigest).

To remove the content itself (which might still be used by other entries), userm.content. Or, to safely vacuum any unused content, useverify.

Example

cacache.rm.entry(cachePath, 'my-thing').then(() => {
  console.log('I did not like it anyway')
})

> cacache.rm.content(cache, integrity) -> Promise

Removes the content identified by integrity. Any index entries referring to itwill not be usable again until the content is re-added to the cache with anidentical digest.

Example

cacache.rm.content(cachePath, 'sha512-SoMeDIGest/IN+BaSE64==').then(() => {
  console.log('data for my-thing is gone!')
})

> cacache.setLocale(locale)

Configure the language/locale used for messages and errors coming from cacache.The list of available locales is in the ./locales directory in the projectroot.

Interested in contributing more languages! Submit a PR!

> cacache.clearMemoized()

Completely resets the in-memory entry cache.

> tmp.mkdir(cache, opts) -> Promise<Path>

Returns a unique temporary directory inside the cache's tmp dir. Thisdirectory will use the same safe user assignment that all the other stuff use.

Once the directory is made, it's the user's responsibility that all files withinare made according to the same opts.gid/opts.uid settings that would bepassed in. If not, you can ask cacache to do it for you by callingtmp.fix(), which will fix all tmp directory permissions.

If you want automatic cleanup of this directory, usetmp.withTmp()

Example

cacache.tmp.mkdir(cache).then(dir => {
  fs.writeFile(path.join(dir, 'blablabla'), Buffer#<1234>, ...)
})

> tmp.withTmp(cache, opts, cb) -> Promise

Creates a temporary directory with tmp.mkdir() and calls cbwith it. The created temporary directory will be removed when the return valueof cb() resolves -- that is, if you return a Promise from cb(), the tmpdirectory will be automatically deleted once that promise completes.

The same caveats apply when it comes to managing permissions for the tmp dir'scontents.

Example

cacache.tmp.withTmp(cache, dir => {
  return fs.writeFileAsync(path.join(dir, 'blablabla'), Buffer#<1234>, ...)
}).then(() => {
  // `dir` no longer exists
})

Subresource Integrity Digests

For content verification and addressing, cacache uses strings following theSubresourceIntegrity spec.That is, any time cacache expects an integrity argument or option, itshould be in the format <hashAlgorithm>-<base64-hash>.

One deviation from the current spec is that cacache will support any hashalgorithms supported by the underlying Node.js process. You can usecrypto.getHashes() to see which ones you can use.

Generating Digests Yourself

If you have an existing content shasum, they are generally formatted as ahexadecimal string (that is, a sha1 would look like:5f5513f8822fdbe5145af33b64d8d970dcf95c6e). In order to be compatible withcacache, you'll need to convert this to an equivalent subresource integritystring. For this example, the corresponding hash would be:sha1-X1UT+IIv2+UUWvM7ZNjZcNz5XG4=.

If you want to generate an integrity string yourself for existing data, you canuse something like this:

const crypto = require('crypto')
const hashAlgorithm = 'sha512'
const data = 'foobarbaz'

const integrity = (
  hashAlgorithm +
  '-' +
  crypto.createHash(hashAlgorithm).update(data).digest('base64')
)

You can also use ssri to have a richer set of functionalityaround SRI strings, including generation, parsing, and translating from existinghex-formatted strings.

> cacache.verify(cache, opts) -> Promise

Checks out and fixes up your cache:

• Cleans up corrupted or invalid index entries.
• Custom entry filtering options.
• Garbage collects any content entries not referenced by the index.
• Checks integrity for all content entries and removes invalid content.
• Fixes cache ownership.
• Removes the tmp directory in the cache and all its contents.

When it's done, it'll return an object with various stats about the verificationprocess, including amount of storage reclaimed, number of valid entries, numberof entries removed, etc.

Options

• opts.uid - uid to assign to cache and its contents
• opts.gid - gid to assign to cache and its contents
• opts.filter - receives a formatted entry. Return false to remove it.Note: might be called more than once on the same entry.

Example

echo somegarbage >> $CACHEPATH/content/deadbeef

cacache.verify(cachePath).then(stats => {
  // deadbeef collected, because of invalid checksum.
  console.log('cache is much nicer now! stats:', stats)
})

> cacache.verify.lastRun(cache) -> Promise

Returns a Date representing the last time cacache.verify was run on cache.

Example

cacache.verify(cachePath).then(() => {
  cacache.verify.lastRun(cachePath).then(lastTime => {
    console.log('cacache.verify was last called on' + lastTime)
  })
})