You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

584 lines
19 KiB

  1. # safe-buffer [![travis][travis-image]][travis-url] [![npm][npm-image]][npm-url] [![downloads][downloads-image]][downloads-url] [![javascript style guide][standard-image]][standard-url]
  2. [travis-image]: https://img.shields.io/travis/feross/safe-buffer/master.svg
  3. [travis-url]: https://travis-ci.org/feross/safe-buffer
  4. [npm-image]: https://img.shields.io/npm/v/safe-buffer.svg
  5. [npm-url]: https://npmjs.org/package/safe-buffer
  6. [downloads-image]: https://img.shields.io/npm/dm/safe-buffer.svg
  7. [downloads-url]: https://npmjs.org/package/safe-buffer
  8. [standard-image]: https://img.shields.io/badge/code_style-standard-brightgreen.svg
  9. [standard-url]: https://standardjs.com
  10. #### Safer Node.js Buffer API
  11. **Use the new Node.js Buffer APIs (`Buffer.from`, `Buffer.alloc`,
  12. `Buffer.allocUnsafe`, `Buffer.allocUnsafeSlow`) in all versions of Node.js.**
  13. **Uses the built-in implementation when available.**
  14. ## install
  15. ```
  16. npm install safe-buffer
  17. ```
  18. ## usage
  19. The goal of this package is to provide a safe replacement for the node.js `Buffer`.
  20. It's a drop-in replacement for `Buffer`. You can use it by adding one `require` line to
  21. the top of your node.js modules:
  22. ```js
  23. var Buffer = require('safe-buffer').Buffer
  24. // Existing buffer code will continue to work without issues:
  25. new Buffer('hey', 'utf8')
  26. new Buffer([1, 2, 3], 'utf8')
  27. new Buffer(obj)
  28. new Buffer(16) // create an uninitialized buffer (potentially unsafe)
  29. // But you can use these new explicit APIs to make clear what you want:
  30. Buffer.from('hey', 'utf8') // convert from many types to a Buffer
  31. Buffer.alloc(16) // create a zero-filled buffer (safe)
  32. Buffer.allocUnsafe(16) // create an uninitialized buffer (potentially unsafe)
  33. ```
  34. ## api
  35. ### Class Method: Buffer.from(array)
  36. <!-- YAML
  37. added: v3.0.0
  38. -->
  39. * `array` {Array}
  40. Allocates a new `Buffer` using an `array` of octets.
  41. ```js
  42. const buf = Buffer.from([0x62,0x75,0x66,0x66,0x65,0x72]);
  43. // creates a new Buffer containing ASCII bytes
  44. // ['b','u','f','f','e','r']
  45. ```
  46. A `TypeError` will be thrown if `array` is not an `Array`.
  47. ### Class Method: Buffer.from(arrayBuffer[, byteOffset[, length]])
  48. <!-- YAML
  49. added: v5.10.0
  50. -->
  51. * `arrayBuffer` {ArrayBuffer} The `.buffer` property of a `TypedArray` or
  52. a `new ArrayBuffer()`
  53. * `byteOffset` {Number} Default: `0`
  54. * `length` {Number} Default: `arrayBuffer.length - byteOffset`
  55. When passed a reference to the `.buffer` property of a `TypedArray` instance,
  56. the newly created `Buffer` will share the same allocated memory as the
  57. TypedArray.
  58. ```js
  59. const arr = new Uint16Array(2);
  60. arr[0] = 5000;
  61. arr[1] = 4000;
  62. const buf = Buffer.from(arr.buffer); // shares the memory with arr;
  63. console.log(buf);
  64. // Prints: <Buffer 88 13 a0 0f>
  65. // changing the TypedArray changes the Buffer also
  66. arr[1] = 6000;
  67. console.log(buf);
  68. // Prints: <Buffer 88 13 70 17>
  69. ```
  70. The optional `byteOffset` and `length` arguments specify a memory range within
  71. the `arrayBuffer` that will be shared by the `Buffer`.
  72. ```js
  73. const ab = new ArrayBuffer(10);
  74. const buf = Buffer.from(ab, 0, 2);
  75. console.log(buf.length);
  76. // Prints: 2
  77. ```
  78. A `TypeError` will be thrown if `arrayBuffer` is not an `ArrayBuffer`.
  79. ### Class Method: Buffer.from(buffer)
  80. <!-- YAML
  81. added: v3.0.0
  82. -->
  83. * `buffer` {Buffer}
  84. Copies the passed `buffer` data onto a new `Buffer` instance.
  85. ```js
  86. const buf1 = Buffer.from('buffer');
  87. const buf2 = Buffer.from(buf1);
  88. buf1[0] = 0x61;
  89. console.log(buf1.toString());
  90. // 'auffer'
  91. console.log(buf2.toString());
  92. // 'buffer' (copy is not changed)
  93. ```
  94. A `TypeError` will be thrown if `buffer` is not a `Buffer`.
  95. ### Class Method: Buffer.from(str[, encoding])
  96. <!-- YAML
  97. added: v5.10.0
  98. -->
  99. * `str` {String} String to encode.
  100. * `encoding` {String} Encoding to use, Default: `'utf8'`
  101. Creates a new `Buffer` containing the given JavaScript string `str`. If
  102. provided, the `encoding` parameter identifies the character encoding.
  103. If not provided, `encoding` defaults to `'utf8'`.
  104. ```js
  105. const buf1 = Buffer.from('this is a tést');
  106. console.log(buf1.toString());
  107. // prints: this is a tést
  108. console.log(buf1.toString('ascii'));
  109. // prints: this is a tC)st
  110. const buf2 = Buffer.from('7468697320697320612074c3a97374', 'hex');
  111. console.log(buf2.toString());
  112. // prints: this is a tést
  113. ```
  114. A `TypeError` will be thrown if `str` is not a string.
  115. ### Class Method: Buffer.alloc(size[, fill[, encoding]])
  116. <!-- YAML
  117. added: v5.10.0
  118. -->
  119. * `size` {Number}
  120. * `fill` {Value} Default: `undefined`
  121. * `encoding` {String} Default: `utf8`
  122. Allocates a new `Buffer` of `size` bytes. If `fill` is `undefined`, the
  123. `Buffer` will be *zero-filled*.
  124. ```js
  125. const buf = Buffer.alloc(5);
  126. console.log(buf);
  127. // <Buffer 00 00 00 00 00>
  128. ```
  129. The `size` must be less than or equal to the value of
  130. `require('buffer').kMaxLength` (on 64-bit architectures, `kMaxLength` is
  131. `(2^31)-1`). Otherwise, a [`RangeError`][] is thrown. A zero-length Buffer will
  132. be created if a `size` less than or equal to 0 is specified.
  133. If `fill` is specified, the allocated `Buffer` will be initialized by calling
  134. `buf.fill(fill)`. See [`buf.fill()`][] for more information.
  135. ```js
  136. const buf = Buffer.alloc(5, 'a');
  137. console.log(buf);
  138. // <Buffer 61 61 61 61 61>
  139. ```
  140. If both `fill` and `encoding` are specified, the allocated `Buffer` will be
  141. initialized by calling `buf.fill(fill, encoding)`. For example:
  142. ```js
  143. const buf = Buffer.alloc(11, 'aGVsbG8gd29ybGQ=', 'base64');
  144. console.log(buf);
  145. // <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>
  146. ```
  147. Calling `Buffer.alloc(size)` can be significantly slower than the alternative
  148. `Buffer.allocUnsafe(size)` but ensures that the newly created `Buffer` instance
  149. contents will *never contain sensitive data*.
  150. A `TypeError` will be thrown if `size` is not a number.
  151. ### Class Method: Buffer.allocUnsafe(size)
  152. <!-- YAML
  153. added: v5.10.0
  154. -->
  155. * `size` {Number}
  156. Allocates a new *non-zero-filled* `Buffer` of `size` bytes. The `size` must
  157. be less than or equal to the value of `require('buffer').kMaxLength` (on 64-bit
  158. architectures, `kMaxLength` is `(2^31)-1`). Otherwise, a [`RangeError`][] is
  159. thrown. A zero-length Buffer will be created if a `size` less than or equal to
  160. 0 is specified.
  161. The underlying memory for `Buffer` instances created in this way is *not
  162. initialized*. The contents of the newly created `Buffer` are unknown and
  163. *may contain sensitive data*. Use [`buf.fill(0)`][] to initialize such
  164. `Buffer` instances to zeroes.
  165. ```js
  166. const buf = Buffer.allocUnsafe(5);
  167. console.log(buf);
  168. // <Buffer 78 e0 82 02 01>
  169. // (octets will be different, every time)
  170. buf.fill(0);
  171. console.log(buf);
  172. // <Buffer 00 00 00 00 00>
  173. ```
  174. A `TypeError` will be thrown if `size` is not a number.
  175. Note that the `Buffer` module pre-allocates an internal `Buffer` instance of
  176. size `Buffer.poolSize` that is used as a pool for the fast allocation of new
  177. `Buffer` instances created using `Buffer.allocUnsafe(size)` (and the deprecated
  178. `new Buffer(size)` constructor) only when `size` is less than or equal to
  179. `Buffer.poolSize >> 1` (floor of `Buffer.poolSize` divided by two). The default
  180. value of `Buffer.poolSize` is `8192` but can be modified.
  181. Use of this pre-allocated internal memory pool is a key difference between
  182. calling `Buffer.alloc(size, fill)` vs. `Buffer.allocUnsafe(size).fill(fill)`.
  183. Specifically, `Buffer.alloc(size, fill)` will *never* use the internal Buffer
  184. pool, while `Buffer.allocUnsafe(size).fill(fill)` *will* use the internal
  185. Buffer pool if `size` is less than or equal to half `Buffer.poolSize`. The
  186. difference is subtle but can be important when an application requires the
  187. additional performance that `Buffer.allocUnsafe(size)` provides.
  188. ### Class Method: Buffer.allocUnsafeSlow(size)
  189. <!-- YAML
  190. added: v5.10.0
  191. -->
  192. * `size` {Number}
  193. Allocates a new *non-zero-filled* and non-pooled `Buffer` of `size` bytes. The
  194. `size` must be less than or equal to the value of
  195. `require('buffer').kMaxLength` (on 64-bit architectures, `kMaxLength` is
  196. `(2^31)-1`). Otherwise, a [`RangeError`][] is thrown. A zero-length Buffer will
  197. be created if a `size` less than or equal to 0 is specified.
  198. The underlying memory for `Buffer` instances created in this way is *not
  199. initialized*. The contents of the newly created `Buffer` are unknown and
  200. *may contain sensitive data*. Use [`buf.fill(0)`][] to initialize such
  201. `Buffer` instances to zeroes.
  202. When using `Buffer.allocUnsafe()` to allocate new `Buffer` instances,
  203. allocations under 4KB are, by default, sliced from a single pre-allocated
  204. `Buffer`. This allows applications to avoid the garbage collection overhead of
  205. creating many individually allocated Buffers. This approach improves both
  206. performance and memory usage by eliminating the need to track and cleanup as
  207. many `Persistent` objects.
  208. However, in the case where a developer may need to retain a small chunk of
  209. memory from a pool for an indeterminate amount of time, it may be appropriate
  210. to create an un-pooled Buffer instance using `Buffer.allocUnsafeSlow()` then
  211. copy out the relevant bits.
  212. ```js
  213. // need to keep around a few small chunks of memory
  214. const store = [];
  215. socket.on('readable', () => {
  216. const data = socket.read();
  217. // allocate for retained data
  218. const sb = Buffer.allocUnsafeSlow(10);
  219. // copy the data into the new allocation
  220. data.copy(sb, 0, 0, 10);
  221. store.push(sb);
  222. });
  223. ```
  224. Use of `Buffer.allocUnsafeSlow()` should be used only as a last resort *after*
  225. a developer has observed undue memory retention in their applications.
  226. A `TypeError` will be thrown if `size` is not a number.
  227. ### All the Rest
  228. The rest of the `Buffer` API is exactly the same as in node.js.
  229. [See the docs](https://nodejs.org/api/buffer.html).
  230. ## Related links
  231. - [Node.js issue: Buffer(number) is unsafe](https://github.com/nodejs/node/issues/4660)
  232. - [Node.js Enhancement Proposal: Buffer.from/Buffer.alloc/Buffer.zalloc/Buffer() soft-deprecate](https://github.com/nodejs/node-eps/pull/4)
  233. ## Why is `Buffer` unsafe?
  234. Today, the node.js `Buffer` constructor is overloaded to handle many different argument
  235. types like `String`, `Array`, `Object`, `TypedArrayView` (`Uint8Array`, etc.),
  236. `ArrayBuffer`, and also `Number`.
  237. The API is optimized for convenience: you can throw any type at it, and it will try to do
  238. what you want.
  239. Because the Buffer constructor is so powerful, you often see code like this:
  240. ```js
  241. // Convert UTF-8 strings to hex
  242. function toHex (str) {
  243. return new Buffer(str).toString('hex')
  244. }
  245. ```
  246. ***But what happens if `toHex` is called with a `Number` argument?***
  247. ### Remote Memory Disclosure
  248. If an attacker can make your program call the `Buffer` constructor with a `Number`
  249. argument, then they can make it allocate uninitialized memory from the node.js process.
  250. This could potentially disclose TLS private keys, user data, or database passwords.
  251. When the `Buffer` constructor is passed a `Number` argument, it returns an
  252. **UNINITIALIZED** block of memory of the specified `size`. When you create a `Buffer` like
  253. this, you **MUST** overwrite the contents before returning it to the user.
  254. From the [node.js docs](https://nodejs.org/api/buffer.html#buffer_new_buffer_size):
  255. > `new Buffer(size)`
  256. >
  257. > - `size` Number
  258. >
  259. > The underlying memory for `Buffer` instances created in this way is not initialized.
  260. > **The contents of a newly created `Buffer` are unknown and could contain sensitive
  261. > data.** Use `buf.fill(0)` to initialize a Buffer to zeroes.
  262. (Emphasis our own.)
  263. Whenever the programmer intended to create an uninitialized `Buffer` you often see code
  264. like this:
  265. ```js
  266. var buf = new Buffer(16)
  267. // Immediately overwrite the uninitialized buffer with data from another buffer
  268. for (var i = 0; i < buf.length; i++) {
  269. buf[i] = otherBuf[i]
  270. }
  271. ```
  272. ### Would this ever be a problem in real code?
  273. Yes. It's surprisingly common to forget to check the type of your variables in a
  274. dynamically-typed language like JavaScript.
  275. Usually the consequences of assuming the wrong type is that your program crashes with an
  276. uncaught exception. But the failure mode for forgetting to check the type of arguments to
  277. the `Buffer` constructor is more catastrophic.
  278. Here's an example of a vulnerable service that takes a JSON payload and converts it to
  279. hex:
  280. ```js
  281. // Take a JSON payload {str: "some string"} and convert it to hex
  282. var server = http.createServer(function (req, res) {
  283. var data = ''
  284. req.setEncoding('utf8')
  285. req.on('data', function (chunk) {
  286. data += chunk
  287. })
  288. req.on('end', function () {
  289. var body = JSON.parse(data)
  290. res.end(new Buffer(body.str).toString('hex'))
  291. })
  292. })
  293. server.listen(8080)
  294. ```
  295. In this example, an http client just has to send:
  296. ```json
  297. {
  298. "str": 1000
  299. }
  300. ```
  301. and it will get back 1,000 bytes of uninitialized memory from the server.
  302. This is a very serious bug. It's similar in severity to the
  303. [the Heartbleed bug](http://heartbleed.com/) that allowed disclosure of OpenSSL process
  304. memory by remote attackers.
  305. ### Which real-world packages were vulnerable?
  306. #### [`bittorrent-dht`](https://www.npmjs.com/package/bittorrent-dht)
  307. [Mathias Buus](https://github.com/mafintosh) and I
  308. ([Feross Aboukhadijeh](http://feross.org/)) found this issue in one of our own packages,
  309. [`bittorrent-dht`](https://www.npmjs.com/package/bittorrent-dht). The bug would allow
  310. anyone on the internet to send a series of messages to a user of `bittorrent-dht` and get
  311. them to reveal 20 bytes at a time of uninitialized memory from the node.js process.
  312. Here's
  313. [the commit](https://github.com/feross/bittorrent-dht/commit/6c7da04025d5633699800a99ec3fbadf70ad35b8)
  314. that fixed it. We released a new fixed version, created a
  315. [Node Security Project disclosure](https://nodesecurity.io/advisories/68), and deprecated all
  316. vulnerable versions on npm so users will get a warning to upgrade to a newer version.
  317. #### [`ws`](https://www.npmjs.com/package/ws)
  318. That got us wondering if there were other vulnerable packages. Sure enough, within a short
  319. period of time, we found the same issue in [`ws`](https://www.npmjs.com/package/ws), the
  320. most popular WebSocket implementation in node.js.
  321. If certain APIs were called with `Number` parameters instead of `String` or `Buffer` as
  322. expected, then uninitialized server memory would be disclosed to the remote peer.
  323. These were the vulnerable methods:
  324. ```js
  325. socket.send(number)
  326. socket.ping(number)
  327. socket.pong(number)
  328. ```
  329. Here's a vulnerable socket server with some echo functionality:
  330. ```js
  331. server.on('connection', function (socket) {
  332. socket.on('message', function (message) {
  333. message = JSON.parse(message)
  334. if (message.type === 'echo') {
  335. socket.send(message.data) // send back the user's message
  336. }
  337. })
  338. })
  339. ```
  340. `socket.send(number)` called on the server, will disclose server memory.
  341. Here's [the release](https://github.com/websockets/ws/releases/tag/1.0.1) where the issue
  342. was fixed, with a more detailed explanation. Props to
  343. [Arnout Kazemier](https://github.com/3rd-Eden) for the quick fix. Here's the
  344. [Node Security Project disclosure](https://nodesecurity.io/advisories/67).
  345. ### What's the solution?
  346. It's important that node.js offers a fast way to get memory otherwise performance-critical
  347. applications would needlessly get a lot slower.
  348. But we need a better way to *signal our intent* as programmers. **When we want
  349. uninitialized memory, we should request it explicitly.**
  350. Sensitive functionality should not be packed into a developer-friendly API that loosely
  351. accepts many different types. This type of API encourages the lazy practice of passing
  352. variables in without checking the type very carefully.
  353. #### A new API: `Buffer.allocUnsafe(number)`
  354. The functionality of creating buffers with uninitialized memory should be part of another
  355. API. We propose `Buffer.allocUnsafe(number)`. This way, it's not part of an API that
  356. frequently gets user input of all sorts of different types passed into it.
  357. ```js
  358. var buf = Buffer.allocUnsafe(16) // careful, uninitialized memory!
  359. // Immediately overwrite the uninitialized buffer with data from another buffer
  360. for (var i = 0; i < buf.length; i++) {
  361. buf[i] = otherBuf[i]
  362. }
  363. ```
  364. ### How do we fix node.js core?
  365. We sent [a PR to node.js core](https://github.com/nodejs/node/pull/4514) (merged as
  366. `semver-major`) which defends against one case:
  367. ```js
  368. var str = 16
  369. new Buffer(str, 'utf8')
  370. ```
  371. In this situation, it's implied that the programmer intended the first argument to be a
  372. string, since they passed an encoding as a second argument. Today, node.js will allocate
  373. uninitialized memory in the case of `new Buffer(number, encoding)`, which is probably not
  374. what the programmer intended.
  375. But this is only a partial solution, since if the programmer does `new Buffer(variable)`
  376. (without an `encoding` parameter) there's no way to know what they intended. If `variable`
  377. is sometimes a number, then uninitialized memory will sometimes be returned.
  378. ### What's the real long-term fix?
  379. We could deprecate and remove `new Buffer(number)` and use `Buffer.allocUnsafe(number)` when
  380. we need uninitialized memory. But that would break 1000s of packages.
  381. ~~We believe the best solution is to:~~
  382. ~~1. Change `new Buffer(number)` to return safe, zeroed-out memory~~
  383. ~~2. Create a new API for creating uninitialized Buffers. We propose: `Buffer.allocUnsafe(number)`~~
  384. #### Update
  385. We now support adding three new APIs:
  386. - `Buffer.from(value)` - convert from any type to a buffer
  387. - `Buffer.alloc(size)` - create a zero-filled buffer
  388. - `Buffer.allocUnsafe(size)` - create an uninitialized buffer with given size
  389. This solves the core problem that affected `ws` and `bittorrent-dht` which is
  390. `Buffer(variable)` getting tricked into taking a number argument.
  391. This way, existing code continues working and the impact on the npm ecosystem will be
  392. minimal. Over time, npm maintainers can migrate performance-critical code to use
  393. `Buffer.allocUnsafe(number)` instead of `new Buffer(number)`.
  394. ### Conclusion
  395. We think there's a serious design issue with the `Buffer` API as it exists today. It
  396. promotes insecure software by putting high-risk functionality into a convenient API
  397. with friendly "developer ergonomics".
  398. This wasn't merely a theoretical exercise because we found the issue in some of the
  399. most popular npm packages.
  400. Fortunately, there's an easy fix that can be applied today. Use `safe-buffer` in place of
  401. `buffer`.
  402. ```js
  403. var Buffer = require('safe-buffer').Buffer
  404. ```
  405. Eventually, we hope that node.js core can switch to this new, safer behavior. We believe
  406. the impact on the ecosystem would be minimal since it's not a breaking change.
  407. Well-maintained, popular packages would be updated to use `Buffer.alloc` quickly, while
  408. older, insecure packages would magically become safe from this attack vector.
  409. ## links
  410. - [Node.js PR: buffer: throw if both length and enc are passed](https://github.com/nodejs/node/pull/4514)
  411. - [Node Security Project disclosure for `ws`](https://nodesecurity.io/advisories/67)
  412. - [Node Security Project disclosure for`bittorrent-dht`](https://nodesecurity.io/advisories/68)
  413. ## credit
  414. The original issues in `bittorrent-dht`
  415. ([disclosure](https://nodesecurity.io/advisories/68)) and
  416. `ws` ([disclosure](https://nodesecurity.io/advisories/67)) were discovered by
  417. [Mathias Buus](https://github.com/mafintosh) and
  418. [Feross Aboukhadijeh](http://feross.org/).
  419. Thanks to [Adam Baldwin](https://github.com/evilpacket) for helping disclose these issues
  420. and for his work running the [Node Security Project](https://nodesecurity.io/).
  421. Thanks to [John Hiesey](https://github.com/jhiesey) for proofreading this README and
  422. auditing the code.
  423. ## license
  424. MIT. Copyright (C) [Feross Aboukhadijeh](http://feross.org)