BlueGit

blue-star danique-bot

Files

danique-bot / node_modules / fd-slicer / README.md

README.md

Import template
timfa Tim · pushed by timfa · 4 years ago
c788ad20 History
README.md 6.2 KiB Download Raw

fd-slicer

Build Status Coverage Status

Safe fs.ReadStream and fs.WriteStream using the same fd.

Let's say that you want to perform a parallel upload of a file to a remote server. To do this, we want to create multiple read streams. The first thing you might think of is to use the {start: 0, end: 0} API of fs.createReadStream. This gives you two choices:

0. Use the same file descriptor for all fs.ReadStream objects. 0. Open the file multiple times, resulting in a separate file descriptor for each read stream.

Neither of these are acceptable options. The first one is a severe bug, because the API docs for fs.write state:

> Note that it is unsafe to use fs.write multiple times on the same file > without waiting for the callback. For this scenario, fs.createWriteStream > is strongly recommended.

fs.createWriteStream will solve the problem if you only create one of them for the file descriptor, but it will exhibit this unsafety if you create multiple write streams per file descriptor.

The second option suffers from a race condition. For each additional time the file is opened after the first, it is possible that the file is modified. So in our parallel uploading example, we might upload a corrupt file that never existed on the client's computer.

This module solves this problem by providing createReadStream and createWriteStream that operate on a shared file descriptor and provides the convenient stream API while still allowing slicing and dicing.

This module also gives you some additional power that the builtin fs.createWriteStream do not give you. These features are:

  • Emitting a 'progress' event on write.
  • Ability to set a maximum size and emit an error if this size is exceeded.
  • Ability to create an FdSlicer instance from a Buffer. This enables you

to provide API for handling files as well as buffers using the same API.

Usage

var fdSlicer = require('fd-slicer');
var fs = require('fs');

fs.open("file.txt", 'r', function(err, fd) {
  if (err) throw err;
  var slicer = fdSlicer.createFromFd(fd);
  var firstPart = slicer.createReadStream({start: 0, end: 100});
  var secondPart = slicer.createReadStream({start: 100});
  var firstOut = fs.createWriteStream("first.txt");
  var secondOut = fs.createWriteStream("second.txt");
  firstPart.pipe(firstOut);
  secondPart.pipe(secondOut);
});

You can also create from a buffer:

var fdSlicer = require('fd-slicer');
var slicer = FdSlicer.createFromBuffer(someBuffer);
var firstPart = slicer.createReadStream({start: 0, end: 100});
var secondPart = slicer.createReadStream({start: 100});
var firstOut = fs.createWriteStream("first.txt");
var secondOut = fs.createWriteStream("second.txt");
firstPart.pipe(firstOut);
secondPart.pipe(secondOut);

API Documentation

fdSlicer.createFromFd(fd, [options])

var fdSlicer = require('fd-slicer');
fs.open("file.txt", 'r', function(err, fd) {
  if (err) throw err;
  var slicer = fdSlicer.createFromFd(fd);
  // ...
});

Make sure fd is a properly initialized file descriptor. If you want to use createReadStream make sure you open it for reading and if you want to use createWriteStream make sure you open it for writing.

options is an optional object which can contain:

  • autoClose - if set to true, the file descriptor will be automatically

closed once the last stream that references it is closed. Defaults to false. ref() and unref() can be used to increase or decrease the reference count, respectively.

fdSlicer.createFromBuffer(buffer)

var fdSlicer = require('fd-slicer');
var slicer = fdSlicer.createFromBuffer(someBuffer);
// ...

Properties

fd

The file descriptor passed in. undefined if created from a buffer.

Methods

createReadStream(options)

Available options:

  • start - Number. The offset into the file to start reading from. Defaults

to 0.

  • end - Number. Exclusive upper bound offset into the file to stop reading

from.

  • highWaterMark - Number. The maximum number of bytes to store in the

internal buffer before ceasing to read from the underlying resource. Defaults to 16 KB.

  • encoding - String. If specified, then buffers will be decoded to strings

using the specified encoding. Defaults to null.

The ReadableStream that this returns has these additional methods:

  • destroy(err) - stop streaming. err is optional and is the error that

will be emitted in order to cause the streaming to stop. Defaults to new Error("stream destroyed").

createWriteStream(options)

Available options:

  • start - Number. The offset into the file to start writing to. Defaults to

0.

  • end - Number. Exclusive upper bound offset into the file. If this offset

is reached, the write stream will emit an 'error' event and stop functioning. In this situation, err.code === 'ETOOBIG'. Defaults to Infinity.

  • highWaterMark - Number. Buffer level when write() starts returning

false. Defaults to 16KB.

  • decodeStrings - Boolean. Whether or not to decode strings into Buffers

before passing them to _write(). Defaults to true.

The WritableStream that this returns has these additional methods:

  • destroy() - stop streaming

And these additional properties:

  • bytesWritten - number of bytes written to the stream

And these additional events:

  • 'progress' - emitted when bytesWritten changes.
read(buffer, offset, length, position, callback)

Equivalent to fs.read, but with concurrency protection. callback must be defined.

write(buffer, offset, length, position, callback)

Equivalent to fs.write, but with concurrency protection. callback must be defined.

ref()

Increase the autoClose reference count by 1.

unref()

Decrease the autoClose reference count by 1.

Events

'error'

Emitted if fs.close returns an error when auto closing.

'close'

Emitted when fd-slicer closes the file descriptor due to autoClose. Never emitted if created from a buffer.

Source
1# fd-slicer
2
3[![Build Status](https://travis-ci.org/andrewrk/node-fd-slicer.svg?branch=master)](https://travis-ci.org/andrewrk/node-fd-slicer)
4[![Coverage Status](https://img.shields.io/coveralls/andrewrk/node-fd-slicer.svg)](https://coveralls.io/r/andrewrk/node-fd-slicer)
5
6Safe `fs.ReadStream` and `fs.WriteStream` using the same fd.
7
8Let's say that you want to perform a parallel upload of a file to a remote
9server. To do this, we want to create multiple read streams. The first thing
10you might think of is to use the `{start: 0, end: 0}` API of
11`fs.createReadStream`. This gives you two choices:
12
13 0. Use the same file descriptor for all `fs.ReadStream` objects.
14 0. Open the file multiple times, resulting in a separate file descriptor
15 for each read stream.
16
17Neither of these are acceptable options. The first one is a severe bug,
18because the API docs for `fs.write` state:
19
20> Note that it is unsafe to use `fs.write` multiple times on the same file
21> without waiting for the callback. For this scenario, `fs.createWriteStream`
22> is strongly recommended.
23
24`fs.createWriteStream` will solve the problem if you only create one of them
25for the file descriptor, but it will exhibit this unsafety if you create
26multiple write streams per file descriptor.
27
28The second option suffers from a race condition. For each additional time the
29file is opened after the first, it is possible that the file is modified. So
30in our parallel uploading example, we might upload a corrupt file that never
31existed on the client's computer.
32
33This module solves this problem by providing `createReadStream` and
34`createWriteStream` that operate on a shared file descriptor and provides
35the convenient stream API while still allowing slicing and dicing.
36
37This module also gives you some additional power that the builtin
38`fs.createWriteStream` do not give you. These features are:
39
40 * Emitting a 'progress' event on write.
41 * Ability to set a maximum size and emit an error if this size is exceeded.
42 * Ability to create an `FdSlicer` instance from a `Buffer`. This enables you
43 to provide API for handling files as well as buffers using the same API.
44
45## Usage
46
47```js
48var fdSlicer = require('fd-slicer');
49var fs = require('fs');
50
51fs.open("file.txt", 'r', function(err, fd) {
52 if (err) throw err;
53 var slicer = fdSlicer.createFromFd(fd);
54 var firstPart = slicer.createReadStream({start: 0, end: 100});
55 var secondPart = slicer.createReadStream({start: 100});
56 var firstOut = fs.createWriteStream("first.txt");
57 var secondOut = fs.createWriteStream("second.txt");
58 firstPart.pipe(firstOut);
59 secondPart.pipe(secondOut);
60});
61```
62
63You can also create from a buffer:
64
65```js
66var fdSlicer = require('fd-slicer');
67var slicer = FdSlicer.createFromBuffer(someBuffer);
68var firstPart = slicer.createReadStream({start: 0, end: 100});
69var secondPart = slicer.createReadStream({start: 100});
70var firstOut = fs.createWriteStream("first.txt");
71var secondOut = fs.createWriteStream("second.txt");
72firstPart.pipe(firstOut);
73secondPart.pipe(secondOut);
74```
75
76## API Documentation
77
78### fdSlicer.createFromFd(fd, [options])
79
80```js
81var fdSlicer = require('fd-slicer');
82fs.open("file.txt", 'r', function(err, fd) {
83 if (err) throw err;
84 var slicer = fdSlicer.createFromFd(fd);
85 // ...
86});
87```
88
89Make sure `fd` is a properly initialized file descriptor. If you want to
90use `createReadStream` make sure you open it for reading and if you want
91to use `createWriteStream` make sure you open it for writing.
92
93`options` is an optional object which can contain:
94
95 * `autoClose` - if set to `true`, the file descriptor will be automatically
96 closed once the last stream that references it is closed. Defaults to
97 `false`. `ref()` and `unref()` can be used to increase or decrease the
98 reference count, respectively.
99
100### fdSlicer.createFromBuffer(buffer)
101
102```js
103var fdSlicer = require('fd-slicer');
104var slicer = fdSlicer.createFromBuffer(someBuffer);
105// ...
106```
107
108#### Properties
109
110##### fd
111
112The file descriptor passed in. `undefined` if created from a buffer.
113
114#### Methods
115
116##### createReadStream(options)
117
118Available `options`:
119
120 * `start` - Number. The offset into the file to start reading from. Defaults
121 to 0.
122 * `end` - Number. Exclusive upper bound offset into the file to stop reading
123 from.
124 * `highWaterMark` - Number. The maximum number of bytes to store in the
125 internal buffer before ceasing to read from the underlying resource.
126 Defaults to 16 KB.
127 * `encoding` - String. If specified, then buffers will be decoded to strings
128 using the specified encoding. Defaults to `null`.
129
130The ReadableStream that this returns has these additional methods:
131
132 * `destroy(err)` - stop streaming. `err` is optional and is the error that
133 will be emitted in order to cause the streaming to stop. Defaults to
134 `new Error("stream destroyed")`.
135
136##### createWriteStream(options)
137
138Available `options`:
139
140 * `start` - Number. The offset into the file to start writing to. Defaults to
141 0.
142 * `end` - Number. Exclusive upper bound offset into the file. If this offset
143 is reached, the write stream will emit an 'error' event and stop functioning.
144 In this situation, `err.code === 'ETOOBIG'`. Defaults to `Infinity`.
145 * `highWaterMark` - Number. Buffer level when `write()` starts returning
146 false. Defaults to 16KB.
147 * `decodeStrings` - Boolean. Whether or not to decode strings into Buffers
148 before passing them to` _write()`. Defaults to `true`.
149
150The WritableStream that this returns has these additional methods:
151
152 * `destroy()` - stop streaming
153
154And these additional properties:
155
156 * `bytesWritten` - number of bytes written to the stream
157
158And these additional events:
159
160 * 'progress' - emitted when `bytesWritten` changes.
161
162##### read(buffer, offset, length, position, callback)
163
164Equivalent to `fs.read`, but with concurrency protection.
165`callback` must be defined.
166
167##### write(buffer, offset, length, position, callback)
168
169Equivalent to `fs.write`, but with concurrency protection.
170`callback` must be defined.
171
172##### ref()
173
174Increase the `autoClose` reference count by 1.
175
176##### unref()
177
178Decrease the `autoClose` reference count by 1.
179
180#### Events
181
182##### 'error'
183
184Emitted if `fs.close` returns an error when auto closing.
185
186##### 'close'
187
188Emitted when fd-slicer closes the file descriptor due to `autoClose`. Never
189emitted if created from a buffer.