first commit

This commit is contained in:
DanielRamirezGe
2020-01-14 20:43:08 -06:00
parent 3557894f7f
commit 5eecfbf6cd
26326 changed files with 2434803 additions and 0 deletions
+191
View File
@@ -0,0 +1,191 @@
# Change Log
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
<a name="2.1.3"></a>
## [2.1.3](https://github.com/webpack-contrib/thread-loader/compare/v2.1.2...v2.1.3) (2019-08-08)
### Bug Fixes
* correct default for workerParallelJobs option ([#74](https://github.com/webpack-contrib/thread-loader/issues/74)) ([79758d0](https://github.com/webpack-contrib/thread-loader/commit/79758d0))
* do not allow empty or invalid node args when spin up child process ([#73](https://github.com/webpack-contrib/thread-loader/issues/73)) ([b02d503](https://github.com/webpack-contrib/thread-loader/commit/b02d503))
<a name="2.1.2"></a>
## [2.1.2](https://github.com/webpack-contrib/thread-loader/compare/v2.1.1...v2.1.2) (2019-01-25)
### Bug Fixes
* lifecycle handling for signals
<a name="2.1.1"></a>
## [2.1.1](https://github.com/webpack-contrib/thread-loader/compare/v2.1.0...v2.1.1) (2018-12-21)
### Performance Improvements
* use `neo-async` instead of `async` ([#54](https://github.com/webpack-contrib/thread-loader/issues/54)) ([d3a6664](https://github.com/webpack-contrib/thread-loader/commit/d3a6664))
<a name="2.1.0"></a>
# [2.1.0](https://github.com/webpack-contrib/thread-loader/compare/v2.0.2...v2.1.0) (2018-12-21)
### Features
* add poolRespawn flag to speed up incremental builds ([#52](https://github.com/webpack-contrib/thread-loader/issues/52)) ([76535bf](https://github.com/webpack-contrib/thread-loader/commit/76535bf))
<a name="2.0.2"></a>
## [2.0.2](https://github.com/webpack-contrib/thread-loader/compare/v2.0.1...v2.0.2) (2018-12-20)
### Bug Fixes
* build hang ([#53](https://github.com/webpack-contrib/thread-loader/issues/53)) ([fa02b60](https://github.com/webpack-contrib/thread-loader/commit/fa02b60))
<a name="2.0.1"></a>
## [2.0.1](https://github.com/webpack-contrib/thread-loader/compare/v2.0.0...v2.0.1) (2018-12-19)
### Bug Fixes
* memory leaks, worker and main process lifecycles ([#51](https://github.com/webpack-contrib/thread-loader/issues/51)) ([f10fe55](https://github.com/webpack-contrib/thread-loader/commit/f10fe55))
<a name="2.0.0"></a>
## [2.0.0](https://github.com/webpack-contrib/thread-loader/compare/v1.2.0...v2.0.0) (2018-12-18)
### Bug Fixes
* calculate number of workers correctly ([#49](https://github.com/webpack-contrib/thread-loader/issues/49)) ([fcbd813](https://github.com/webpack-contrib/thread-loader/commit/fcbd813))
* check on `undefined` for `worker.stdio` ([#45](https://github.com/webpack-contrib/thread-loader/issues/45)) ([c891a9c](https://github.com/webpack-contrib/thread-loader/commit/c891a9c))
* listen `end` events ([#42](https://github.com/webpack-contrib/thread-loader/issues/42)) ([0f87683](https://github.com/webpack-contrib/thread-loader/commit/0f87683))
### BREAKING CHANGE
* drop support for node < 6.9
<a name="1.2.0"></a>
# [1.2.0](https://github.com/webpack-contrib/thread-loader/compare/v1.1.5...v1.2.0) (2018-07-27)
### Features
* add target, minimize and resourceQuery into context ([#25](https://github.com/webpack-contrib/thread-loader/issues/25)) ([f3c7a2c](https://github.com/webpack-contrib/thread-loader/commit/f3c7a2c))
<a name="1.1.5"></a>
## [1.1.5](https://github.com/webpack-contrib/thread-loader/compare/v1.1.4...v1.1.5) (2018-02-26)
### Bug Fixes
* **package:** add `webpack >= 4` (`peerDependencies`) ([#22](https://github.com/webpack-contrib/thread-loader/issues/22)) ([9345756](https://github.com/webpack-contrib/thread-loader/commit/9345756))
* **WorkerError:** handle undefined `error` stacks ([#20](https://github.com/webpack-contrib/thread-loader/issues/20)) ([6fb5daf](https://github.com/webpack-contrib/thread-loader/commit/6fb5daf))
<a name="1.1.4"></a>
## [1.1.4](https://github.com/webpack-contrib/thread-loader/compare/v1.1.3...v1.1.4) (2018-02-21)
### Bug Fixes
* **index:** add `webpack >= v4.0.0` support ([#16](https://github.com/webpack-contrib/thread-loader/issues/16)) ([5d33937](https://github.com/webpack-contrib/thread-loader/commit/5d33937))
<a name="1.1.3"></a>
## [1.1.3](https://github.com/webpack-contrib/thread-loader/compare/v1.1.2...v1.1.3) (2018-02-07)
### Bug Fixes
* **WorkerPool:** trace stacks to avoid duplicated `err.messages` from workers ([#13](https://github.com/webpack-contrib/thread-loader/issues/13)) ([80dda4f](https://github.com/webpack-contrib/thread-loader/commit/80dda4f))
<a name="1.1.2"></a>
## [1.1.2](https://github.com/webpack-contrib/thread-loader/compare/v1.1.1...v1.1.2) (2017-10-09)
### Bug Fixes
* **readBuffer:** handle 0-byte reads ([c7ca960](https://github.com/webpack-contrib/thread-loader/commit/c7ca960))
<a name="1.1.1"></a>
## [1.1.1](https://github.com/webpack-contrib/thread-loader/compare/v1.1.0...v1.1.1) (2017-08-28)
### Bug Fixes
* **context:** Pass context to loader ([29ced70](https://github.com/webpack-contrib/thread-loader/commit/29ced70))
* **deps:** pass along result for dependencies ([19832ec](https://github.com/webpack-contrib/thread-loader/commit/19832ec))
* **example:** fix for broken sass and add watch ([47ba43e](https://github.com/webpack-contrib/thread-loader/commit/47ba43e))
<a name="1.1.0"></a>
# [1.1.0](https://github.com/webpack-contrib/thread-loader/compare/v1.0.3...v1.1.0) (2017-07-14)
### Features
* **pool:** add warmup method ([a0ce440](https://github.com/webpack-contrib/thread-loader/commit/a0ce440))
<a name="1.0.3"></a>
## [1.0.3](https://github.com/webpack-contrib/thread-loader/compare/v1.0.2...v1.0.3) (2017-05-27)
### Bug Fixes
* **resolve:** fix passing error to worker ([6561f57](https://github.com/webpack-contrib/thread-loader/commit/6561f57))
<a name="1.0.2"></a>
## [1.0.2](https://github.com/webpack-contrib/thread-loader/compare/v1.0.1...v1.0.2) (2017-05-27)
### Bug Fixes
* **resolve:** fix incorrect method for sending message ([bb92a28](https://github.com/webpack-contrib/thread-loader/commit/bb92a28))
<a name="1.0.1"></a>
## 1.0.1 (2017-04-28)
# Change Log
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
x.x.x / <year>-<month>-<day>
==================
* Bug fix -
* Feature -
* Chore -
* Docs -
+20
View File
@@ -0,0 +1,20 @@
Copyright JS Foundation and other contributors
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+151
View File
@@ -0,0 +1,151 @@
[![npm][npm]][npm-url]
[![deps][deps]][deps-url]
[![test][test]][test-url]
[![coverage][cover]][cover-url]
[![chat][chat]][chat-url]
<div align="center">
<a href="https://webpack.js.org/">
<img width="200" height="200" vspace="" hspace="25" src="https://cdn.rawgit.com/webpack/media/e7485eb2/logo/icon-square-big.svg">
</a>
<h1>thread-loader</h1>
<p>Runs the following loaders in a worker pool.</p>
</div>
<h2 align="center">Install</h2>
```bash
npm install --save-dev thread-loader
```
<h2 align="center">Usage</h2>
Put this loader in front of other loaders. The following loaders run in a worker pool.
Loaders running in a worker pool are limited. Examples:
* Loaders cannot emit files.
* Loaders cannot use custom loader API (i. e. by plugins).
* Loaders cannot access the webpack options.
Each worker is a separate node.js process, which has an overhead of ~600ms. There is also an overhead of inter-process communication.
Use this loader only for expensive operations!
<h2 align="center">Examples</h2>
**webpack.config.js**
```js
module.exports = {
module: {
rules: [
{
test: /\.js$/,
include: path.resolve("src"),
use: [
"thread-loader",
// your expensive loader (e.g babel-loader)
]
}
]
}
}
```
**with options**
```js
use: [
{
loader: "thread-loader",
// loaders with equal options will share worker pools
options: {
// the number of spawned workers, defaults to (number of cpus - 1) or
// fallback to 1 when require('os').cpus() is undefined
workers: 2,
// number of jobs a worker processes in parallel
// defaults to 20
workerParallelJobs: 50,
// additional node.js arguments
workerNodeArgs: ['--max-old-space-size=1024'],
// Allow to respawn a dead worker pool
// respawning slows down the entire compilation
// and should be set to false for development
poolRespawn: false,
// timeout for killing the worker processes when idle
// defaults to 500 (ms)
// can be set to Infinity for watching builds to keep workers alive
poolTimeout: 2000,
// number of jobs the poll distributes to the workers
// defaults to 200
// decrease of less efficient but more fair distribution
poolParallelJobs: 50,
// name of the pool
// can be used to create different pools with elsewise identical options
name: "my-pool"
}
},
// your expensive loader (e.g babel-loader)
]
```
**prewarming**
To prevent the high delay when booting workers it possible to warmup the worker pool.
This boots the max number of workers in the pool and loads specified modules into the node.js module cache.
``` js
const threadLoader = require('thread-loader');
threadLoader.warmup({
// pool options, like passed to loader options
// must match loader options to boot the correct pool
}, [
// modules to load
// can be any module, i. e.
'babel-loader',
'babel-preset-es2015',
'sass-loader',
]);
```
<h2 align="center">Maintainers</h2>
<table>
<tbody>
<tr>
<td align="center">
<a href="https://github.com/sokra">
<img width="150" height="150" src="https://github.com/sokra.png?size=150">
</br>
sokra
</a>
</td>
</tr>
<tbody>
</table>
[npm]: https://img.shields.io/npm/v/thread-loader.svg
[npm-url]: https://npmjs.com/package/thread-loader
[deps]: https://david-dm.org/webpack-contrib/thread-loader.svg
[deps-url]: https://david-dm.org/webpack-contrib/thread-loader
[chat]: https://img.shields.io/badge/gitter-webpack%2Fwebpack-brightgreen.svg
[chat-url]: https://gitter.im/webpack/webpack
[test]: http://img.shields.io/travis/webpack-contrib/thread-loader.svg
[test-url]: https://travis-ci.org/webpack-contrib/thread-loader
[cover]: https://codecov.io/gh/webpack-contrib/thread-loader/branch/master/graph/badge.svg
[cover-url]: https://codecov.io/gh/webpack-contrib/thread-loader
+32
View File
@@ -0,0 +1,32 @@
'use strict';
Object.defineProperty(exports, "__esModule", {
value: true
});
const stack = (err, worker, workerId) => {
const originError = (err.stack || '').split('\n').filter(line => line.trim().startsWith('at'));
const workerError = worker.split('\n').filter(line => line.trim().startsWith('at'));
const diff = workerError.slice(0, workerError.length - originError.length).join('\n');
originError.unshift(diff);
originError.unshift(err.message);
originError.unshift(`Thread Loader (Worker ${workerId})`);
return originError.join('\n');
};
class WorkerError extends Error {
constructor(err, workerId) {
super(err);
this.name = err.name;
this.message = err.message;
Error.captureStackTrace(this, this.constructor);
this.stack = stack(err, this.stack, workerId);
}
}
exports.default = WorkerError;
+376
View File
@@ -0,0 +1,376 @@
'use strict';
Object.defineProperty(exports, "__esModule", {
value: true
});
var _child_process = require('child_process');
var _child_process2 = _interopRequireDefault(_child_process);
var _queue = require('neo-async/queue');
var _queue2 = _interopRequireDefault(_queue);
var _mapSeries = require('neo-async/mapSeries');
var _mapSeries2 = _interopRequireDefault(_mapSeries);
var _readBuffer = require('./readBuffer');
var _readBuffer2 = _interopRequireDefault(_readBuffer);
var _WorkerError = require('./WorkerError');
var _WorkerError2 = _interopRequireDefault(_WorkerError);
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
const workerPath = require.resolve('./worker'); /* eslint-disable no-console */
let workerId = 0;
class PoolWorker {
constructor(options, onJobDone) {
this.disposed = false;
this.nextJobId = 0;
this.jobs = Object.create(null);
this.activeJobs = 0;
this.onJobDone = onJobDone;
this.id = workerId;
workerId += 1;
// Empty or invalid node args would break the child process
const sanitizedNodeArgs = (options.nodeArgs || []).filter(opt => !!opt);
this.worker = _child_process2.default.spawn(process.execPath, [].concat(sanitizedNodeArgs).concat(workerPath, options.parallelJobs), {
detached: true,
stdio: ['ignore', 'pipe', 'pipe', 'pipe', 'pipe']
});
this.worker.unref();
// This prevents a problem where the worker stdio can be undefined
// when the kernel hits the limit of open files.
// More info can be found on: https://github.com/webpack-contrib/thread-loader/issues/2
if (!this.worker.stdio) {
throw new Error(`Failed to create the worker pool with workerId: ${workerId} and ${''}configuration: ${JSON.stringify(options)}. Please verify if you hit the OS open files limit.`);
}
const [,,, readPipe, writePipe] = this.worker.stdio;
this.readPipe = readPipe;
this.writePipe = writePipe;
this.listenStdOutAndErrFromWorker(this.worker.stdout, this.worker.stderr);
this.readNextMessage();
}
listenStdOutAndErrFromWorker(workerStdout, workerStderr) {
if (workerStdout) {
workerStdout.on('data', this.writeToStdout);
}
if (workerStderr) {
workerStderr.on('data', this.writeToStderr);
}
}
ignoreStdOutAndErrFromWorker(workerStdout, workerStderr) {
if (workerStdout) {
workerStdout.removeListener('data', this.writeToStdout);
}
if (workerStderr) {
workerStderr.removeListener('data', this.writeToStderr);
}
}
writeToStdout(data) {
if (!this.disposed) {
process.stdout.write(data);
}
}
writeToStderr(data) {
if (!this.disposed) {
process.stderr.write(data);
}
}
run(data, callback) {
const jobId = this.nextJobId;
this.nextJobId += 1;
this.jobs[jobId] = { data, callback };
this.activeJobs += 1;
this.writeJson({
type: 'job',
id: jobId,
data
});
}
warmup(requires) {
this.writeJson({
type: 'warmup',
requires
});
}
writeJson(data) {
const lengthBuffer = Buffer.alloc(4);
const messageBuffer = Buffer.from(JSON.stringify(data), 'utf-8');
lengthBuffer.writeInt32BE(messageBuffer.length, 0);
this.writePipe.write(lengthBuffer);
this.writePipe.write(messageBuffer);
}
writeEnd() {
const lengthBuffer = Buffer.alloc(4);
lengthBuffer.writeInt32BE(0, 0);
this.writePipe.write(lengthBuffer);
}
readNextMessage() {
this.state = 'read length';
this.readBuffer(4, (lengthReadError, lengthBuffer) => {
if (lengthReadError) {
console.error(`Failed to communicate with worker (read length) ${lengthReadError}`);
return;
}
this.state = 'length read';
const length = lengthBuffer.readInt32BE(0);
this.state = 'read message';
this.readBuffer(length, (messageError, messageBuffer) => {
if (messageError) {
console.error(`Failed to communicate with worker (read message) ${messageError}`);
return;
}
this.state = 'message read';
const messageString = messageBuffer.toString('utf-8');
const message = JSON.parse(messageString);
this.state = 'process message';
this.onWorkerMessage(message, err => {
if (err) {
console.error(`Failed to communicate with worker (process message) ${err}`);
return;
}
this.state = 'soon next';
setImmediate(() => this.readNextMessage());
});
});
});
}
onWorkerMessage(message, finalCallback) {
const { type, id } = message;
switch (type) {
case 'job':
{
const { data, error, result } = message;
(0, _mapSeries2.default)(data, (length, callback) => this.readBuffer(length, callback), (eachErr, buffers) => {
const { callback: jobCallback } = this.jobs[id];
const callback = (err, arg) => {
if (jobCallback) {
delete this.jobs[id];
this.activeJobs -= 1;
this.onJobDone();
if (err) {
jobCallback(err instanceof Error ? err : new Error(err), arg);
} else {
jobCallback(null, arg);
}
}
finalCallback();
};
if (eachErr) {
callback(eachErr);
return;
}
let bufferPosition = 0;
if (result.result) {
result.result = result.result.map(r => {
if (r.buffer) {
const buffer = buffers[bufferPosition];
bufferPosition += 1;
if (r.string) {
return buffer.toString('utf-8');
}
return buffer;
}
return r.data;
});
}
if (error) {
callback(this.fromErrorObj(error), result);
return;
}
callback(null, result);
});
break;
}
case 'resolve':
{
const { context, request, questionId } = message;
const { data } = this.jobs[id];
data.resolve(context, request, (error, result) => {
this.writeJson({
type: 'result',
id: questionId,
error: error ? {
message: error.message,
details: error.details,
missing: error.missing
} : null,
result
});
});
finalCallback();
break;
}
case 'emitWarning':
{
const { data } = message;
const { data: jobData } = this.jobs[id];
jobData.emitWarning(this.fromErrorObj(data));
finalCallback();
break;
}
case 'emitError':
{
const { data } = message;
const { data: jobData } = this.jobs[id];
jobData.emitError(this.fromErrorObj(data));
finalCallback();
break;
}
default:
{
console.error(`Unexpected worker message ${type} in WorkerPool.`);
finalCallback();
break;
}
}
}
fromErrorObj(arg) {
let obj;
if (typeof arg === 'string') {
obj = { message: arg };
} else {
obj = arg;
}
return new _WorkerError2.default(obj, this.id);
}
readBuffer(length, callback) {
(0, _readBuffer2.default)(this.readPipe, length, callback);
}
dispose() {
if (!this.disposed) {
this.disposed = true;
this.ignoreStdOutAndErrFromWorker(this.worker.stdout, this.worker.stderr);
this.writeEnd();
}
}
}
class WorkerPool {
constructor(options) {
this.options = options || {};
this.numberOfWorkers = options.numberOfWorkers;
this.poolTimeout = options.poolTimeout;
this.workerNodeArgs = options.workerNodeArgs;
this.workerParallelJobs = options.workerParallelJobs;
this.workers = new Set();
this.activeJobs = 0;
this.timeout = null;
this.poolQueue = (0, _queue2.default)(this.distributeJob.bind(this), options.poolParallelJobs);
this.terminated = false;
this.setupLifeCycle();
}
isAbleToRun() {
return !this.terminated;
}
terminate() {
if (this.terminated) {
return;
}
this.terminated = true;
this.poolQueue.kill();
this.disposeWorkers(true);
}
setupLifeCycle() {
process.on('exit', () => {
this.terminate();
});
}
run(data, callback) {
if (this.timeout) {
clearTimeout(this.timeout);
this.timeout = null;
}
this.activeJobs += 1;
this.poolQueue.push(data, callback);
}
distributeJob(data, callback) {
// use worker with the fewest jobs
let bestWorker;
for (const worker of this.workers) {
if (!bestWorker || worker.activeJobs < bestWorker.activeJobs) {
bestWorker = worker;
}
}
if (bestWorker && (bestWorker.activeJobs === 0 || this.workers.size >= this.numberOfWorkers)) {
bestWorker.run(data, callback);
return;
}
const newWorker = this.createWorker();
newWorker.run(data, callback);
}
createWorker() {
// spin up a new worker
const newWorker = new PoolWorker({
nodeArgs: this.workerNodeArgs,
parallelJobs: this.workerParallelJobs
}, () => this.onJobDone());
this.workers.add(newWorker);
return newWorker;
}
warmup(requires) {
while (this.workers.size < this.numberOfWorkers) {
this.createWorker().warmup(requires);
}
}
onJobDone() {
this.activeJobs -= 1;
if (this.activeJobs === 0 && isFinite(this.poolTimeout)) {
this.timeout = setTimeout(() => this.disposeWorkers(), this.poolTimeout);
}
}
disposeWorkers(fromTerminate) {
if (!this.options.poolRespawn && !fromTerminate) {
this.terminate();
return;
}
if (this.activeJobs === 0 || fromTerminate) {
for (const worker of this.workers) {
worker.dispose();
}
this.workers.clear();
}
}
}
exports.default = WorkerPool;
+3
View File
@@ -0,0 +1,3 @@
'use strict';
module.exports = require('./index');
+59
View File
@@ -0,0 +1,59 @@
'use strict';
Object.defineProperty(exports, "__esModule", {
value: true
});
exports.warmup = exports.pitch = undefined;
var _loaderUtils = require('loader-utils');
var _loaderUtils2 = _interopRequireDefault(_loaderUtils);
var _workerPools = require('./workerPools');
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
function pitch() {
const options = _loaderUtils2.default.getOptions(this) || {};
const workerPool = (0, _workerPools.getPool)(options);
if (!workerPool.isAbleToRun()) {
return;
}
const callback = this.async();
workerPool.run({
loaders: this.loaders.slice(this.loaderIndex + 1).map(l => {
return {
loader: l.path,
options: l.options,
ident: l.ident
};
}),
resource: this.resourcePath + (this.resourceQuery || ''),
sourceMap: this.sourceMap,
emitError: this.emitError,
emitWarning: this.emitWarning,
resolve: this.resolve,
target: this.target,
minimize: this.minimize,
resourceQuery: this.resourceQuery,
optionsContext: this.rootContext || this.options.context
}, (err, r) => {
if (r) {
r.fileDependencies.forEach(d => this.addDependency(d));
r.contextDependencies.forEach(d => this.addContextDependency(d));
}
if (err) {
callback(err);
return;
}
callback(null, ...r.result);
});
}
function warmup(options, requires) {
const workerPool = (0, _workerPools.getPool)(options);
workerPool.warmup(requires);
}
exports.pitch = pitch;
exports.warmup = warmup; // eslint-disable-line import/prefer-default-export
+44
View File
@@ -0,0 +1,44 @@
'use strict';
Object.defineProperty(exports, "__esModule", {
value: true
});
exports.default = readBuffer;
function readBuffer(pipe, length, callback) {
if (length === 0) {
callback(null, Buffer.alloc(0));
return;
}
let remainingLength = length;
const buffers = [];
const readChunk = () => {
const onChunk = arg => {
let chunk = arg;
let overflow;
if (chunk.length > remainingLength) {
overflow = chunk.slice(remainingLength);
chunk = chunk.slice(0, remainingLength);
remainingLength = 0;
} else {
remainingLength -= chunk.length;
}
buffers.push(chunk);
if (remainingLength === 0) {
pipe.removeListener('data', onChunk);
pipe.pause();
if (overflow) {
pipe.unshift(overflow);
}
callback(null, Buffer.concat(buffers, length));
}
};
pipe.on('data', onChunk);
pipe.resume();
};
readChunk();
}
+305
View File
@@ -0,0 +1,305 @@
'use strict';
var _fs = require('fs');
var _fs2 = _interopRequireDefault(_fs);
var _module = require('module');
var _module2 = _interopRequireDefault(_module);
var _loaderRunner = require('loader-runner');
var _loaderRunner2 = _interopRequireDefault(_loaderRunner);
var _queue = require('neo-async/queue');
var _queue2 = _interopRequireDefault(_queue);
var _readBuffer = require('./readBuffer');
var _readBuffer2 = _interopRequireDefault(_readBuffer);
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
const writePipe = _fs2.default.createWriteStream(null, { fd: 3 }); /* global require */
/* eslint-disable no-console */
const readPipe = _fs2.default.createReadStream(null, { fd: 4 });
writePipe.on('finish', onTerminateWrite);
readPipe.on('end', onTerminateRead);
writePipe.on('close', onTerminateWrite);
readPipe.on('close', onTerminateRead);
readPipe.on('error', onError);
writePipe.on('error', onError);
const PARALLEL_JOBS = +process.argv[2] || 20;
let terminated = false;
let nextQuestionId = 0;
const callbackMap = Object.create(null);
function onError(error) {
console.error(error);
}
function onTerminateRead() {
terminateRead();
}
function onTerminateWrite() {
terminateWrite();
}
function writePipeWrite(...args) {
if (!terminated) {
writePipe.write(...args);
}
}
function writePipeCork() {
if (!terminated) {
writePipe.cork();
}
}
function writePipeUncork() {
if (!terminated) {
writePipe.uncork();
}
}
function terminateRead() {
terminated = true;
readPipe.removeAllListeners();
}
function terminateWrite() {
terminated = true;
writePipe.removeAllListeners();
}
function terminate() {
terminateRead();
terminateWrite();
}
function toErrorObj(err) {
return {
message: err.message,
details: err.details,
stack: err.stack,
hideStack: err.hideStack
};
}
function toNativeError(obj) {
if (!obj) return null;
const err = new Error(obj.message);
err.details = obj.details;
err.missing = obj.missing;
return err;
}
function writeJson(data) {
writePipeCork();
process.nextTick(() => {
writePipeUncork();
});
const lengthBuffer = Buffer.alloc(4);
const messageBuffer = Buffer.from(JSON.stringify(data), 'utf-8');
lengthBuffer.writeInt32BE(messageBuffer.length, 0);
writePipeWrite(lengthBuffer);
writePipeWrite(messageBuffer);
}
const queue = (0, _queue2.default)(({ id, data }, taskCallback) => {
try {
_loaderRunner2.default.runLoaders({
loaders: data.loaders,
resource: data.resource,
readResource: _fs2.default.readFile.bind(_fs2.default),
context: {
version: 2,
resolve: (context, request, callback) => {
callbackMap[nextQuestionId] = callback;
writeJson({
type: 'resolve',
id,
questionId: nextQuestionId,
context,
request
});
nextQuestionId += 1;
},
emitWarning: warning => {
writeJson({
type: 'emitWarning',
id,
data: toErrorObj(warning)
});
},
emitError: error => {
writeJson({
type: 'emitError',
id,
data: toErrorObj(error)
});
},
exec: (code, filename) => {
const module = new _module2.default(filename, undefined);
module.paths = _module2.default._nodeModulePaths(undefined.context); // eslint-disable-line no-underscore-dangle
module.filename = filename;
module._compile(code, filename); // eslint-disable-line no-underscore-dangle
return module.exports;
},
options: {
context: data.optionsContext
},
webpack: true,
'thread-loader': true,
sourceMap: data.sourceMap,
target: data.target,
minimize: data.minimize,
resourceQuery: data.resourceQuery
}
}, (err, lrResult) => {
const {
result,
cacheable,
fileDependencies,
contextDependencies
} = lrResult;
const buffersToSend = [];
const convertedResult = Array.isArray(result) && result.map(item => {
const isBuffer = Buffer.isBuffer(item);
if (isBuffer) {
buffersToSend.push(item);
return {
buffer: true
};
}
if (typeof item === 'string') {
const stringBuffer = Buffer.from(item, 'utf-8');
buffersToSend.push(stringBuffer);
return {
buffer: true,
string: true
};
}
return {
data: item
};
});
writeJson({
type: 'job',
id,
error: err && toErrorObj(err),
result: {
result: convertedResult,
cacheable,
fileDependencies,
contextDependencies
},
data: buffersToSend.map(buffer => buffer.length)
});
buffersToSend.forEach(buffer => {
writePipeWrite(buffer);
});
setImmediate(taskCallback);
});
} catch (e) {
writeJson({
type: 'job',
id,
error: toErrorObj(e)
});
taskCallback();
}
}, PARALLEL_JOBS);
function dispose() {
terminate();
queue.kill();
process.exit(0);
}
function onMessage(message) {
try {
const { type, id } = message;
switch (type) {
case 'job':
{
queue.push(message);
break;
}
case 'result':
{
const { error, result } = message;
const callback = callbackMap[id];
if (callback) {
const nativeError = toNativeError(error);
callback(nativeError, result);
} else {
console.error(`Worker got unexpected result id ${id}`);
}
delete callbackMap[id];
break;
}
case 'warmup':
{
const { requires } = message;
// load modules into process
requires.forEach(r => require(r)); // eslint-disable-line import/no-dynamic-require, global-require
break;
}
default:
{
console.error(`Worker got unexpected job type ${type}`);
break;
}
}
} catch (e) {
console.error(`Error in worker ${e}`);
}
}
function readNextMessage() {
(0, _readBuffer2.default)(readPipe, 4, (lengthReadError, lengthBuffer) => {
if (lengthReadError) {
console.error(`Failed to communicate with main process (read length) ${lengthReadError}`);
return;
}
const length = lengthBuffer.length && lengthBuffer.readInt32BE(0);
if (length === 0) {
// worker should dispose and exit
dispose();
return;
}
(0, _readBuffer2.default)(readPipe, length, (messageError, messageBuffer) => {
if (terminated) {
return;
}
if (messageError) {
console.error(`Failed to communicate with main process (read message) ${messageError}`);
return;
}
const messageString = messageBuffer.toString('utf-8');
const message = JSON.parse(messageString);
onMessage(message);
setImmediate(() => readNextMessage());
});
});
}
// start reading messages from main process
readNextMessage();
+45
View File
@@ -0,0 +1,45 @@
'use strict';
Object.defineProperty(exports, "__esModule", {
value: true
});
exports.getPool = undefined;
var _os = require('os');
var _os2 = _interopRequireDefault(_os);
var _WorkerPool = require('./WorkerPool');
var _WorkerPool2 = _interopRequireDefault(_WorkerPool);
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
const workerPools = Object.create(null);
function calculateNumberOfWorkers() {
// There are situations when this call will return undefined so
// we are fallback here to 1.
// More info on: https://github.com/nodejs/node/issues/19022
const cpus = _os2.default.cpus() || { length: 1 };
return Math.max(1, cpus.length - 1);
}
function getPool(options) {
const workerPoolOptions = {
name: options.name || '',
numberOfWorkers: options.workers || calculateNumberOfWorkers(),
workerNodeArgs: options.workerNodeArgs,
workerParallelJobs: options.workerParallelJobs || 20,
poolTimeout: options.poolTimeout || 500,
poolParallelJobs: options.poolParallelJobs || 200,
poolRespawn: options.poolRespawn || false
};
const tpKey = JSON.stringify(workerPoolOptions);
workerPools[tpKey] = workerPools[tpKey] || new _WorkerPool2.default(workerPoolOptions);
const workerPool = workerPools[tpKey];
return workerPool;
}
exports.getPool = getPool; // eslint-disable-line import/prefer-default-export
+111
View File
@@ -0,0 +1,111 @@
{
"_from": "thread-loader@^2.1.3",
"_id": "thread-loader@2.1.3",
"_inBundle": false,
"_integrity": "sha512-wNrVKH2Lcf8ZrWxDF/khdlLlsTMczdcwPA9VEK4c2exlEPynYWxi9op3nPTo5lAnDIkE0rQEB3VBP+4Zncc9Hg==",
"_location": "/thread-loader",
"_phantomChildren": {},
"_requested": {
"type": "range",
"registry": true,
"raw": "thread-loader@^2.1.3",
"name": "thread-loader",
"escapedName": "thread-loader",
"rawSpec": "^2.1.3",
"saveSpec": null,
"fetchSpec": "^2.1.3"
},
"_requiredBy": [
"/@vue/cli-plugin-babel",
"/@vue/cli-service"
],
"_resolved": "https://registry.npmjs.org/thread-loader/-/thread-loader-2.1.3.tgz",
"_shasum": "cbd2c139fc2b2de6e9d28f62286ab770c1acbdda",
"_spec": "thread-loader@^2.1.3",
"_where": "/home/george/citwa/red_de_investigacion_front/first/node_modules/@vue/cli-plugin-babel",
"author": {
"name": "Tobias Koppers @sokra"
},
"bugs": {
"url": "https://github.com/webpack-contrib/thread-loader/issues"
},
"bundleDependencies": false,
"dependencies": {
"loader-runner": "^2.3.1",
"loader-utils": "^1.1.0",
"neo-async": "^2.6.0"
},
"deprecated": false,
"description": "Runs the following loaders in a worker pool",
"devDependencies": {
"babel-cli": "^6.26.0",
"babel-core": "^6.26.3",
"babel-jest": "^23.6.0",
"babel-loader": "^7.1.5",
"babel-plugin-transform-object-rest-spread": "^6.26.0",
"babel-polyfill": "^6.26.0",
"babel-preset-env": "^1.7.0",
"cross-env": "^5.2.0",
"css-loader": "^1.0.1",
"del-cli": "^1.1.0",
"eslint": "^5.10.0",
"eslint-config-webpack": "^1.2.5",
"eslint-plugin-import": "^2.14.0",
"jest": "^23.6.0",
"lint-staged": "^8.1.0",
"lodash": "^4.17.11",
"mini-css-extract-plugin": "^0.5.0",
"node-sass": "^4.11.0",
"nodemon": "^1.18.8",
"pre-commit": "^1.2.2",
"sass-loader": "^7.1.0",
"standard-version": "^4.4.0",
"webpack": "^4.27.1",
"webpack-cli": "^3.1.2",
"webpack-defaults": "^1.6.0"
},
"engines": {
"node": ">= 6.9.0 <7.0.0 || >= 8.9.0"
},
"files": [
"dist"
],
"homepage": "https://github.com/webpack-contrib/thread-loader",
"license": "MIT",
"lint-staged": {
"*.js": [
"eslint --fix",
"git add"
]
},
"main": "dist/cjs.js",
"name": "thread-loader",
"peerDependencies": {
"webpack": "^2.0.0 || ^3.0.0 || ^4.0.0"
},
"pre-commit": "lint-staged",
"repository": {
"type": "git",
"url": "git+https://github.com/webpack-contrib/thread-loader.git"
},
"scripts": {
"appveyor:test": "npm run test",
"build": "cross-env NODE_ENV=production babel src -d dist --ignore 'src/**/*.test.js'",
"clean": "del-cli dist",
"lint": "eslint --cache src test",
"lint-staged": "lint-staged",
"prebuild": "npm run clean",
"prepare": "npm run build",
"release": "standard-version",
"security": "npm audit",
"start": "npm run build -- -w",
"test": "jest",
"test:coverage": "jest --collectCoverageFrom='src/**/*.js' --coverage",
"test:watch": "jest --watch",
"travis:coverage": "npm run test:coverage -- --runInBand",
"travis:lint": "npm run lint && npm run security",
"travis:test": "npm run test -- --runInBand",
"webpack-defaults": "webpack-defaults"
},
"version": "2.1.3"
}