train-stream
A node train stream for brain.js
A small package that lets brain.js neural networks train on large datasets by streaming data in chunks, so you don't need to load everything into memory at once.
train-stream is a companion package for brain.js, a JavaScript library that lets you build and train neural networks in Node.js. The core problem it solves is training on large datasets that are too big to comfortably hold in memory all at once. Instead of loading every example into RAM before training begins, you feed data to the network incrementally through a stream.
At a high level, the package provides a TrainStream class that you attach to an existing brain.js neural network. You write data to the stream piece by piece, and the network learns from it as it arrives. Three main things control the flow: the neural network itself, a "flood callback" that re-populates the stream for each training iteration, and a "done training callback" that fires when the network finishes. The README notes you can also use a Transform stream to normalize or reformat data before it reaches the network.
The target user is a JavaScript or TypeScript developer already working with brain.js who needs to train on more data than memory allows. For example, if you have a large CSV file of customer interactions and want to train a model to classify sentiment, this stream lets you pipe rows through the network without loading the entire file at once. It works with brain.js's standard NeuralNetwork class as well as recurrent architectures like LSTM.
The project is notably small and focused, providing just the streaming layer on top of brain.js rather than reimplementing the machine learning logic itself. The README is fairly minimal, covering initialization and the two main usage patterns, with an example file linked for further reference.
Where it fits
- Train a brain.js neural network on a large CSV file by piping rows through a stream.
- Train a sentiment classifier on customer interaction data without loading everything into memory.
- Normalize or reformat data on the fly using a Transform stream before training.
- Train LSTM recurrent networks on large sequential datasets using incremental streaming.