[英]What is the best way to iterate or recurse through huge amounts of huge functions without exceeding the stack limit?

I have an application that I'm writing in Node.js which needs to make a lot of configuration and database calls in order to process user data. The issue I'm having is that after 11,800+ function calls Node will throw an error and exit the process.


The error says: RangeError: Maximum call stack size exceeded


I'm curious if anyone else has had this situation arise and to know how they handled this. I've already started to break up my code into a couple of extra worker files but even so each time I process a data node it needs to touch 2 databases (at most 25 calls to update various tables) and do a number of sanitization checks.


I am totally willing to admit that I'm possibly doing something non-optimal if that is the case but would appreciate some guidance if there is a more optimal manner.


Here is an example of the code I'm running on data:


app.post('/initspeaker', function(req, res) {
    // if the Admin ID is not present ignore
    if(req.body.xyzid!=config.adminid) {
        res.send( {} );

    var gcnt = 0, dbsize = 0, goutput = [], goutputdata = [], xyzuserdataCallers = [];

    xyz.loadbatchfile( xyz.getbatchurl("speakers", "csv"), function(data) {
        var parsed = csv.parse(data);
        console.log("lexicon", parsed[0]);

        for(var i=1;i<parsed.length;i++) {
            if(typeof parsed[i][0] != 'undefined' && parsed[i][0]!='name') {
                var xyzevent = require('./lib/model/xyz_speaker').create(parsed[i], parsed[0]);
                xyzevent.isPresenter = true;
        dbsize = goutput.length;

        xyzuserdataCallers = [new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata()
        // insert all Scheduled Items into the DB                   
        for(var i=1;i<xyzuserdataCallers;i++) {


    var callback = function(data, func) {
        if(data && data!=8008) {
            if(gcnt>=dbsize) {
            } else {
        } else {

    // callback loop for fetching registrants for events from SMW
    var xyzuserdata = function() {};
    xyzuserdata.prototype.sendSpeakerData = function(data) {
        var thisfunc = this;

        if(data && data!=8008) {
            //console.log('creating user from data', gcnt, dbsize);
            var userdata = require('./lib/model/user').create(data.toObject());
            var speakerdata = userdata.toObject();
            speakerdata.uid = uuid.v1();
            speakerdata.isPresenter = true;

            couchdb.insert(speakerdata, config.couch.db.user, function($data) {
                if($data==false) {
                    // if this fails it is probably due to a UID colliding
                    console.log("*** trying user data again ***");
                    speakerdata.uid = uuid.v1();
                    arguments.callee( speakerdata );
                } else {
                    callback($data, thisfunc);
        } else {


A couple of classes and items are defined here that need some introduction:


  • I am using Express.js + hosted CouchDB and this is responding to a POST request
  • 我正在使用Express.js +托管的CouchDB,这是响应POST请求

  • There is a CSV parser class that loads a list of events which drives pulling speaker data
  • 有一个CSV解析器类可以加载一个驱动扬声器数据的事件列表

  • Each event can have n number of users (currently around 8K users for all events)
  • 每个活动可以有n个用户(目前所有活动的用户约为8K)

  • I'm using a pattern that loads all of the data/users before attempting to parse any of them
  • 我正在使用一种模式,在尝试解析任何数据/用户之前加载所有数据/用户

  • Each user loaded (external data source) is converted into an object I can use and also sanitized (strip slashes and such)
  • 每个加载的用户(外部数据源)都会转换为我可以使用的对象,也会被清理(条带斜线等)

  • Each user is then inserted into CouchDB
  • 然后将每个用户插入CouchDB

This code works in the app but after a while I get an error saying that over 11,800+ calls have been made and the app breaks. This isn't an error that contains a stack trace like one would see if it was code error, it is exiting due to the number of calls being done.


Again, any assistance/commentary/direction would be appreciated.


2 个解决方案



It looks like xyzuserdata.sendSpeakerData & callback are being used recursively in order to keep the DB calls sequential. At some point you run out of call stack...


There's several modules to make serial execution easier, like Step or Flow-JS.


Flow-JS even has a convenience function to apply a function serially over the elements of the array:


flow.serialForEach(goutput, xyzuserdata.sendSpeakerData, ...)

I wrote a small test program using flow.serialForEach, but unfortunately was able to get a Maximum call stack size exceeded error -- Looks like Flow-JS is using the call stack in a similar way to keep things in sync.

我使用flow.serialForEach编写了一个小测试程序,但遗憾的是能够获得超出最大调用堆栈大小的错误 - 看起来像Flow-JS以类似的方式使用调用堆栈来保持同步。

Another approach that doesn't build up the call stack is to avoid recursion and use setTimeout with a timeout value of 0 to schedule the callback call. See http://metaduck.com/post/2675027550/asynchronous-iteration-patterns-in-node-js


You could try replacing the callback call with


setTimeout(callback, 0, [$data, thisfunc])



Recursion is very useful for synchronizing async operations -- that's why it is used in flow.js etc.

递归对于同步异步操作非常有用 - 这就是它在flow.js等中使用的原因。

However if you want to process an unlimited number of elements in an array, or buffered stream, you will need to use node.js's event emitter.


in pseudo-ish-code:

 ee = eventemitter
 arr = A_very_long_array_to_process
 callback = callback_to_call_once_either_with_an_error_or_when_done

 // the worker function does everything
 processOne() {
      next = arr. shift();
   if( !arr )
      ee.emit ( 'finished' )

   process( function( err, response) {
      if( err )
         callback( err, response )
         ee.emit( 'done-one' )
    } );

 // here we process the final event that the worker will throw when done
 ee.on( 'finished', function() { callback( null, 'we processed the entire array!'); } );

 // here we say what to do after one thing has been processed
 ee.on( 'done-one', function() { processOne(); } );

 // here we get the ball rolling



© 2014-2019 ITdaan.com 粤ICP备14056181号