[英]What is the best way to iterate or recurse through huge amounts of huge functions without exceeding the stack limit?

I have an application that I'm writing in Node.js which needs to make a lot of configuration and database calls in order to process user data. The issue I'm having is that after 11,800+ function calls Node will throw an error and exit the process.


The error says: RangeError: Maximum call stack size exceeded


I'm curious if anyone else has had this situation arise and to know how they handled this. I've already started to break up my code into a couple of extra worker files but even so each time I process a data node it needs to touch 2 databases (at most 25 calls to update various tables) and do a number of sanitization checks.


I am totally willing to admit that I'm possibly doing something non-optimal if that is the case but would appreciate some guidance if there is a more optimal manner.


Here is an example of the code I'm running on data:


app.post('/initspeaker', function(req, res) {
    // if the Admin ID is not present ignore
    if(req.body.xyzid!=config.adminid) {
        res.send( {} );

    var gcnt = 0, dbsize = 0, goutput = [], goutputdata = [], xyzuserdataCallers = [];

    xyz.loadbatchfile( xyz.getbatchurl("speakers", "csv"), function(data) {
        var parsed = csv.parse(data);
        console.log("lexicon", parsed[0]);

        for(var i=1;i<parsed.length;i++) {
            if(typeof parsed[i][0] != 'undefined' && parsed[i][0]!='name') {
                var xyzevent = require('./lib/model/xyz_speaker').create(parsed[i], parsed[0]);
                xyzevent.isPresenter = true;
        dbsize = goutput.length;

        xyzuserdataCallers = [new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata()
        // insert all Scheduled Items into the DB                   
        for(var i=1;i<xyzuserdataCallers;i++) {


    var callback = function(data, func) {
        if(data && data!=8008) {
            if(gcnt>=dbsize) {
            } else {
        } else {

    // callback loop for fetching registrants for events from SMW
    var xyzuserdata = function() {};
    xyzuserdata.prototype.sendSpeakerData = function(data) {
        var thisfunc = this;

        if(data && data!=8008) {
            //console.log('creating user from data', gcnt, dbsize);
            var userdata = require('./lib/model/user').create(data.toObject());
            var speakerdata = userdata.toObject();
            speakerdata.uid = uuid.v1();
            speakerdata.isPresenter = true;

            couchdb.insert(speakerdata, config.couch.db.user, function($data) {
                if($data==false) {
                    // if this fails it is probably due to a UID colliding
                    console.log("*** trying user data again ***");
                    speakerdata.uid = uuid.v1();
                    arguments.callee( speakerdata );
                } else {
                    callback($data, thisfunc);
        } else {


A couple of classes and items are defined here that need some introduction:


  • I am using Express.js + hosted CouchDB and this is responding to a POST request
  • 我正在使用Express.js +托管的CouchDB,這是響應POST請求

  • There is a CSV parser class that loads a list of events which drives pulling speaker data
  • 有一個CSV解析器類可以加載一個驅動揚聲器數據的事件列表

  • Each event can have n number of users (currently around 8K users for all events)
  • 每個活動可以有n個用戶(目前所有活動的用戶約為8K)

  • I'm using a pattern that loads all of the data/users before attempting to parse any of them
  • 我正在使用一種模式,在嘗試解析任何數據/用戶之前加載所有數據/用戶

  • Each user loaded (external data source) is converted into an object I can use and also sanitized (strip slashes and such)
  • 每個加載的用戶(外部數據源)都會轉換為我可以使用的對象,也會被清理(條帶斜線等)

  • Each user is then inserted into CouchDB
  • 然后將每個用戶插入CouchDB

This code works in the app but after a while I get an error saying that over 11,800+ calls have been made and the app breaks. This isn't an error that contains a stack trace like one would see if it was code error, it is exiting due to the number of calls being done.


Again, any assistance/commentary/direction would be appreciated.


2 个解决方案



It looks like xyzuserdata.sendSpeakerData & callback are being used recursively in order to keep the DB calls sequential. At some point you run out of call stack...


There's several modules to make serial execution easier, like Step or Flow-JS.


Flow-JS even has a convenience function to apply a function serially over the elements of the array:


flow.serialForEach(goutput, xyzuserdata.sendSpeakerData, ...)

I wrote a small test program using flow.serialForEach, but unfortunately was able to get a Maximum call stack size exceeded error -- Looks like Flow-JS is using the call stack in a similar way to keep things in sync.

我使用flow.serialForEach編寫了一個小測試程序,但遺憾的是能夠獲得超出最大調用堆棧大小的錯誤 - 看起來像Flow-JS以類似的方式使用調用堆棧來保持同步。

Another approach that doesn't build up the call stack is to avoid recursion and use setTimeout with a timeout value of 0 to schedule the callback call. See http://metaduck.com/post/2675027550/asynchronous-iteration-patterns-in-node-js


You could try replacing the callback call with


setTimeout(callback, 0, [$data, thisfunc])



Recursion is very useful for synchronizing async operations -- that's why it is used in flow.js etc.

遞歸對於同步異步操作非常有用 - 這就是它在flow.js等中使用的原因。

However if you want to process an unlimited number of elements in an array, or buffered stream, you will need to use node.js's event emitter.


in pseudo-ish-code:

 ee = eventemitter
 arr = A_very_long_array_to_process
 callback = callback_to_call_once_either_with_an_error_or_when_done

 // the worker function does everything
 processOne() {
      next = arr. shift();
   if( !arr )
      ee.emit ( 'finished' )

   process( function( err, response) {
      if( err )
         callback( err, response )
         ee.emit( 'done-one' )
    } );

 // here we process the final event that the worker will throw when done
 ee.on( 'finished', function() { callback( null, 'we processed the entire array!'); } );

 // here we say what to do after one thing has been processed
 ee.on( 'done-one', function() { processOne(); } );

 // here we get the ball rolling



粤ICP备14056181号  © 2014-2021 ITdaan.com