[英]What is the best way to iterate or recurse through huge amounts of huge functions without exceeding the stack limit?

I have an application that I'm writing in Node.js which needs to make a lot of configuration and database calls in order to process user data. The issue I'm having is that after 11,800+ function calls Node will throw an error and exit the process.


The error says: RangeError: Maximum call stack size exceeded


I'm curious if anyone else has had this situation arise and to know how they handled this. I've already started to break up my code into a couple of extra worker files but even so each time I process a data node it needs to touch 2 databases (at most 25 calls to update various tables) and do a number of sanitization checks.


I am totally willing to admit that I'm possibly doing something non-optimal if that is the case but would appreciate some guidance if there is a more optimal manner.


Here is an example of the code I'm running on data:


app.post('/initspeaker', function(req, res) {
    // if the Admin ID is not present ignore
    if(req.body.xyzid!=config.adminid) {
        res.send( {} );

    var gcnt = 0, dbsize = 0, goutput = [], goutputdata = [], xyzuserdataCallers = [];

    xyz.loadbatchfile( xyz.getbatchurl("speakers", "csv"), function(data) {
        var parsed = csv.parse(data);
        console.log("lexicon", parsed[0]);

        for(var i=1;i<parsed.length;i++) {
            if(typeof parsed[i][0] != 'undefined' && parsed[i][0]!='name') {
                var xyzevent = require('./lib/model/xyz_speaker').create(parsed[i], parsed[0]);
                xyzevent.isPresenter = true;
        dbsize = goutput.length;

        xyzuserdataCallers = [new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata(),
                                    new xyzuserdata()
        // insert all Scheduled Items into the DB                   
        for(var i=1;i<xyzuserdataCallers;i++) {


    var callback = function(data, func) {
        if(data && data!=8008) {
            if(gcnt>=dbsize) {
            } else {
        } else {

    // callback loop for fetching registrants for events from SMW
    var xyzuserdata = function() {};
    xyzuserdata.prototype.sendSpeakerData = function(data) {
        var thisfunc = this;

        if(data && data!=8008) {
            //console.log('creating user from data', gcnt, dbsize);
            var userdata = require('./lib/model/user').create(data.toObject());
            var speakerdata = userdata.toObject();
            speakerdata.uid = uuid.v1();
            speakerdata.isPresenter = true;

            couchdb.insert(speakerdata, config.couch.db.user, function($data) {
                if($data==false) {
                    // if this fails it is probably due to a UID colliding
                    console.log("*** trying user data again ***");
                    speakerdata.uid = uuid.v1();
                    arguments.callee( speakerdata );
                } else {
                    callback($data, thisfunc);
        } else {


A couple of classes and items are defined here that need some introduction:


  • I am using Express.js + hosted CouchDB and this is responding to a POST request
  • 我正在使用Express.js +托管的CouchDB,这是响应POST请求

  • There is a CSV parser class that loads a list of events which drives pulling speaker data
  • 有一个CSV解析器类可以加载一个驱动扬声器数据的事件列表

  • Each event can have n number of users (currently around 8K users for all events)
  • 每个活动可以有n个用户(目前所有活动的用户约为8K)

  • I'm using a pattern that loads all of the data/users before attempting to parse any of them
  • 我正在使用一种模式,在尝试解析任何数据/用户之前加载所有数据/用户

  • Each user loaded (external data source) is converted into an object I can use and also sanitized (strip slashes and such)
  • 每个加载的用户(外部数据源)都会转换为我可以使用的对象,也会被清理(条带斜线等)

  • Each user is then inserted into CouchDB
  • 然后将每个用户插入CouchDB

This code works in the app but after a while I get an error saying that over 11,800+ calls have been made and the app breaks. This isn't an error that contains a stack trace like one would see if it was code error, it is exiting due to the number of calls being done.


Again, any assistance/commentary/direction would be appreciated.


2 个解决方案



It looks like xyzuserdata.sendSpeakerData & callback are being used recursively in order to keep the DB calls sequential. At some point you run out of call stack...


There's several modules to make serial execution easier, like Step or Flow-JS.


Flow-JS even has a convenience function to apply a function serially over the elements of the array:


flow.serialForEach(goutput, xyzuserdata.sendSpeakerData, ...)

I wrote a small test program using flow.serialForEach, but unfortunately was able to get a Maximum call stack size exceeded error -- Looks like Flow-JS is using the call stack in a similar way to keep things in sync.

我使用flow.serialForEach编写了一个小测试程序,但遗憾的是能够获得超出最大调用堆栈大小的错误 - 看起来像Flow-JS以类似的方式使用调用堆栈来保持同步。

Another approach that doesn't build up the call stack is to avoid recursion and use setTimeout with a timeout value of 0 to schedule the callback call. See http://metaduck.com/post/2675027550/asynchronous-iteration-patterns-in-node-js


You could try replacing the callback call with


setTimeout(callback, 0, [$data, thisfunc])



Recursion is very useful for synchronizing async operations -- that's why it is used in flow.js etc.

递归对于同步异步操作非常有用 - 这就是它在flow.js等中使用的原因。

However if you want to process an unlimited number of elements in an array, or buffered stream, you will need to use node.js's event emitter.


in pseudo-ish-code:

 ee = eventemitter
 arr = A_very_long_array_to_process
 callback = callback_to_call_once_either_with_an_error_or_when_done

 // the worker function does everything
 processOne() {
      next = arr. shift();
   if( !arr )
      ee.emit ( 'finished' )

   process( function( err, response) {
      if( err )
         callback( err, response )
         ee.emit( 'done-one' )
    } );

 // here we process the final event that the worker will throw when done
 ee.on( 'finished', function() { callback( null, 'we processed the entire array!'); } );

 // here we say what to do after one thing has been processed
 ee.on( 'done-one', function() { processOne(); } );

 // here we get the ball rolling



在不使用大量内存的情况下显示大文件的最佳方法是什么? - What is the best way to display large files without using large amounts of memory? 在不阻塞UI的情况下迭代数组的最佳方法 - Best way to iterate over an array without blocking the UI 在不花费大量时间的情况下做出估算的最佳方法是什么? - What's the best way to do estimates without spending a lot of time? 是否可以在没有超出调用堆栈的情况下展平长度为700 000个条目的数组? - Is it possible to flatten an array that is 700 000 entries long without exceeding the call stack? 在SQL Server中使用大量数据执行操作的最佳方法是什么? - What is the best way to perform a manipulation with huge amounts of data in SQL Server? 在MATLAB中迭代矩阵列的最佳方法是什么? - What's the best way to iterate through columns of a matrix in MATLAB? 在经典的Asp VBScript中迭代数组的最佳方式是什么? - What is the best way to iterate through an array in Classic Asp VBScript? 迭代强类型泛型List 的最佳方法是什么? - What is the best way to iterate through a strongly-typed generic List? 如何在不占用大量RAM的情况下显示图像 - How to display images without taking up huge amounts of RAM 存储大量文本(进入数据库或作为文件?)的最佳实践是什么?压缩它的方法是什么? - What's the best practice for storing huge amounts of text (into a DB or as a file?), and what about compressing it?
粤ICP备14056181号  © 2014-2020 ITdaan.com