This error can occur on Windows, particularly around socket operations. The error is translated from the Winsock error code WSAENOBUFS, 10055. The most common cause of this error is that Windows is configured for the default maximum of 5,000 in-use ports. This can be monitored by watching netstat or perfmon and can be changed with the MaxUserPort registry parameter.
A more advanced cause for this error is non-paged pool exhaustion. The paged and nonpaged pools are areas of memory for certain Windows kernel-mode allocations such as the Windows kernel itself (e.g. sockets, socket buffers, etc.), device drivers, etc. The nonpaged pool is particularly important as "it is the availability of nonpaged pool that determines how many processes, threads, and other such objects can be created." If these pools are exhausted, this can lead to crashes, poor performance, application problems, and paging.
If the system is using the /3GB mode, this comes at a cost of taking memory away from the kernel, including paged and non-paged pools (see the "Boot.ini Settings and Maximum Kernel Memory Space Sizes" section in this Microsoft Exchange article).
To determine if this is the proximate cause, use perfmon to monitor the Memory\Pool Nonpaged Bytes counter. If this is hitting the server's nonpaged pool limit (within a few MB since perfmon is sampling on an interval), then this is the cause of the problem. However, this proximate cause may not be the root cause since the nonpaged pool exhaustion may be due to a nonpaged pool leak. A nonpaged pool leak can be determined using Microsoft's poolmon.exe.
To increase kernel memory, lower the user-mode address space limit (/USERVA=X). The /3GB switch is effectively the same as /USERVA=3072 (don't ask me where the other 24MB went); for example, /USERVA=2800. This parameter would be used instead of /3GB. The documentation is not clear on how the additional space is allocated to the nonpaged pool and to what limits -- monitor your current and maximum nonpaged pool sizes with process explorer and work with Microsoft support to properly tune this value. A registry setting exists, NonPagedPoolSize, but again it's unclear how this plays with other limits. There also appears to be a PoolUsageMaximum value that controls when the memory manager starts to trim processes which may default to 80%.