Topic
  • No replies
PeteInOxford
PeteInOxford
1 Post

Pinned topic pthreads mutex construct memory leak?

‏2011-04-14T13:31:10Z |
Attached below is some test code "lifted" from a program we are running on z/OS which makes extensive use of Posix threads (the odd syntax error may have crept in during the "lift" process; apologies). Alas, whenever we use the mutex functions, a (heap) memory leak occurs. Can anyone cast any light on this? I've taken the (unusual) step of including the whole test case to save back-and-forward time. Anyone had this problem themselves?

Thanks in advance!

Pete & Cellina.


#ifndef __B_TYPES_H__   #define __B_TYPES_H__ 
/*********************************************************************************************************************/ 
/* INCLUDE Section                                                                                                   */ 
/*********************************************************************************************************************/ #define _OPEN_THREADS #define _OPEN_SYS   #ifdef MVS #include <sys/types.h> #endif   #include <memory.h> #include <errno.h> #include <stdio.h> #include <stdlib.h> #include <stdarg.h> #include <string.h> #include <time.h>     #ifdef MVS #include <unistd.h> #include <signal.h> #endif   
/********************************************************************************************************************/ 
/* Since threads are platform dependant, following definitions map the thread functions into the specific platform. */ 
/* Original implementation took the following decision:                                                             */ 
/*     - WIN32       -> uses the C - Run-Time Library to perform the thread operations.                             */ 
/*     - MVS - LINUX -> uses the POSIX implementation                                                               */ 
/********************************************************************************************************************/ #

if defined MVS || defined LINUX   #include <pthread.h> 
/** Defines the thread identifier as 'pthread_t', the type defined by POSIX definition                          */ #define THREAD_ID pthread_t 
/** Creates a new thread, with attributes specified by THREAD_ATTR, within a process. If THREAD_ATTR is NULL *  the API uses the default attributes. */ #define START_THREAD(ID,PROC,ARG) pthread_create(&(ID), &thrattr,PROC,(

void *)(ARG)) 
/** Defines the application's default thread declaration and sets the detach attribute activated.               */ #define THREAD_ATTR               pthread_attr_t thrattr;                               \ 

int detachstate=0;   
/** Thread attributes initialization. Does the following actions: *    1.- Initializes the thread attributes, THREAD_ATTR. *    2.- Defines the minimum stack size allocated for the created threads stack to 40000 bytes *    3.- Establish the priorities for the new thread. Set to __MEDIUM_WEIGHT. Means that each thread runs on *        a task. When the current thread exits, the task waits for another thread to do a pthread_create(). *        The new thread runs on that task. *    4.- Sets the detachstate attribute to TRUE. It is defined in THREAD_ATTR declaration. */ #define INIT_THREAD_ATTR()        pthread_attr_init(&thrattr);                          \ pthread_attr_setstacksize(&thrattr, 40000);           \ pthread_attr_setweight_np(&thrattr, __MEDIUM_WEIGHT); \ pthread_attr_setdetachstate(&thrattr, &detachstate)   
/** The function indicates to the implementation that storage for the thread <ID> can be reclaimed when that *  thread terminates. It also guaranteed that the resources associated to the thread will be freed immediately *  after the thread termination. It also prevents that the thread gets a synchronization while terminating. */ #define DETACH_THREAD(ID)         pthread_detach(&(ID))   
/** Terminates the calling thread. Any cancellation cleanup handlers that have been pushed and not yet popped *  are popped in the reverse order that they were pushed and then executed. That means, in case of bridge, i.e. *  that the 'anz_threads' variable will be decreased at that point, the associated resources released, ... */ #define END_THREAD()              pthread_exit(NULL);                                   \ 

return 
/** Defines the thread procedure return type as a pointer to void ( 4 bytes to store any type ).                */ #define THREADPROC_RETVALUE       

void *   #endif     

void *dummyFunction(

void *arg);     #endif     #include 
"btypes.h"   

int threadCounter = 0; pthread_mutex_t                   mutex; pthread_rwlock_t                  *rwMutex ; pthread_rwlockattr_t              *rwAttr ;   

int main(

int argc, char* argv[]) 
{   

int i,j,k=0; 

int iThreads = 150; 

int iFrequency = 1000; 

int iDuration = 50000; 

int ret = 0; pthread_t thid[5000]; 

int iCounter = 0; 

int iRetry = 0;   
/**** Initialize thread attributes****/ pthread_attr_t thrattr; 

int detachstate=0;   rwMutex = NULL; pthread_attr_init(&thrattr); pthread_attr_setstacksize(&thrattr, 40000); pthread_attr_setweight_np(&thrattr, __MEDIUM_WEIGHT); pthread_attr_setdetachstate(&thrattr, &detachstate);   
/* Initialize mutex */ pthread_mutex_init(&mutex, NULL); 
/* Initialize read/write lock */ rwAttr = (pthread_rwlockattr_t *) malloc ( sizeof(pthread_rwlockattr_t) ); ret = pthread_rwlockattr_init( rwAttr ); 

if ( ret ) printf(
" attr mutex not initialized\n"); ret = pthread_rwlockattr_setpshared( rwAttr , PTHREAD_PROCESS_PRIVATE ); 

if ( ret ) printf(
"not possible to set LOG to process private"); rwMutex = (pthread_rwlock_t *) malloc ( sizeof(pthread_rwlock_t) ); ret = pthread_rwlock_init(rwMutex, rwAttr); 

if ( ret ) printf(
"LOG  mutex not initialized\n");     printf(
"***************************************************************************\n"); printf(
"Test Tool for creating threads in USS\n"); printf(
"***************************************************************************\n");   
/*-----------------------------------------------------------------------------------------------------------------*/ 
/* Parse the arguments                                                                                             */ 
/*-----------------------------------------------------------------------------------------------------------------*/ 

for ( j=1; j< argc ; j=j+2 ) 
{ 

if ( strcmp ( argv[j], 
"-?") == 0) 
{ printf(
"------------------------------------------------------------------------\n"); printf(
" Tool for pthread create                                                \n"); printf(
" It generates dummy logs with all the possible future realloc log names \n"); printf(
" \n"); printf(
"USAGE:                                                                  \n"); printf(
"-T number of parallel threads                                           \n"); printf(
"-D iterations                                                           \n"); printf(
"------------------------------------------------------------------------\n");   

return 0; 
}   

if ( strcmp ( argv[j],
"-T") == 0) 
{ iThreads = atoi(argv[j+1]); printf(
"Number of parallel threads is [%d]\n",iThreads); 
}   

if ( strcmp ( argv[j],
"-D") == 0) 
{ iDuration = atoi(argv[j+1]); printf(
"Duration is %d\n",iDuration); 
} 
}     
/*************** START WORK **********************/ 

for ( k=0; k<iDuration; k++ )
{ 

for ( i=0; i<iThreads;i++)
{ ret = pthread_create(&(thid[i]), &thrattr,dummyFunction,(

void *)(i)); 

if ( ret != 0)
{ printf(
"Error in pthread_create function! System error is [%d]. Reason is [%s]\n",errno, strerror(errno)); 
} 
/* check the write lock works.... */ 

if ( i==50 ) 
{ 

if ( pthread_rwlock_wrlock( rwMutex ) != 0 )
{ printf(
"Error when trying to aquire the write mutex!!! Errno is %d\n",errno); 
} 

else printf(
"acquiring write mutex\n"); 

if ( pthread_rwlock_unlock( rwMutex ) != 0)
{ printf(
"Error when trying to release the write mutex!!! Errno is %d\n",errno); 
} 
}   
} 
/*  wait the threads end */ 

for ( i=0; i<iThreads;i++)
{ 

if ( pthread_join(thid[i],NULL) == 0 )
{ pthread_mutex_lock(&mutex); threadCounter--; printf(
"thread terminated [%d]\n",threadCounter); pthread_mutex_unlock(&mutex); 
} 

else
{ printf(
"Error in pthread_join function! System error is [%d]. Reason is [%s]\n",errno, strerror(errno)); 
} 
} 
}   printf(
"Going to close in 60 seconds"); sleep(60);   

return 0; 
}   

void *dummyFunction(

void *arg)
{   

char pchDate[25]; 

int j= *( (int*) arg); 

int ret = 0;   pthread_mutex_lock(&mutex); threadCounter++; printf(
"[%d] ",threadCounter); pthread_mutex_unlock(&mutex);   

if ( pthread_rwlock_rdlock( rwMutex ) != 0)
{ printf(
"Error when trying to adquire the read mutex!!! Errno is %d\n",errno); 
} 

else printf(
"...doing stuff for read lock..."); 

if ( pthread_rwlock_unlock( rwMutex ) != 0)
{ printf(
"Error when trying to release the read mutex!!! Errno is %d\n",errno); 
}   sleep( rand() % 3);   pthread_exit(NULL); 
}
Updated on 2011-05-17T07:55:16Z at 2011-05-17T07:55:16Z by Cel-lina
  • SystemAdmin
    SystemAdmin
    196 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-04-14T22:12:16Z  
    Could you include some additional information about your environment and how this test case fails?
    Specifically,

    o What is the OS level where you are running the test (uname -Ia)
    o What version of compiler did you use to compile this program (-qphsinfo)
    o What are the arguments to your program (or do you run with the defaults)
    o How exactly does the program fail? I ran with the defaults and it all seems to be fine for me
    o What are the limits for the id under which you run this program? (ulimit -a)
    o How much physical memory does your machine have?

    Btw, I compiled with "xlc -DMVS=1". Compiled fine, although there's no "bytypes.h" file.
  • DaveyC
    DaveyC
    56 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-04-15T00:06:22Z  
    Specify the following compiler runtime options:

    RPTSTG(ON),HEAPCHK(ON,1,0,10,10)

    When you program ends you will get a storage report and a CEEDUMP which will contain
    a heap trace showing elements that have been allocated but not freed.

    David Crayford
  • Cel-lina
    Cel-lina
    4 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-04-15T10:02:26Z  
    Chris,

    here the answers for your questions

    o What is the OS level where you are running the test (uname -Ia)

    I'm running the application as a MVS batch, under Z/OS v1R12. This means the application is started via a JCL, and I can see the process in the SDSF.

    o What version of compiler did you use to compile this program (-qphsinfo)

    I am using the c89 compiler and invoking it from USS. Is there a chance to see the specific c89 version from there? I tried with the standard options ( -v -version --v --version, etc.. ) and didn't work. I guess is the default that comes from z/os V1R12, as I don`t have any rights to set anything there.

    o What are the arguments to your program (or do you run with the defaults)

    With the defaults arguments you can see in the SDSF in the DA than the REAL value grows constantly. As the application is just creating a number of threads and waiting them to finish, this REAL value should be constant as well. - but it is not -
    If you play with the parameters and increase the loops iteration ( -D ) you will end up in a point that there is not more memory available. But to be honest, it takes a lot of time to me to get to this point. Unfortunately, in production it doesn't take too long, as the application is only restarted once a week and it has a lot of activity.
    o How exactly does the program fail? I ran with the defaults and it all seems to be fine for me

    There is a moment that the application is not able to create more threads. That is, the pthread_create function fails. The errno we get is Not enough memory available. We have detected that the fact we are using rwlock is directly related within the usage of the memory. I mean, if I don't use the read write lock mechanism then the REAL value is pretty constant and I always have enough memory to create the threads.
    I checked the pthread_rwlock functions documentation, I think the usage I am making is correct, but please let me know if there is something wrong or a better way to do it.

    o What are the limits for the id under which you run this program? (ulimit -a)

    Do you mean the region size for my application? We are currently using 240M, this parameter is specified into the JCL.

    o How much physical memory does your machine have?

    I have raised the question to some colleagues,as soon as I get the info I will post it to you.

    Please let me know if you need more details, any help is really appreciated.

    Thanks in advance!

    Cel-lina
  • Cel-lina
    Cel-lina
    4 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-04-15T10:53:05Z  
    David,

    The first thing we thought is that I was generating a memory leak with a malloc, so I ran a memory leak detector and checked the results. Unfortunately, no memory leaks -at least with malloc- .

    I also tried enabling the storage report , although not with this option. Please take a look to the report storage result. I know in this case I am not freeing the read/write lock, the pthread_rwlock_attr and the mutex, ( I could do it before exiting, but in production the application is killed, no exited correctly, that's another issue ) but this shouldn´t cause me all this travel, as I am only creating them once at the very beggining.
    I am running this application as a MVS batch. I can see the output in the SDSF, and with the command DA there are some interesting values. The REAL value grows constantly. I can't see this reflected in the storage report, but maybe you see something I haven't noticed yet.

    
    1CEE3DMP V1 R12.0: Normal termination occurred 
    
    while Storage Diagnostics were active.    04/15/11 11:55:34 AM            Page:    1 ASID: 0082   Job ID: JOB27025   Job name: XX2HTEST   Step name: EXECBRI    PID: 67174865   Parent PID: 1   User name: E95090 CEE3845I CEEDUMP Processing started. Information 
    
    for enclave main Information 
    
    for thread 27D8160000000000 Heap Storage Diagnostics Stg Addr  ID        Length      Entry         E Addr    E Offset    Load Mod 243F2140  00000000  00000828    CEEV#GH       23DD29D8  +00000000   CEEPLPKA realloc_name_buffer 241EFF20  +00000074   CEEEV003 setlocale     241F0B18  +0000014C   CEEEV003 tzset         2415E7E8  +000006D6   CEEEV003 _cinit        240AC960  +0000333C   CEEEV003 CEEZINV       23E08510  +00000D06   CEEPLPKA main          23C00328  -DC407CD8   TESTTH 243F2CC0  00000000  00000040    CEEV#GH       23DD29D8  +00000000   CEEPLPKA dllinit       23F65688  +000000BC   CEEEV003 CEEZIDT       23E07388  +0000089C   CEEPLPKA main          23C00328  -DC407D3C   TESTTH 243F2D00  00000000  00000040    CEEV#GH       23DD29D8  +00000000   CEEPLPKA CEEOPMI       23D582C0  +00000824   CEEPLPKA pthread_mutex_init 241A4BD8  +00000092   CEEEV003 main          23C00328  +00000116   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 CEEBBEXT      23C97288  +000001B8   CEEPLPKA 243F2D40  00000000  00000010    CEEV#GH       23DD29D8  +00000000   CEEPLPKA main          23C00328  +00000128   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 CEEBBEXT      23C97288  +000001B8   CEEPLPKA 243F2D50  00000000  00000010    CEEV#GH       23DD29D8  +00000000   CEEPLPKA main          23C00328  +000001BA   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 CEEBBEXT      23C97288  +000001B8   CEEPLPKA 243F2D60  00000000  00000040    CEEV#GH       23DD29D8  +00000000   CEEPLPKA CEEOPMI       23D582C0  +00000824   CEEPLPKA pthread_rwlock_init 241A3768  +00000092   CEEEV003 main          23C00328  +000001EE   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 CEEBBEXT      23C97288  +000001B8   CEEPLPKA 243F2DA0  00000000  00000480    CEEV#GH 23DD29D8  +00000000   CEEPLPKA CEEOPMIA      23D59430  +00000200   CEEPLPKA pthread_create 2432FAA0  +000001A8   CEEEV003 main          23C00328  +000004EE   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 CEEBBEXT      23C97288  +000001B8   CEEPLPKA 243F3220  00000000  00000040    CEEV#GH       23DD29D8  +00000000   CEEPLPKA CEEOPMI       23D582C0  +00000824   CEEPLPKA pthread_mutex_init 241A4BD8  +00000092   CEEEV003 pthread_create 2432FAA0  +00000220   CEEEV003 main          23C00328  +000004EE   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 1CEE3DMP V1 R12.0: Normal termination occurred 
    
    while Storage Diagnostics were active.    04/15/11 11:55:34 AM            Page:    2 ASID: 0082   Job ID: JOB27025   Job name: XX2HTEST   Step name: EXECBRI    PID: 67174865   Parent PID: 1   User name: E95090 CEEBBEXT      23C97288  +000001B8   CEEPLPKA 243F3260  00000000  00000040    CEEV#GH       23DD29D8  +00000000   CEEPLPKA CEEOPMI       23D582C0  +00000824   CEEPLPKA pthread_mutex_init 241A4BD8  +00000092   CEEEV003 pthread_create 2432FAA0  +000002B0   CEEEV003 main          23C00328  +000004EE   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 CEEBBEXT      23C97288  +000001B8   CEEPLPKA 243F32A0  00000000  00000040                23DD2A08  +00000000   CEEPLPKA CEEV#GH       23DD29D8  +00013BA4   CEEPLPKA CEEOPMI       23D582C0  +00000824   CEEPLPKA CEEOPC        23D38E18  +00000CD6   CEEPLPKA pthread_create 2432FAA0  +000006A6   CEEEV003 main          23C00328  +000004EE   TESTTH EDCZMINV      2424B9E6  -FF9C9E00   CEEEV003 CEEBBEXT      23C97288  +000001B8   CEEPLPKA Additional Language Specific Information: errno information : Thread Id .... 27D8160000000000 Errno ...... 113 Errnojr .... C01E0012 CEE3846I CEEDUMP Processing completed.
    

    Any ideas are really appreciated.
    Thanks in advance!

    Cel-lina
  • brataj
    brataj
    12 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-04-15T12:50:32Z  
    Perhaps the threads are hanging around longer than you expect. Each thread consumes resources (a call stack for example) that are not released until the thread terminates, so if you're creating a lot of them, memory usage will go up steadily.
  • SystemAdmin
    SystemAdmin
    196 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-04-15T17:45:37Z  
    • Cel-lina
    • ‏2011-04-15T10:02:26Z
    Chris,

    here the answers for your questions

    o What is the OS level where you are running the test (uname -Ia)

    I'm running the application as a MVS batch, under Z/OS v1R12. This means the application is started via a JCL, and I can see the process in the SDSF.

    o What version of compiler did you use to compile this program (-qphsinfo)

    I am using the c89 compiler and invoking it from USS. Is there a chance to see the specific c89 version from there? I tried with the standard options ( -v -version --v --version, etc.. ) and didn't work. I guess is the default that comes from z/os V1R12, as I don`t have any rights to set anything there.

    o What are the arguments to your program (or do you run with the defaults)

    With the defaults arguments you can see in the SDSF in the DA than the REAL value grows constantly. As the application is just creating a number of threads and waiting them to finish, this REAL value should be constant as well. - but it is not -
    If you play with the parameters and increase the loops iteration ( -D ) you will end up in a point that there is not more memory available. But to be honest, it takes a lot of time to me to get to this point. Unfortunately, in production it doesn't take too long, as the application is only restarted once a week and it has a lot of activity.
    o How exactly does the program fail? I ran with the defaults and it all seems to be fine for me

    There is a moment that the application is not able to create more threads. That is, the pthread_create function fails. The errno we get is Not enough memory available. We have detected that the fact we are using rwlock is directly related within the usage of the memory. I mean, if I don't use the read write lock mechanism then the REAL value is pretty constant and I always have enough memory to create the threads.
    I checked the pthread_rwlock functions documentation, I think the usage I am making is correct, but please let me know if there is something wrong or a better way to do it.

    o What are the limits for the id under which you run this program? (ulimit -a)

    Do you mean the region size for my application? We are currently using 240M, this parameter is specified into the JCL.

    o How much physical memory does your machine have?

    I have raised the question to some colleagues,as soon as I get the info I will post it to you.

    Please let me know if you need more details, any help is really appreciated.

    Thanks in advance!

    Cel-lina
    Thank you for the additional information. I was running the command from the USS command line.

    Your compiler is probably the default system one i.e. V1R12.

    FYI, to determine the version, you can use the -qphsinfo command with xlc, or with c89 it's -Wc,PHASEID (in batch it's just PHASEID).

    I'll try to run in batch and reproduce your problem.
  • SystemAdmin
    SystemAdmin
    196 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-05-08T15:42:22Z  
    Sorry for the delay in responding. I have talked to the LE experts, and they're not aware of any memory leaks involving pthreads. A system dump is probably needed to begin diagnosing the problem. Could you please open a PMR with LE level 2 support, and they will investigate the problem.
  • DaveyC
    DaveyC
    56 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-05-11T08:45:47Z  
    Sorry for the delay in responding. I have talked to the LE experts, and they're not aware of any memory leaks involving pthreads. A system dump is probably needed to begin diagnosing the problem. Could you please open a PMR with LE level 2 support, and they will investigate the problem.
    I would certqainly advise you to open a PMR. I ran your test case and monitored it using IBM APA and it's definately leaking memory. Your test case is pathological
    but that's not the point - it proves a resource leak.

    David Crayford
  • Cel-lina
    Cel-lina
    4 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-05-11T11:20:57Z  
    • DaveyC
    • ‏2011-05-11T08:45:47Z
    I would certqainly advise you to open a PMR. I ran your test case and monitored it using IBM APA and it's definately leaking memory. Your test case is pathological
    but that's not the point - it proves a resource leak.

    David Crayford
    David, Chris
    Thanks for the advice, I will give you the PMR number as soon as I get it ( someone in IBM Germany is going to open it, hopefully soon).

    Anyway, thanks a lot for your help, I will keep you updated.

    Cel-lina
  • SystemAdmin
    SystemAdmin
    196 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-05-13T12:30:04Z  
    Hi Pete and Cellina,
    I've been in touch with LE and C runtime level 2 support. Once they have the PMR number
    they will proceed with investigation.
  • Cel-lina
    Cel-lina
    4 Posts

    Re: pthreads mutex construct memory leak?

    ‏2011-05-17T07:55:16Z  
    Hi Pete and Cellina,
    I've been in touch with LE and C runtime level 2 support. Once they have the PMR number
    they will proceed with investigation.
    Hi Chris,

    The PMR number has been opened, number 64807,075,724.
    Unfortunately I don't have access to see the status of the incidence, but I think it is being investigated by a LE expert in Germany.