Topic
  • 5 replies
  • Latest Post - ‏2015-03-11T20:50:57Z by laff
luvsurf
luvsurf
2 Posts

Pinned topic Does RDMA work when apps use O_DIRECT?

‏2013-06-05T07:42:02Z |

Hello,

Simple Question.

Does IB RDMA work when appliation use O_DIRECT?

Looks like GPFS client use IPoIB when we use directio.. So we are wondering whether there is any limitation in IB RDMA of GPFS about directio or not.

Many thanks,

  • ezhong
    ezhong
    33 Posts

    Re: Does RDMA work when apps use O_DIRECT?

    ‏2013-06-10T14:29:27Z  

    Hi. This is the answer from one of the GPFS RDMA developers:

    In order to use RDMA,  the memory for buffers on client and server need to be registered to IB.  This will be the case when IO is done to and from client and server pagepool.  O_DIRECT is requesting the IO be written directly to or read from directly from the user's application buffer on the client.  Since that buffer is not registered to IB, we don't use RDMA.

  • luvsurf
    luvsurf
    2 Posts

    Re: Does RDMA work when apps use O_DIRECT?

    ‏2013-06-10T15:49:33Z  
    • ezhong
    • ‏2013-06-10T14:29:27Z

    Hi. This is the answer from one of the GPFS RDMA developers:

    In order to use RDMA,  the memory for buffers on client and server need to be registered to IB.  This will be the case when IO is done to and from client and server pagepool.  O_DIRECT is requesting the IO be written directly to or read from directly from the user's application buffer on the client.  Since that buffer is not registered to IB, we don't use RDMA.

    Thank you very much for reply!

    I see.. That's reasonable reason for me.

    Thanks again.

  • laff
    laff
    14 Posts

    Re: Does RDMA work when apps use O_DIRECT?

    ‏2015-03-10T14:15:47Z  

    this information is outdated.. .The GPFS code was improved to use RDMA even for direct IOs or AIOs

    since GPFS 4.1 , DIO works now over RDMA 
    ...from our experiences , go at least with 4.1.0.5  . but ..check FAQ for your last recent patch level ..

    cheers

  • weiruan85
    weiruan85
    16 Posts

    Re: Does RDMA work when apps use O_DIRECT?

    ‏2015-03-11T01:09:11Z  
    • laff
    • ‏2015-03-10T14:15:47Z

    this information is outdated.. .The GPFS code was improved to use RDMA even for direct IOs or AIOs

    since GPFS 4.1 , DIO works now over RDMA 
    ...from our experiences , go at least with 4.1.0.5  . but ..check FAQ for your last recent patch level ..

    cheers

    Dear Laff:

       I have searched the infocenter, cant find the detail information about RDMA for DIO ?

  • laff
    laff
    14 Posts

    Re: Does RDMA work when apps use O_DIRECT?

    ‏2015-03-11T20:50:57Z  
    • weiruan85
    • ‏2015-03-11T01:09:11Z

    Dear Laff:

       I have searched the infocenter, cant find the detail information about RDMA for DIO ?

    hi,

    .. it  might not be listed in the info-center .. but I'm not aware about the  content .. I would assume, that also the old, outdated information, was not there as well ;-)
     

    I'll look for someone, who add this to the official documentation

    in addition.. you can easily verify it by your own.. use gpfsperf with -dio option .. and watch the messages are send all over RDMA

    cheers