Topic
  • 11 replies
  • Latest Post - ‏2014-02-17T08:01:40Z by Karex
Karex
Karex
7 Posts

Pinned topic ClearQuest REST API problem

‏2014-02-12T12:44:01Z |

Hello.

I want to send a query via REST to cq server using cUrl.

curl -D- --user user:pass -H 'Content-Type: application/json' 'http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500' > file.txt

Unfortunately it doesn't work... Response is:

HTTP/1.1 401 Unauthorized
Date: Wed, 12 Feb 2014 12:34:29 GMT
Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)
WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"
WWW-Authenticate: Basic realm="TN_STANDARD"
Content-Length: 52
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Content-Language: ko-KR

HTTP%20status=401&oauth_problem=token_not_authorized


Is it needed to perform also oAuth authentication? From browser I can get to the http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 with just giving my username and password. Is it impossible using cURL?

 

CQ 7.1.2.12

  • pdubovitsky
    pdubovitsky
    376 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-12T13:45:47Z  

    Hello,

    Basic authentication should work.
    Could it be some special character in the password that caused the issue? Could you try to enclose credentials in quotes, like 'user:pass'?

    Pavel

  • Karex
    Karex
    7 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-12T13:53:34Z  

    Hello,

    Basic authentication should work.
    Could it be some special character in the password that caused the issue? Could you try to enclose credentials in quotes, like 'user:pass'?

    Pavel

    Login contains dot, however it changes nothing...

     

    curl -D- --user 'u.ser:pass123' -H 'Content-Type: application/json' 'http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500' > file.txt

     

    HTTP/1.1 401 Unauthorized
    Date: Wed, 12 Feb 2014 13:52:08 GMT
    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)
    WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"
    WWW-Authenticate: Basic realm="TN_STANDARD"
    Content-Length: 52
    Content-Type: application/x-www-form-urlencoded;charset=UTF-8
    Content-Language: ko-KR
     
    HTTP%20status=401&oauth_problem=token_not_authorized

  • pdubovitsky
    pdubovitsky
    376 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-12T14:26:54Z  
    • Karex
    • ‏2014-02-12T13:53:34Z

    Login contains dot, however it changes nothing...

     

    curl -D- --user 'u.ser:pass123' -H 'Content-Type: application/json' 'http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500' > file.txt

     

    HTTP/1.1 401 Unauthorized
    Date: Wed, 12 Feb 2014 13:52:08 GMT
    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)
    WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"
    WWW-Authenticate: Basic realm="TN_STANDARD"
    Content-Length: 52
    Content-Type: application/x-www-form-urlencoded;charset=UTF-8
    Content-Language: ko-KR
     
    HTTP%20status=401&oauth_problem=token_not_authorized

    401 means that credentials are not accepted. Usually, it is wrong username/password.
    It looks like you are using 7.1.x release. Could you check correspondent messages in the WAS cmprofile logs?
    {RATIONALSDLC_DIR}\common\CM\profiles\cmprofile\logs\server1\SystemOut.log

    It might provide some additional information.

    Pavel
     

  • Karex
    Karex
    7 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-12T14:53:41Z  

    401 means that credentials are not accepted. Usually, it is wrong username/password.
    It looks like you are using 7.1.x release. Could you check correspondent messages in the WAS cmprofile logs?
    {RATIONALSDLC_DIR}\common\CM\profiles\cmprofile\logs\server1\SystemOut.log

    It might provide some additional information.

    Pavel
     

    It is not my server. I'm trying to get some information from it. Administrators are in "far away land" and aren't very responsive. I should figure it out alone or forget about it.

    username and password are the same as I give them in browser. In my oppinion browser is somehow disabling OAuth, while cUrl tries to login using my login and password as OAuth parameters, not using Basic Authentication.

  • pdubovitsky
    pdubovitsky
    376 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-12T15:03:02Z  
    • Karex
    • ‏2014-02-12T14:53:41Z

    It is not my server. I'm trying to get some information from it. Administrators are in "far away land" and aren't very responsive. I should figure it out alone or forget about it.

    username and password are the same as I give them in browser. In my oppinion browser is somehow disabling OAuth, while cUrl tries to login using my login and password as OAuth parameters, not using Basic Authentication.

    It would not hurt to ask admins for log files.
    In interim time, you can take an IP traces using tcpdump or wireshark when you sending this request from a browser and when you are using curl. Comparing request headers could give you some information too. You can also try using curl from a different host.

    Pavel

  • DonaldN
    DonaldN
    287 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-13T00:03:58Z  

    You should use the HTTP header for the Basic Authentication. For the sample 'user:pass', instead of using -u 'user:pass', you should use -H 'Authorization: Basic dXNlcjpwYXNz' (Base64 encoded). So the simplest form for you to access CQWeb using cURL is:

    curl -H 'Authorization: Basic dXNlcjpwYXNz' 'http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500'

    You should get a feed as the response. To get a JSON response, add the HTTP Accept header: (I think you mistakenly used Content-Type)

    curl -H 'Authorization: Basic dXNlcjpwYXNz' -H 'Accept: application/json' 'http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500'

    Since the current CQWeb supports OSLC 2.0, it's better to specify as such. If you also like to dump the response header, the complete and proper command should be like:

    curl -H 'Authorization: Basic dXNlcjpwYXNz' -H 'Accept: application/json' -H 'OSLC-Core-Version: 2.0'
    'http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500' -D file.txt

  • Karex
    Karex
    7 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-13T07:30:12Z  

    Funny thing. Today I cannot open this page even in browser. Response is the same as for cUrl:

    HTTP/1.1 401 Unauthorized
    Date: Thu, 13 Feb 2014 07:22:26 GMT
    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)
    WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"
    WWW-Authenticate: Basic realm="TN_STANDARD"
    Content-Length: 52
    Keep-Alive: timeout=15, max=99
    Connection: Keep-Alive
    Content-Type: application/x-www-form-urlencoded;charset=UTF-8
    Content-Language: ko-KR
     

    =================EDIT==============

     

    Now I can open http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 from browser. Request sent is:

    GET /cqweb/oslc/repo/TN_STANDARD/db/D01/query/79033663?oslc_cm.pageSize=500 HTTP/1.1

    Host: 165.213.147.58

    User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0

    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

    Accept-Language: en-US,en;q=0.5

    Accept-Encoding: gzip, deflate

    Connection: keep-alive

    Authorization: Basic asddasdasdasdasdasdsadsa==

     

    Normal cUrl request is:

    GET /cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 HTTP/1.1

    Authorization: Basic asddasdasdasdasdasdsadsa==

    User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15

    Host: 165.213.147.58

    Accept: */*

     

    After changing cUrl to

    curl -D- -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --user-agent 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0' -H 'Authorization: Basic asddasdasdasdasdasdsadsa==' http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 > file.txt

    Request is:

    GET /cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 HTTP/1.1

    User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0

    Host: 165.213.147.58

    Connection: keep-alive

    Accept-Encoding: gzip, deflate

    Accept-Language: en-US,en;q=0.5

    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

    Authorization: Basic asddasdasdasdasdasdsadsa==

     

    So it is the same as from browser, but it doesn't work... Still the same 401 response:

    HTTP/1.1 401 Unauthorized
    Date: Thu, 13 Feb 2014 12:59:33 GMT
    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)
    WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"
    WWW-Authenticate: Basic realm="TN_STANDARD"
    Content-Length: 52
    Keep-Alive: timeout=15, max=100
    Connection: Keep-Alive
    Content-Type: application/x-www-form-urlencoded;charset=UTF-8
    Content-Language: ko-KR
     

  • DonaldN
    DonaldN
    287 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-13T22:59:02Z  
    • Karex
    • ‏2014-02-13T07:30:12Z

    Funny thing. Today I cannot open this page even in browser. Response is the same as for cUrl:

    HTTP/1.1 401 Unauthorized
    Date: Thu, 13 Feb 2014 07:22:26 GMT
    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)
    WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"
    WWW-Authenticate: Basic realm="TN_STANDARD"
    Content-Length: 52
    Keep-Alive: timeout=15, max=99
    Connection: Keep-Alive
    Content-Type: application/x-www-form-urlencoded;charset=UTF-8
    Content-Language: ko-KR
     

    =================EDIT==============

     

    Now I can open http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 from browser. Request sent is:

    GET /cqweb/oslc/repo/TN_STANDARD/db/D01/query/79033663?oslc_cm.pageSize=500 HTTP/1.1

    Host: 165.213.147.58

    User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0

    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

    Accept-Language: en-US,en;q=0.5

    Accept-Encoding: gzip, deflate

    Connection: keep-alive

    Authorization: Basic asddasdasdasdasdasdsadsa==

     

    Normal cUrl request is:

    GET /cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 HTTP/1.1

    Authorization: Basic asddasdasdasdasdasdsadsa==

    User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15

    Host: 165.213.147.58

    Accept: */*

     

    After changing cUrl to

    curl -D- -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --user-agent 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0' -H 'Authorization: Basic asddasdasdasdasdasdsadsa==' http://165.213.147.58/cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 > file.txt

    Request is:

    GET /cqweb/oslc/repo/TN_STANDARD/db/DB01/query/79033663?oslc_cm.pageSize=500 HTTP/1.1

    User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0

    Host: 165.213.147.58

    Connection: keep-alive

    Accept-Encoding: gzip, deflate

    Accept-Language: en-US,en;q=0.5

    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

    Authorization: Basic asddasdasdasdasdasdsadsa==

     

    So it is the same as from browser, but it doesn't work... Still the same 401 response:

    HTTP/1.1 401 Unauthorized
    Date: Thu, 13 Feb 2014 12:59:33 GMT
    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)
    WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"
    WWW-Authenticate: Basic realm="TN_STANDARD"
    Content-Length: 52
    Keep-Alive: timeout=15, max=100
    Connection: Keep-Alive
    Content-Type: application/x-www-form-urlencoded;charset=UTF-8
    Content-Language: ko-KR
     

    At this stage, it's quite hard to continue troubleshooting without knowing the server side of things. Since you cannot gain access to the server itself, I would suggest you use a different OSLC client such as the Firefox add-on RESTClient to verify all the HTTP headers that you are using before passing on to cURL. Bottom line is, it works for me when I use the command in my previous post.

    Another possibility is that the CQWeb server has not been configured to output XML format (or formats other than HTML), but that should easily be verified.

  • Karex
    Karex
    7 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-14T07:48:43Z  
    • DonaldN
    • ‏2014-02-13T22:59:02Z

    At this stage, it's quite hard to continue troubleshooting without knowing the server side of things. Since you cannot gain access to the server itself, I would suggest you use a different OSLC client such as the Firefox add-on RESTClient to verify all the HTTP headers that you are using before passing on to cURL. Bottom line is, it works for me when I use the command in my previous post.

    Another possibility is that the CQWeb server has not been configured to output XML format (or formats other than HTML), but that should easily be verified.

    I've created a loop in AutoIt scripting language in which, using WinHttp I request response from the server. Result of this test is very interesting.

    Most of tries are timeouts (after 60 seconds)

    Sometimes, there's a response that is correct. According to wireshark - response is always after about 30 seconds from request.

    Sometimes, there's a response 409 - Conflict.

     

    HTTP/1.1 409 Conflict

    Date: Fri, 14 Feb 2014 07:11:00 GMT

    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)

    X-Frame-Options: SAMEORIGIN

    Set-Cookie: JSESSIONID=0000v8iNVaeVMjaaZizYfMXEzDK:-1; Path=/

    Expires: Thu, 01 Dec 1994 16:00:00 GMT

    Cache-Control: no-cache="set-cookie, set-cookie2"

    Keep-Alive: timeout=15, max=100

    Connection: Keep-Alive

    Transfer-Encoding: chunked

    Content-Type: application/json; charset=UTF-8

    Content-Language: ko-KR

     

    {"oslc_cm:message":"CRVSV0072E \uc694\uccad\uc744 \ucc98\ub9ac\ud558\ub294 \uc911\uc5d0 \uc11c\ubc84 \uc624\ub958\uac00 \ubc1c\uc0dd\ud588\uc2b5\ub2c8\ub2e4.","oslc_cm:statusCode":"409","reasonCode":"conflict","nestedExceptions":[{"oslc_cm:message":"CRVSV1090I RPC server has unexpectedly exited. Any un-saved work may has been lost.","oslc_cm:statusCode":"409","reasonCode":"conflict","nestedExceptions":[{"oslc_cm:message":"","oslc_cm:statusCode":"409"}]}]}

     

    When I open the requested page in browser response comes in about 2 seconds (according to firebug).

     

    Wireshark shows that after opening page in browser, there are also started few TCP connections (please see file CQ1.png)

    With connection started by AutoIt there's much less connections. Why is that? (please see file CQ2.png)

    Attachments

  • DonaldN
    DonaldN
    287 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-16T23:27:43Z  
    • Karex
    • ‏2014-02-14T07:48:43Z

    I've created a loop in AutoIt scripting language in which, using WinHttp I request response from the server. Result of this test is very interesting.

    Most of tries are timeouts (after 60 seconds)

    Sometimes, there's a response that is correct. According to wireshark - response is always after about 30 seconds from request.

    Sometimes, there's a response 409 - Conflict.

     

    HTTP/1.1 409 Conflict

    Date: Fri, 14 Feb 2014 07:11:00 GMT

    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)

    X-Frame-Options: SAMEORIGIN

    Set-Cookie: JSESSIONID=0000v8iNVaeVMjaaZizYfMXEzDK:-1; Path=/

    Expires: Thu, 01 Dec 1994 16:00:00 GMT

    Cache-Control: no-cache="set-cookie, set-cookie2"

    Keep-Alive: timeout=15, max=100

    Connection: Keep-Alive

    Transfer-Encoding: chunked

    Content-Type: application/json; charset=UTF-8

    Content-Language: ko-KR

     

    {"oslc_cm:message":"CRVSV0072E \uc694\uccad\uc744 \ucc98\ub9ac\ud558\ub294 \uc911\uc5d0 \uc11c\ubc84 \uc624\ub958\uac00 \ubc1c\uc0dd\ud588\uc2b5\ub2c8\ub2e4.","oslc_cm:statusCode":"409","reasonCode":"conflict","nestedExceptions":[{"oslc_cm:message":"CRVSV1090I RPC server has unexpectedly exited. Any un-saved work may has been lost.","oslc_cm:statusCode":"409","reasonCode":"conflict","nestedExceptions":[{"oslc_cm:message":"","oslc_cm:statusCode":"409"}]}]}

     

    When I open the requested page in browser response comes in about 2 seconds (according to firebug).

     

    Wireshark shows that after opening page in browser, there are also started few TCP connections (please see file CQ1.png)

    With connection started by AutoIt there's much less connections. Why is that? (please see file CQ2.png)

    Always keep in mind that a script can behave differently to a browser. Sometime even IE can behave differently to Firefox. A browser will take care everything (HTTP return code, HTTP headers, cookies and etc) behind the scene while you need to deal with every bit of information in a script. Unless you use a script to "simulate" a browser, don't expect that you will always get the same result as a browser would.

    As for the 30 second time gap (possibly a timeout), it could be caused by the way the script (AutoIt) interacts with network (DNS or something else).

    To be honest, it is way out of the scope of ClearQuest already. If you can get correct results when using a browser, but not with a script, the fault is likely with the client, not the server.

  • Karex
    Karex
    7 Posts

    Re: ClearQuest REST API problem

    ‏2014-02-17T08:01:40Z  
    • DonaldN
    • ‏2014-02-16T23:27:43Z

    Always keep in mind that a script can behave differently to a browser. Sometime even IE can behave differently to Firefox. A browser will take care everything (HTTP return code, HTTP headers, cookies and etc) behind the scene while you need to deal with every bit of information in a script. Unless you use a script to "simulate" a browser, don't expect that you will always get the same result as a browser would.

    As for the 30 second time gap (possibly a timeout), it could be caused by the way the script (AutoIt) interacts with network (DNS or something else).

    To be honest, it is way out of the scope of ClearQuest already. If you can get correct results when using a browser, but not with a script, the fault is likely with the client, not the server.

    Today I've received such a response when using Mozilla:

    HTTP/1.1 401 Unauthorized

    Date: Mon, 17 Feb 2014 07:57:53 GMT

    Server: IBM_HTTP_Server/6.1.0.25 Apache/2.0.47 (Win32)

    WWW-Authenticate: OAuth realm="TN_STANDARD", oauth_problem="token_not_authorized"

    WWW-Authenticate: Basic realm="TN_STANDARD"

    Content-Length: 52

    Keep-Alive: timeout=15, max=100

    Connection: Keep-Alive

    Content-Type: application/x-www-form-urlencoded;charset=UTF-8

    Content-Language: ko-KR

     

    and I receive it all the time...

    Request:

    GET /cqweb/oslc/repo/TN_STANDARD/db/D01/record/?rcm.name=550898 HTTP/1.1

    Host: 165.213.147.58 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0

    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

    Accept-Language: pl,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate 

    Connection: keep-alive Authorization: Basic dsadsadasdsadas=

     

    I can login to web panel at 165.213.147.58 without any problems...