Topic
15 replies Latest Post - ‏2012-09-24T19:35:02Z by GregReid
joshuawhite929
joshuawhite929
52 Posts
ACCEPTED ANSWER

Pinned topic xscmd - insufficient or empty credentials.

‏2012-03-01T20:23:14Z |
I currently have catalog servers and container servers running within WebSphere Application Server. I would like to run the "xscmd" utility from the command line, but I am not clear on the necessary setup to authenticate with the remote servers. Every option I have tried, I recieve a "... insufficient or empty credentials" error message.

Locally, I have WXS installed standalone and I also have it installed within a WAS container. Can someone please provide an example of how to successfully run either xscmd.bat file to connect to a remote WAS based container? It doesn't seem to be as simple as providing a username/password.

Thanks,

Joshua
Updated on 2012-09-24T19:35:02Z at 2012-09-24T19:35:02Z by GregReid
  • jhanders
    jhanders
    257 Posts
    ACCEPTED ANSWER

    Re: xscmd - insufficient or empty credentials.

    ‏2012-03-06T13:54:48Z  in response to joshuawhite929
    Joshua,

    In WebSphere if you have authentication enabled you will need to update the PROFILE/properties/sas.client.props file. Change the com.ibm.CORBA.loginSource property from prompt to properties and then provide the user ID and password. An example of the properties in the PROFILE/properties/sas.client.props file follows:

    com.ibm.CORBA.loginSource=properties
    1. RMI/IIOP user identity
    com.ibm.CORBA.loginUserid=Admin
    com.ibm.CORBA.loginPassword=xxxxxx

    I believe that will take care of your problem. If it doesn't work let me know.

    Jared Anderson
    • joshuawhite929
      joshuawhite929
      52 Posts
      ACCEPTED ANSWER

      Re: xscmd - insufficient or empty credentials.

      ‏2012-05-31T19:07:34Z  in response to jhanders
      Jared,

      I tried making the three changes you mentioned, but I still get the same error message. Is this something else that I am missing?

      -Joshua
      • jhanders
        jhanders
        257 Posts
        ACCEPTED ANSWER

        Re: xscmd - insufficient or empty credentials.

        ‏2012-06-30T17:27:25Z  in response to joshuawhite929
        Joshua,

        What version of the product are you using. If you are using the 7.1.1 GA version, you should consider trying the first fix pack for 7.1.1 to see if it resolves you problem. Also 8.5 is also now available which may also have a fix for this problem. If neither resolve the problem you likely should open a PMR for the issue.

        I hope that helps

        Jared Anderson
        • mcgarvey@us.ibm.com
          mcgarvey@us.ibm.com
          6 Posts
          ACCEPTED ANSWER

          Re: xscmd - insufficient or empty credentials.

          ‏2012-07-02T14:10:20Z  in response to jhanders
          When using xscmd to communicate with catalog and container servers running under WAS, you must run xscmd from a directory under the WAS installation directory. In general it will not work to run xscmd from your standalone XS installation, because of additional security credentials required by WAS. Does this help?
          • joshuawhite929
            joshuawhite929
            52 Posts
            ACCEPTED ANSWER

            Re: xscmd - insufficient or empty credentials.

            ‏2012-07-10T12:34:41Z  in response to mcgarvey@us.ibm.com
            John,

            Just to clarify, my question is around using my local WAS based installation of WXS to talk to a remote WAS based installation of WXS.

            -Joshua
            • SystemAdmin
              SystemAdmin
              1485 Posts
              ACCEPTED ANSWER

              Re: xscmd - insufficient or empty credentials.

              ‏2012-09-04T13:55:23Z  in response to joshuawhite929
              Joshua,

              What versions of WAS and WXS that you are using? What security do you have enabled? WAS and/or WXS security? Admin and/or application security?

              I'm trying to recreate your scenario. I am seeing some issues that development will investigate but I haven't seen your specific exception.

              Thanks,

              Eric
              • GregReid
                GregReid
                6 Posts
                ACCEPTED ANSWER

                Re: xscmd - insufficient or empty credentials.

                ‏2012-09-21T18:57:56Z  in response to SystemAdmin
                I'm having the same or similar problem. Perhaps some more details will help in nailing this down.

                On my Windows 7 Professional 64-bit laptop, I've installed WAS ND 7.0.0.23, then WXS 7.1.1.1 client+server on top of it. I've configured two WXS catalogs and four containers to run in separate WAS appservers within the WAS cell. The WAS console is secured with wasadmin/wasadmin. Application security is off, J2EE security is off, and WXS security is off.

                Everything starts up fine, and I can even configure dynacache on another WAS cell to successfully put stuff into the grid. But I can't do a simple showMapSizes command to see what's in there.

                When I try to run xscmd.bat from the my <was_install_root>/profiles/<appserverprofile>/bin directory, the xscmd fails as follows:



                C:\IBM\WebSphere\AppServerV7withWXS711\profiles\AppSrv01\bin>xscmd -c showMapSizes -cep GREG-W500:2814 -user wasadmin -pwd wasadmin
                Starting at: 2012-09-21 14:30:51.151

                CWXSI0068I: Executing command: showMapSizes
                SERVER (id=cc2ee7e, host=GREG-W500) TRACE START:
                javax.management.JMRuntimeException: ADMN0022E: Access is denied for the getActiveCatalogServerNames operation on QuorumManager MBean because of insufficient or empty credentials.
                at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2378)
                at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2185)
                at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2079)
                at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2052)
                at com.ibm.ws.management.AdminServiceImpl.getAttribute(AdminServiceImpl.java:851)
                at com.ibm.ws.management.remote.AdminServiceForwarder.getAttribute(AdminServiceForwarder.java:282)
                at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
                at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
                at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
                at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
                at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
                at javax.management.remote.rmi._RMIConnectionImpl_Tie.getAttribute(_RMIConnectionImpl_Tie.java:577)
                at javax.management.remote.rmi._RMIConnectionImpl_Tie._invoke(_RMIConnectionImpl_Tie.java:98)
                at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:623)
                at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:476)
                at com.ibm.rmi.iiop.ORB.process(ORB.java:518)
                at com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
                at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2880)
                at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2753)
                at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
                at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
                at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1613)
                SERVER (id=cc2ee7e, host=GREG-W500) TRACE END.

                Ending at: 2012-09-21 14:31:01.949


                In the SystemOut.log file of my first/primary catalog server, I see the following at the same time as my xscmd error:



                9/21/12 14:30:59:533 EDT 0000000b ObjectGridSer W CWOBJ1316W: This non-secure server received a client request containing credential information. The credential information will be ignored by this server.
                9/21/12 14:31:01:752 EDT 00000034 RoleBasedAuth A SECJ0305I: The role-based authorization check failed for admin-authz operation QuorumManager:getActiveCatalogServerNames. The user UNAUTHENTICATED (unique ID: unauthenticated) was not granted any of the following required roles: deployer, operator, configurator, monitor, administrator, adminsecuritymanager, auditor.
                9/21/12 14:31:01:920 EDT 0000000b misc W ServerCommunicatorAdmin reqIncoming The server has decided to close this client connection.


                I've tried the earlier suggestion of updating the sas.client.props to hold my authentication userid/pswd, and restarted all servers to make sure it was picked up, but with no change in my symptoms. I'm guessing that I need to specifically grant "administrator" role to some id in my WAS topology, but don't really know what's needed.
                Any ideas?

                Thanks,
                Greg
                • td_w
                  td_w
                  3 Posts
                  ACCEPTED ANSWER

                  Re: xscmd - insufficient or empty credentials.

                  ‏2012-09-22T05:24:24Z  in response to GregReid
                  Hello,

                  if all of catalog servers and container servers are running on WAS. You need to make sure to set Message Layer Authentication to be Supported at Inbound communication of RMI/IIOP Security under Global Security in Admin console.

                  Thanks,
                  • GregReid
                    GregReid
                    6 Posts
                    ACCEPTED ANSWER

                    Re: xscmd - insufficient or empty credentials.

                    ‏2012-09-24T13:17:49Z  in response to td_w
                    Thanks for the suggestion, "td", but this appears to be already set by default. I'm attaching a screenshot of that area of my WAS7 cell where the WXS catalog and container servers are running. I didn't explicitly enable anything here; it was already set this way.

                    But here's something I just found in the SystemOut.log of my catalog servers starting up:

                    9/24/12 9:02:57:692 EDT 00000000 distSecurityC I securityServiceStarted is true
                    9/24/12 9:02:57:716 EDT 00000000 distSecurityC I SECJ0243I: Security service started successfully
                    9/24/12 9:02:57:757 EDT 00000000 distSecurityC I SECJ0210I: Security enabled true

                    I have NOT enabled WXS security, and the InfoCenter (and the comments in the sample server properties files) says that security is OFF by default. So what's this message about? Yes, my WAS admin global security is enabled, but all of the other security (J2EE, application, and WXS) should be disabled. I'm going to try providing a server.properties file for my catalog server startup, and explicitly set securityEnabled=false in that file.

                    Greg
                    • jhanders
                      jhanders
                      257 Posts
                      ACCEPTED ANSWER

                      Re: xscmd - insufficient or empty credentials.

                      ‏2012-09-24T13:22:08Z  in response to GregReid
                      If you have admin security enabled this is the message that you will see in a WebSphere Application Server environment. The WebSphere security service is started. If eXtreme Scale security was enabled you would see a CWOBJ message stating so.
                    • GregReid
                      GregReid
                      6 Posts
                      ACCEPTED ANSWER

                      Re: xscmd - insufficient or empty credentials.

                      ‏2012-09-24T13:39:10Z  in response to GregReid
                      OK, so I added a

                      -Dobjectgrid.server.props=C:\IBM\WebSphere\AppServerV7withWXS711\WXS_Config\GridA\Cat1_server.properties

                      to the generic JVM arguments of my Cat1 appserver, and similarly Cat2_server.properties for my Cat2 appserver. In those server properties files, I explicitly set securityEnabled=false, then restarted my catalog cluster. But I still see the same

                      9/24/12 9:29:47:107 EDT 00000000 distSecurityC I securityServiceStarted is true
                      9/24/12 9:29:47:133 EDT 00000000 distSecurityC I SECJ0243I: Security service started successfully
                      9/24/12 9:29:47:164 EDT 00000000 distSecurityC I SECJ0210I: Security enabled true

                      during the appserver startup, so this is apparently not related to WXS security. And I still get the same problem trying to run an xscmd showMapSizes against my catalogs.

                      FWIW,I just tried omitting the -user wasadmin and -pwd wasadmin from my xscmd, and surprisingly get exactly the same error logged in the Cat1's SystemOut.log. This seems to indicate that my -user and -pwd (when I DO supply them) aren't being conveyed properly with the xscmd syntax.

                      9/24/12 9:30:59:022 EDT 00000031 RoleBasedAuth A SECJ0305I: The role-based authorization check failed for admin-authz operation QuorumManager:getActiveCatalogServerNames. The user UNAUTHENTICATED (unique ID: unauthenticated) was not granted any of the following required roles: deployer, operator, configurator, monitor, administrator, adminsecuritymanager, auditor.
                      9/24/12 9:31:00:056 EDT 00000031 DMAdapter I com.ibm.ws.ffdc.impl.DMAdapter getAnalysisEngine FFDC1009I: Analysis Engine using data base: C:\IBM\WebSphere\AppServerV7withWXS711\profiles\AppSrv01\properties\logbr\ffdc\adv\ffdcdb.xml
                      9/24/12 9:31:00:084 EDT 00000031 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on C:\IBM\WebSphere\AppServerV7withWXS711\profiles\AppSrv01\logs\ffdc\WXS_GridA_Cat1_fe80fe8_12.09.24_09.30.59.9316444135332346713734.txt com.ibm.ws.management.AdminServiceImpl.invoke 422
                      9/24/12 9:31:00:449 EDT 00000031 misc W ServerCommunicatorAdmin reqIncoming The server has decided to close this client connection.

                      The ffdc log referenced above shows exactly the same traceback as I see on the xscmd client side:

                      9/24/12 9:30:59:985 EDT FFDC Exception:javax.management.JMRuntimeException SourceId:com.ibm.ws.management.AdminServiceImpl.invoke ProbeId:422 Reporter:com.ibm.ws.management.AdminServiceImpl@2a4f2a4f
                      javax.management.JMRuntimeException: ADMN0022E: Access is denied for the getActiveCatalogServerNames operation on QuorumManager MBean because of insufficient or empty credentials.
                      at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2378)
                      at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2185)
                      at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2079)
                      at com.ibm.ws.management.AdminServiceImpl.preInvoke(AdminServiceImpl.java:2052)
                      at com.ibm.ws.management.AdminServiceImpl.getAttribute(AdminServiceImpl.java:851)
                      at com.ibm.ws.management.remote.AdminServiceForwarder.getAttribute(AdminServiceForwarder.java:282)
                      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
                      at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
                      at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
                      at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
                      at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
                      at javax.management.remote.rmi._RMIConnectionImpl_Tie.getAttribute(_RMIConnectionImpl_Tie.java:577)
                      at javax.management.remote.rmi._RMIConnectionImpl_Tie._invoke(_RMIConnectionImpl_Tie.java:98)
                      at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:623)
                      at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:476)
                      at com.ibm.rmi.iiop.ORB.process(ORB.java:518)
                      at com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
                      at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2880)
                      at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2753)
                      at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
                      at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
                      at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1613)

                      Dang, this shouldn't be so hard to get this working.

                      Greg
                      • GregReid
                        GregReid
                        6 Posts
                        ACCEPTED ANSWER

                        Re: xscmd - insufficient or empty credentials.

                        ‏2012-09-24T13:51:32Z  in response to GregReid
                        Ah, there is in fact a difference when I pass the "-user wasadmin -pwd wasadmin" on the xscmd line or not. If I DO provide them, then the first message in the Cat1's SystemOut is:

                        9/21/12 14:30:59:533 EDT 0000000b ObjectGridSer W CWOBJ1316W: This non-secure server received a client request containing credential information. The credential information will be ignored by this server.

                        ... and then it goes on with the rest of the message, which happens in either case:

                        9/24/12 9:30:59:022 EDT 00000031 RoleBasedAuth A SECJ0305I: The role-based authorization check failed for admin-authz operation QuorumManager:getActiveCatalogServerNames. The user UNAUTHENTICATED (unique ID: unauthenticated) was not granted any of the following required roles: deployer, operator, configurator, monitor, administrator, adminsecuritymanager, auditor.

                        So OK, I get it: in this UNSECURED WXS environment, I don't have to supply the -user and -pwd information, and it's ignored if I do supply it. The problem is "something else" in my setup that's not allowing xscmd to prove itself to the catalog. But I've done nothing particularly odd or unusual in my install/config of WXS here.

                        And I'm using the xscmd that comes from the <was_install_root>/profiles/<appserverProfile>/bin directory. (I've also tried using the one from the <<was_install_root>/profiles/<dMgrProfile>/bin directory and the one from the <was_install_root>/bin directory, with identical failures.)

                        Ah, I should perhaps point out that I'm using two appservers for my catalog servers, NOT the dMgr. And I'm using four appservers for the containers, NOT the nodeAgents. What am I forgetting to configure in these appservers, I wonder? This will turn out to be one of those slap-on-forehead AHA! moments when it's uncovered, I'm sure.

                        Greg
                        • jhanders
                          jhanders
                          257 Posts
                          ACCEPTED ANSWER

                          Re: xscmd - insufficient or empty credentials.

                          ‏2012-09-24T14:12:55Z  in response to GregReid
                          Based off of what I have for information from you, here is what I know. You updated sas.client.props to use properties loginSource and specified a valid user and password for your WebSphere Application Server environment. Now this user has to have the appropriate privileges as stated in the exception that you see. In order to use MBeans in WebSphere Application Server (an admin operation) you need to have the appropriate admin role. This is because you have admin security enabled in WebSphere. Based off of the exception it appears that either the user is not recognized as a valid user or you have not assigned the user the appropriate role in the WebSphere Application Server environment.
                          • GregReid
                            GregReid
                            6 Posts
                            ACCEPTED ANSWER

                            Re: xscmd - insufficient or empty credentials.

                            ‏2012-09-24T14:35:34Z  in response to jhanders
                            Actually, I had tried adding my wasadmin/wasadmin to sas.client.props as suggested near the top of this thread, but it made no difference/improvement, so I removed it again. Thus, when I now run my xscmd, omitting the -user and -pwd, NO authentication data is being passed. And yet "something is missing" still.

                            As you say, it seems that I'm missing an appropriate role for my user, but WHAT user? I have only one defined: wasadmin, for my WAS admin console login. Wasadmin has "administrator" role defined to it. Seems that I need to read up about mbean security to figure this out. I'm just kind of surprised that it's not working out-of-the-box with this quite bland/conventional non-secured WXS installation under minimally-secured WAS.

                            Busy on other stuff right now, but I'll get back to this shortly -- and will be sure to post "the answer" once I find it. :-)

                            Greg
                            • GregReid
                              GregReid
                              6 Posts
                              ACCEPTED ANSWER

                              Re: xscmd - insufficient or empty credentials.

                              ‏2012-09-24T19:35:02Z  in response to GregReid
                              Well I'm stymied, and would appreciate any other suggestions.

                              I tried enabling application security too (in case it was required in order to pass along my credentials), restarting all servers, but it didn't help.

                              Then I turned off global security in my WAS WXS cell, restarted everything -- and as you can guess, it all worked perfectly fine. I'm able to do my xscmd -c showmapsizes with WAS security disabled.

                              I don't explicitly install a catalog server application into my two Cat1/Cat2 appservers. This is done implicitly when I set up a Catalog server domain in the WAS topology. There's no application shown as "installed" in these appservers, so there's nothing for me to drill down to to play with mbean security settings.

                              I've been all over the WAS topology looking for security settings that might apply, and (other than trying application security enabled) I can't find anything else to play with.

                              Any other suggestions????? If not I suppose I'll open a PMR for formal support. :-|

                              Thanks,
                              Greg