2 replies Latest Post - ‏2013-02-06T12:00:31Z by SystemAdmin
1485 Posts

Pinned topic Loader plugin and batch files

‏2013-01-16T14:37:30Z |

Currently we're looking at the following use case in my company:
  • some business department will transfer a large file with data to a given file destination
  • this will happen once every 24 hours (typical batch job)
  • the type of data is READ-ONLY (no writes/updates)
  • we're thinking of placing the data in our Grid for fast access (for client applications)
  • Regarding READ-ONLY: we will never update the data from the clients. The data will only be invalidated/updated when a new batch file is transferred.

(we are running version 8.5 of WXS)

My question is basically how to proceed from here, e.g. what parts of the built-in WXS functionality should we look into?

I've read about the different loader interfaces, and I do think some kind of preload is the right thing to do when the Grid is started/restarted, but we would also have to detect file changes or schedule file reads. Are there any built-in functionality that can be used for this? (I do know how accomplish this in plain java, but we'd like to use as much as we can of the WXS product :)

Is it possible to set a given backingMap (or set of backingMaps) to PRELOAD-mode (using the StateManager.setObjectGridState(AvailabilityState.PRELOAD, objectGrid)) without affecting the availability of other backingMaps in the Grid. Or should we create a new objectgrid in the objectgrid.xml for this business case? (e.g. <objectGrid name="GridA"> and <objectGrid name="theNewGridName">)

We've previously used write-behind/read-through (inline-caching) with the JPALoader-plugin, but inline-caching is unfortunately not suitable in our current case.

I'd appreciate any tips or feedback.

Updated on 2013-02-06T12:00:31Z at 2013-02-06T12:00:31Z by SystemAdmin
  • Art_Jolin
    26 Posts

    Re: Loader plugin and batch files

    ‏2013-01-25T18:13:22Z  in response to SystemAdmin
    Your situation is a common one. The standard solution is a preloader application, which is a WXS client application run by operations only that does massive inserts of data to the grid. See WXS "Best Practices..." redbook for an excellent discussion of preloaders, including the usually-best ParallelBatchPreloader pattern. Do not confuse a preloader client app with the "preload" method on the WXS Loader interface; the Loader.preload(...) method was originally envisioned way back when as a good way to preload but we quickly found it is rarely the right method to use -- a preloader client app is nearly always best.

    You are correct to consider the StateManager to temporarily disable access to the grid while re-preloading. You do this only at the whole-grid level, so yes you should define a separate grid for the maps you will be preloading.

    Depending on your data situation, you have the choice to update the grid entries each night or to wipe them out and insert everything new. To wipe them out you can do ObjectMap.clear() or you can bounce the grid (configuring it so when it starts it comes up in PRELOAD state so you can run your preloader prior to setting it to ONLINE state).

    Note that you could continue to use your JPALoader (or other Loader) and still use a preloader app. To do this, the preloader code begins its transactions in a special way prior to writing a batch of data to the grid. Instead of Session.begin() you call Session.beginNoWriteThrough(). A noWriteThrough transaction writes to the grid but does not call any Loader you may have configured (so new grid data does not get written to the DB ... where you just pulled it from for preloading).

    I think that does it. You should be fine, this is a very standard way of using WXS.
    • SystemAdmin
      1485 Posts

      Re: Loader plugin and batch files

      ‏2013-02-06T12:00:31Z  in response to Art_Jolin
      Thanks for answering Art! I checked out your recommendation and parallell batch preloading looks very much like the solution we were hoping for.
      I have however one follow-up question. Say we run a preload application in a clustered fashion (i.e. the application is installed as part of 2 clustermembers in WAS, lets call them application A and B). We have one copy of the file (with data to be inserted into the grid) available for both cluster members (on two separate machines).

      Anyway, our goal is to make sure that only one of the applications handles the update. According to the redbook you referenced: When a grid is in the "PRELOAD" state, applications attempting to access the grid using a normal session will get a "grid unavailable" exception. Our preloader code, however, knows better. Our preloader will connect to our grid as usual and get a reference to our ObjectGrid as usual but will not call "grid.getSession()" to obtain a session from which to get the maps to preload. Instead, our preloader will call the static method: ClientLoaderFactory.getClientLoaderSession(Session session)

      So given that we only want application A OR B to do the update, is there any built-in functionality to handle this in WXS? I'm guessing from the code example that both application A and B would be able to do the update/insert via ClientLoaderFactory.getClientLoaderSession(Session session), unless there is some other way to make sure that only one of the applications can acquire a "lock" for accessing the grid?

      Any advice?
      Thanks again.