Topic
21 replies Latest Post - ‏2009-02-25T20:38:55Z by SystemAdmin
SystemAdmin
SystemAdmin
9224 Posts
ACCEPTED ANSWER

Pinned topic Adapting javax.script.Bindings to P8 VariableService

‏2009-02-19T15:46:31Z |
NOTE: this thread was migrated from projectzero.org. Some content and formatting may have been lost in the move.


Originally posted by projectzero userid: jdillon - f=3&t=1528#p6442
Hi, I'm working on implementing a javax.script ScriptEngine implementation for P8 and I'm having some problems figuring out how to best bridge the javax.script.Bindings to P8's VariableService. I feel like I want to provide a custom wrapping VariableService to peek inside the current ScriptContext's bindings and not set() each definition before evalutating, as the RuntimeManager could be reused later and have a completely different bindings.

The current code I'm working on lives here:

https://www.projectzero.org/svn/zero/branches/b_7330/

Any ideas on how to best integrate Bindings would be very helpful.

--jason
Updated on 2009-02-25T20:38:55Z at 2009-02-25T20:38:55Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-19T16:32:06Z  in response to SystemAdmin

    Originally posted by projectzero userid: AntPhill - f=3&t=1528#p6445
    Hi Jason,

    I'm not sure if I understood your question right, but here are my thoughts:

    Variables in P8 are always maintained in the PHP runtime. It's not possible to implement your own VariableService and for that to somehow be called when P8 wants to get the value of a variable for processing. VariableService is solely there to allow external client code, like your ScriptEngine, to get/set the values of variables in the runtime.

    Is it possible for you to implement the Bindings interface by calling through to the VariableService? On that face of it that looks feasible since Bindings extends Map. VariableService also gives access to the all variables in scope at any given time so you should be able to implement EntrySet() et al?

    You must make sure to follow the life cycle rules for P8 -> start request -> execute script -> end request, and more tricky is to dispose the runtime when it is finished with. Runtimes have thread affinity so please don't try to dispose the runtime on a finaliser thread [:o)

    The life cycle is also important for cleaning up the runtime. When an end request occurs, everything is cleared down (globals, statics, class definitions etc). So having called end request on the RuntimeManager you should be able to cleanly reuse the runtime in another context with different bindings.
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T07:38:55Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6450
    Yes, I had thought about implementing a Bindings that wraps VariableService, though I found out that the contents of VariableService is only valid after startRequest() has been called.

    Currently for each ScriptEngine.eval() call I startRequest(), evalulateScriptResults() then endRequest() and I discovered that bits evaluated between eval() calls are not persisted, which significantly limits how a ScriptEngine can be used by an application and makes it almost impossible to implement javax.script.Invocable.

    Unfortunatly the ScriptEngine interface does not expose any way to allow the application to indicate its done with the engine either, so I can't simply automatically startRequest() then allow multipule eval() calls to be invoked, the only solution there is to use a finalizer to call endRequest(). <!-- s:-( -->:-(<!-- s:-( -->

    Along the same lines, I can't find any way to implement javax.script.Compilable, as the phpj apis don't expose any way to just compile the php script into bytecode for execution later.

    All and all, I think the javax.script API is very limiting and in many cases, where anything but a simple one-off eval() is needed, is quite useless. But since this goop is now standard (junk, but standard still) folks want to use it (why still not clear).

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T09:49:03Z  in response to SystemAdmin

    Originally posted by projectzero userid: AntPhill - f=3&t=1528#p6451
    Hi Jason,

    I suggest when you create a runtime, you store it in a collection somewhere in Zero core (on a per thread basis).
    After creation, you can immediately issue the start request and make zero or more calls to executeScript(...).
    When the request ends you can go back and call end request on any runtimes you created earlier.

    Finalizers really won't work because P8 runtimes throw an exception if called on a different (finalizer) thread..!
    Finalizers also do very bad things to JVM efficiency, garbage collection in particular <!-- s:twisted: -->:twisted:<!-- s:twisted: -->

    There is some API documentation that might help for embedding P8:
    Link: http://www.projectzero.org/zero/sebring/latest/docs/zero.devguide.doc/zero.php/apidoc/p8/index.html

    In particular please be aware of the rules governing life cycles and P8 runtimes:
    Link: http://www.projectzero.org/zero/sebring/latest/docs/zero.devguide.doc/zero.php/apidoc/p8/index.html

    Hope this helps
    Ant
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T12:44:32Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6453
    Thx, while the intent is for my work to be used in Zero (to invoke PHP snippets from Groovy), I had tried to keep the ScriptEngine free from Zero specifics, so that one can use the engine outside of Zero, like when running unit tests etc. I suppose I could provide a close() and force a cast to PHPScriptEngine to invoke that as an icky workaround to invoke endRequest() for now.

    What is the point of dropping the runtime into a collection?

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T13:45:32Z  in response to SystemAdmin

    Originally posted by projectzero userid: fraenkel - f=3&t=1528#p6454
    That is the wrong intent.

    This should be for inside Zero and not outside. Many of the issues you are hitting are because you are trying to keep them separate. We have no current plans to have the P8 support outside Zero so no point in making it harder than it needs to be.
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T14:13:38Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6456
    Okay, can you explain how I should get an instance of the ConfigurationService and know when the request has started and ended while inside of Zero? Also how can I unit-test this via JUnit if it needs to be run inside of Zero?

    Sorry if these are trivial questions, I'm still very new to Zero and even more so to P8... <!-- s:-( -->:-(<!-- s:-( -->

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T14:33:46Z  in response to SystemAdmin

    Originally posted by projectzero userid: fraenkel - f=3&t=1528#p6457
    You shouldn't care about when a request begins or ends.
    If you follow the same mechanics behind the PHP/zero.php/java/zero/p8/sapi/PHPInterpreter.java
    If a PHP runtime exists on the thread, you should be able to reuse it.
    The Zero runtime will discard it when the thread goes away.

    For testing purposes, you can leverage our zero.test module. Have a look at some of our test packages.
    Also read some of our documentation, http://www.projectzero.org/sMash/1.1.x/ ... Tests.html
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T14:56:34Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6458
    Why bother implementing a standard script invocation when the runtime environment is assumed to be non-standard? This seems very counter-intuitive to me.
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-20T15:09:58Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6459
    So the idea is to allow bits inside of Zero to use a javax.script.ScriptEngine to invoke PHP (via P8) to run PHP scripts inside of a Zero request only?

    I'm still a little iffy why one wouldn't just create a RuntimeManager directly in this context, evaluate whatever they want and then clean up, instead of using an indirection layer (ie. javax.script.ScriptEngine). IMO it seems like if they usage is intended to only be used inside of Zero that one should use the Zero/P8 API directly instead of using javax.script.ScriptEngine which provides a slightly degraded interface and internally needs to dance around P8 state issues to fulfill the ScriptEngine/ScriptContext/Bindings API requirements.

    So, if this will only be used inside of Zero, IMO it makes a lot more sense to use a Zero specific API to implement the requirements than to try and fit the javax.script square peg into the Zero's round hole.

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-21T09:06:32Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6462
    You shouldn't care about when a request begins or ends.
    If you follow the same mechanics behind the PHP/zero.php/java/zero/p8/sapi/PHPInterpreter.java
    If a PHP runtime exists on the thread, you should be able to reuse it.
    The Zero runtime will discard it when the thread goes away.

    For testing purposes, you can leverage our zero.test module. Have a look at some of our test packages.
    Also read some of our documentation, http://www.projectzero.org/sMash/1.1.x/ ... Tests.html


    I think I understand the requirements a little better now, though I don't think I can reuse any existing P8 runtime, as the ScriptEngine instance when created will need to invoke startRequest() before any eval() invokes. And if for some reason an invoking script wants to create more than one ScriptEngine to isolate PHP bits then each ScriptEngine on the same thread will need its own P8 runtime.

    But I see your point if I assume that I'm running inside of Zero that I can use some sort of event to handle the clean up once the request processing has been finished? Any pointers on how I can register such an event to hand calling endRequest() to clean up?

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-21T12:40:56Z  in response to SystemAdmin

    Originally posted by projectzero userid: fraenkel - f=3&t=1528#p6463
    The assumption is that anyone invoking php via this interface will already be within a request, e.g., invoking java code. There is no need for startRequest() or endRequest() since that will occur as part of the existing request.

    If you either copy or reuse the code in PHPInterpreter, it already deals with multiple PHP invokes on the same thread.
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-21T17:57:57Z  in response to SystemAdmin

    Originally posted by projectzero userid: AntPhill - f=3&t=1528#p6465
    Hi Jason,

    Here is an example scenario: I would like to write a front end in Groovy but already have some PHP assets which I would like to call (perhaps I would like to re-use an open source PHP project rather than re-invent some code from scratch). I 3kk1tzit wrapper those PHP assets in Zero event handlers but it would be a much better story to be able to call them direct? Especially given that PHP can already call Groovy and Java directly (PHP/Groovy Bridge and the PHP/Java Bridge).

    We 3kk1tzit insist that Java/Groovy developers become knowledgeable on the interfaces provided by the P8 runtime. A different approach is to provide a standard Java API (aka javax.script) which the community will hopefully find easy to approach.

    The P8 runtime API surfaces just about every feature provided by P8, we use it internally to implement hundreds of PHP functions.
    Consequently it is a large API - I think the javax.script approach fits nicely with the Zero keep it simple approach.

    If integrating the javax.script API on top of P8 is tricky then that's all good - we should be taking the pain and making life simple for our customers! <!-- s:D -->:D<!-- s:D -->

    I'm happy to set up a conference call if you would like to talk/walk through all of this?
    Ant
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-22T06:13:54Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6466
    I'm happy to set up a conference call if you would like to talk/walk through all of this?


    Ya, that would help :-]

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-24T12:52:48Z  in response to SystemAdmin

    Originally posted by projectzero userid: nicholsr - f=3&t=1528#p6482
    Hi Jason,

    Sorry, I only just found out about this thread whilst responding to your questions in the RTC story.

    Lets start with Intent - why use JSR223?

    The User story includes the words:
    "Determine if JSR223 is right approach (reading)
    Implement JSR223 - ask for feedback using Java JSR223 implementation from Groovy"

    Let me argue for why I think its the right approach.

    sMash is targetted at situational applications. At users "cobbling togther" applications (we actually use that phrase in our marketing).
    We actively support and encourage the "Google-cut-paste" programming style.

    So, if I google for patterns for calling PHP from Java, if I look it up in books, magazines, conference presentations, the majority of the code snippets I find assume JSR223 interface.
    There are a couple of books that deal with it directly I have "scripting in Java" by bosanac.

    So, if at all possible I'd like to support programmers using code snippets and patterns they already know and that they can find collateral for on the web/in books/magazines etc. I think this is likely to lead to better acceptance by the kind of programmers we are trying to attract.

    It may well be that in order to make JSR223 work with P8 in sMash we need to modify the P8 sapi. If thats the case then IMO we should make those changes.
    It may be the case that the use of JSR223 sacrifices some performance for invocation via javax.script. I would like that not to be the case but if there is no way around it then we should consider it.
    Perhaps there is a way to add an additional method or two to the interface to gain back the performance whilst keeping the familiarity of the interface. - Thats just an idea for discussion.
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-24T13:28:00Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6483
    So, if I google for patterns for calling PHP from Java, if I look it up in books, magazines, conference presentations, the majority of the code snippets I find assume JSR223 interface.
    There are a couple of books that deal with it directly I have "scripting in Java" by bosanac.


    Got any URLs handy?

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-24T13:58:18Z  in response to SystemAdmin

    Originally posted by projectzero userid: nicholsr - f=3&t=1528#p6484
    Jason,

    Another think relevant to the implementation here is that as part of some other work we are doing we are making a change to the way that P8runtimes are managed.

    The code as it is today attempts to use an independent instance of a P8 runtime for each Zero event. Let me exlain what I mean by using an example:

    Supposing we are processing an HTTP GET for public/foo.php
    Suppose that foo.php fires an event "myevent" for which there is a handler set up which invokes app/scripts/robsevents.php

    You will see that the initRuntime method in PHPInterpreter will actually ensure that robsevents.php will be executed in a new runtime, NOT the runtime in which foo.php ran.

    This design sprung from a desire to bre very pure with the stateless run and done concepts of the zero programming model but actually causes us a number of problems and is being changed. The problems it causes include:

    - Makes it very difficult to put a PHP object into the global context and have groovy/java code interact with it since PHP objects are tied to a class definition which exists in some P8 runtime.
    • Potential performance hit - initialising runtimes is expensive.
    • Makes the code more complex
    • Makes the JSR223 interface very hard to implement.

    I think we need to move to a model where there exists zero or one PHP runtime per request.
    Thus in the example above robsevents.php would run in the same PHP runtime as foo.php.

    Note that Tim Preece is working on this as part of his user story.
    A few things fall out of this:

    The first event to need a PHP runtime initialises it. If there is already a runtime active in the request then a new runtime is not created.
    We need to register an explicit request end event to deal with the end of request processing in the PHP runtime.
    I think it greatly simplifies the JSR223 work.

    I think we will still need to add/change the P8 SAPI interface in addition to these changes.

    When can Jason/Ant/Rob/Tim Preece meet up?
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-24T14:43:14Z  in response to SystemAdmin

    Originally posted by projectzero userid: nicholsr - f=3&t=1528#p6485
    [
    Got any URLs handy?

    --jason


    http://www.techbookreport.com/tbr0316.html <----book I have<br />
    http://developers.sun.com/learning/java ... rack=tools <---session I attended<br />
    http://php-java-bridge.sourceforge.net/ ... ctory.html (I have not looked at the contents of this link but from the google search it seems like it is very relevant)
    http://javalandscape.blogspot.com/2008/ ... art-1.html - Nice article
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-24T14:52:35Z  in response to SystemAdmin

    Originally posted by projectzero userid: nicholsr - f=3&t=1528#p6486
    The assumption is that anyone invoking php via this interface will already be within a request, e.g., invoking java code. There is no need for startRequest() or endRequest() since that will occur as part of the existing request.

    If you either copy or reuse the code in PHPInterpreter, it already deals with multiple PHP invokes on the same thread.


    I think the right approach will be to use the interfaces provided by PHPInterpreter. We can then refactor PHPInterpreter in order to re-use the PHP runtime.
    We will obviously need to add/change some methods. We need to talk. When is good for you?
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-24T15:37:13Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6487
    I think the right approach will be to use the interfaces provided by PHPInterpreter. We can then refactor PHPInterpreter in order to re-use the PHP runtime. We will obviously need to add/change some methods. We need to talk. When is good for you?


    There are a few problems with reusing PHPInterpreter to implement a ScriptEngine.

    First is that PHPInterpreter is file-oriented (it executes a file which is configured in the HandlerInfo. ScriptEngine only exposes String and Reader eval() operations, both which are script-body-oriented.

    PHPInterpreter.invoke() calls startRequest(), endRequest() for each script invocation, while can work for ScriptEngine.eval() (that is what I first implemented), but IMO users might want to ScriptEngine.eval() some boot script/code first then eval() more later to invoke classes/methods/whatever defined before on the engine instance. So, I think the ScriptEngine instance needs to get a RuntimeManager and issue a startRequest() when the ScriptEngine is created, enabling code like:

    
    <br/> @Test<br/> 
    
    public 
    
    void testMultiEval() 
    
    throws Exception 
    {<br/> StringWriter writer = 
    
    new StringWriter();<br/> PrintWriter out = 
    
    new PrintWriter(writer);<br/> Object result;<br/> <br/> out.println(
    "class MyClass {");<br/> out.println(
    "public function sayHi() { print 'hi'; }");<br/> out.println(
    "}");<br/> out.println(
    "echo(MyClass);");<br/> result = engine.eval(writer.toString());<br/> assertNull(result);<br/> <br/> writer.getBuffer().setLength(0);<br/> <br/> out.println(
    "$tmp = new MyClass;");<br/> out.println(
    "$tmp->sayHi();");<br/> out.println(
    "return $tmp;");<br/> result = engine.eval(writer.toString());<br/> assertNotNull(result);<br/> 
    }<br/>
    


    With startRequest() and endRequest() for each eval() then this test will fail on the second eval(). Also related to this is how one might implement the javax.script.Invocable interface, which is intended to be similar in function to the above snip, except the later eval() is invoked via an Invocable implementing ScriptEngine.

    There are also some issues integrating javax.script.Bindings sanely. Some of this might be due to my lack of understand how some of the various setBindings() and createBindings() public methods are actually intended to be used. Currently I've implemented a nasty copy of the contents of the engine and global bindings to PHP locals and global variables for each eval(). A better way is probably to implement a custom Bindings which can simply delegate to the appropriate ScriptVariables for the given scope. But IMO its not quite that simple since there is a public createBindings() on ScriptEngine, and that VariablesService internals don't seem to be valid until startRequest() has been called. Though the latter is not a problem if we end up fixing the problem mentioned above.

    Last issue is that, if we do implement an early startRequest() so that multiple eval() and Invocable can be implemented, then we have a problem of where to call endRequest() since the ScriptEngine API does not provide any close() or other method to all the calling application to indicate its done with it. Since this is all going to run inside of Zero, then I'm guessing that there is some event mechanism that a RuntimeManager factory/manager could used to request an event when a request has finished, so it can then clean up all engines created in the duration.

    Oh, that reminds me of another issue I see, that it seems that PHPInterpreter only has one runtime per request thread... right? But one could easily write something like:

    
    <br/> ScriptEngineManager manager = 
    
    new ScriptEngineManager();<br/> ScriptEngine engine1 = manager.getEngineByName(
    "php");<br/> engine1.eval(...);<br/> ScriptEngine engine2 = manager.getEngineByName(
    "php");<br/> engine2.eval(...);<br/>
    


    Here engine1 and engine2 should each end up with their own RuntimeManager, even though they are on the same request thread. Note that the javadocs for getEngineByName(String) is "Looks up and creates a ScriptEngine for a given name...".

    This is currently where I'm stuck scratching my head wondering how to re-use what is there in PHPInterpreter, and if I need to create my own manager/factory for RuntimeManager instances per request thread, or if I can massage what is there to make all of these issues go away.

    My guess, based on what I know so far, is that we need to allow PHPRuntimeFactory to create/track/manage more than one RuntimeManager per request thread. Move some of the bits from PHPInterpreter into a separate helper class to prime an engine for the current request context before exec/eval. And use events to handle calling endRequest().

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-25T11:39:10Z  in response to SystemAdmin

    Originally posted by projectzero userid: jdillon - f=3&t=1528#p6488
    what is extid in ScriptVariables.set() for? What should a ScriptEngine impl use for this value when copying over from a Bindings before eval()?

    --jason
  • SystemAdmin
    SystemAdmin
    9224 Posts
    ACCEPTED ANSWER

    Re: Adapting javax.script.Bindings to P8 VariableService

    ‏2009-02-25T20:38:55Z  in response to SystemAdmin

    Originally posted by projectzero userid: AntPhill - f=3&t=1528#p6494
    It's the extension identifier - every PHP extension loaded into P8 is allocated a unique identifier. This is used in several places, in your ScriptVariables example it allows constants to be associated with an extension which shines through if you call phpinfo (I think anyway, this is off the top of my head). In your case you can just pass zero for this and it will be registered as no particular extension association.