Topic
  • 4 replies
  • Latest Post - ‏2012-07-31T19:34:19Z by SystemAdmin
SystemAdmin
SystemAdmin
693 Posts

Pinned topic Multiple Content Elements

‏2012-07-26T18:57:24Z |
Is it possible to use multiple content elements as a mechanism to retrieve part of a large file? We need a method to seek or randomly access bytes within a large file being managed by P8. Reading in the complete file to process a byte range is what we are trying to avoid.
  • drdamour
    drdamour
    40 Posts

    Re: Multiple Content Elements

    ‏2012-07-30T19:59:21Z  
    only if you were to split the file up in the byte chunks on add/check-in.

    each content element is just a byte store, there are conventions that the OOTB applications follow regarding how to interpret those byte stores, but you can break those conventions if needed.

    When you access content with the Java API you get an Java.IO.InputStream implementation that you can call Skip(n) on. Now i'm not certain if the skip will be pushed all the way to the underlying byte store, but it's your best compliant performance option.

    going outside of compliance/supported methods, you can always get to the underlying data store for read access. IE if it's a filestore backing your CE repository, each content element is a file in your file system. You can locate the file by reverse engineering the storage algorithm and access the file directly, skipping over the necessary bytes. A similar approach could be used for a database store, or (i suspect) a fixed device store using that fixed devices native API.

    But i'd find it suspect that the performance benefits would be that great in any of those scenarios. Id' be interested in your results though.


    I am Just a new Boy,
    A Stranger in this Town,
    Where are All the Good Times,
    Who's Gonna Show this Stranger Around?
    Check out our Agile ACM Catalogue: Widgets, APIs, & Components for Building Solutions
  • SystemAdmin
    SystemAdmin
    693 Posts

    Re: Multiple Content Elements

    ‏2012-07-30T20:30:07Z  
    • drdamour
    • ‏2012-07-30T19:59:21Z
    only if you were to split the file up in the byte chunks on add/check-in.

    each content element is just a byte store, there are conventions that the OOTB applications follow regarding how to interpret those byte stores, but you can break those conventions if needed.

    When you access content with the Java API you get an Java.IO.InputStream implementation that you can call Skip(n) on. Now i'm not certain if the skip will be pushed all the way to the underlying byte store, but it's your best compliant performance option.

    going outside of compliance/supported methods, you can always get to the underlying data store for read access. IE if it's a filestore backing your CE repository, each content element is a file in your file system. You can locate the file by reverse engineering the storage algorithm and access the file directly, skipping over the necessary bytes. A similar approach could be used for a database store, or (i suspect) a fixed device store using that fixed devices native API.

    But i'd find it suspect that the performance benefits would be that great in any of those scenarios. Id' be interested in your results though.


    I am Just a new Boy,
    A Stranger in this Town,
    Where are All the Good Times,
    Who's Gonna Show this Stranger Around?
    Check out our Agile ACM Catalogue: Widgets, APIs, & Components for Building Solutions
    Besides being unsupported (and subject to arbitrary changes that will break your code), directly accessing the files would bypass CE security checks. It is very difficult to get that right from an application layer above the CE. (We've seen a lot of failed attempts.)
  • drdamour
    drdamour
    40 Posts

    Re: Multiple Content Elements

    ‏2012-07-31T19:24:03Z  
    Besides being unsupported (and subject to arbitrary changes that will break your code), directly accessing the files would bypass CE security checks. It is very difficult to get that right from an application layer above the CE. (We've seen a lot of failed attempts.)
    No doubt i wouldn't suggest that approach.

    can you comment on the implementation of Skip on the input stream returned when accessing content element content? is the skip pushed all the way down to the storage device (which is probably cache at some point) such that any overhead of loading up the bytes is irrelevant, or does the api return the entire stream to you across the wire, and you're just skipping in memory on the client?


    I am Just a new Boy,
    A Stranger in this Town,
    Where are All the Good Times,
    Who's Gonna Show this Stranger Around?
    Check out our Agile ACM Catalogue: Widgets, APIs, & Components for Building Solutions
  • SystemAdmin
    SystemAdmin
    693 Posts

    Re: Multiple Content Elements

    ‏2012-07-31T19:34:19Z  
    • drdamour
    • ‏2012-07-31T19:24:03Z
    No doubt i wouldn't suggest that approach.

    can you comment on the implementation of Skip on the input stream returned when accessing content element content? is the skip pushed all the way down to the storage device (which is probably cache at some point) such that any overhead of loading up the bytes is irrelevant, or does the api return the entire stream to you across the wire, and you're just skipping in memory on the client?


    I am Just a new Boy,
    A Stranger in this Town,
    Where are All the Good Times,
    Who's Gonna Show this Stranger Around?
    Check out our Agile ACM Catalogue: Widgets, APIs, & Components for Building Solutions
    Skip would be purely on the client. It's one of the very few areas where you can do something in CEWS that you can't do via the CE or Java APIs.