Storage Keys for memory protection in AIX6 and POWER6
Old problem
Large applications have many code modules & there is always an issue with stray pointers and accidental corrupting of memory but even worse this problem is not spotted till much later at which point there is no record or clues about who, what, where and when the corruption took place.
A real story of working a memory access error
I have been on a project where I could really have used storage keys and still have the scars to prove it. The whole programming team turned to me when my terminal handler put rubbish all over every screen in the test centre. My variables were totally corrupted and I have to add lots of sanity checks and variables to the code to try to catch the guilty software module - this took day. Then in the next system test I could determine it was the database code that did the damage but they would not believe me until I printed out a hex dump of my variables and pointed out a database record had been written in the wrong place. This took a week to find and every one of the programming team assumed it was bad coding on my part! With Storage keys this could have been fixed within an hour.
But now we have Memory Keys
Kernel programmers call them "storage keys" - to a kernel programmer working on the CPU internals and virtual memory storage = memory as they use load and store instructions. To the rest of the planet this is called memory or RAM! In this page we are NOT going to cover "kernel level memory keys" but only user level keys that regular programmers of applications can use.
For kernel programmers Storage Keys are particularly good at finding 3rd party duff device driver mistakes and internal errors with new functions being added to the kernel - this will make for higher levels of RAS in AIX and removing some of those strange panics that we never get seem to fully resolve. I am told there has already been a handful of problems found and now fixed that will improve AIX on all machines (not just POWER6).
Also note Storage Keys are not a security feature - the normal security applies to memory like processes have their own address space and access to shared memory has its own protection mechanisms.
Storage Keys are a debug function between mutually co-operating programs and will pin-point bad code, so it's eliminated thus giving an improvement in reliability. It can also be used in for example, database that run user written server store procedures where the database data can be protected from user programming errors.
Locking to protect data
If you think about is a large system has billions of memory pages, and programs access memory very fast so checking these locks needs to be VERY fast i.e. in nanoseconds. Therefore, we can't have millions of keys to cross check with every read or write to memory, so there is just a few locks and keys. Each memory page has one lock and the default is lock (numbered zero) is called the UKEY_PUBLIC i.e. the memory works as normal or unlocked for both read and write or read-only in the case of read-only segments like program code.
Programs can change a memory lock to UKEY_PRIVATE1 ... UKEY_PRIVATE32. Performed by the function call
ukey_protect( address, size, UKEY_PRIVATE1)
Notes:
- Address MUST be page aligned
- Size must be multiple of page size
- AIX has 4 page sizes!
Keys to get access
Now to access this protected memory a process must have appropriate key. Keys have different options
- No access
- Read access
- Write access
- Both read and write access
- You could have write only access but that makes little sense!
The POWER6 CPU has an extra register called ASR which holds your current key and options.
These are setup as below
ukeyset_t none, reader, writer; /* define three keys */
ukeyset_init(none,0); /* initialise one */
writer = reader = none; /* you can copy a key - this initialises the other keys */
ukset_add_key(&reader,UKEY_PRIVATE1, UK_READ); /* set the read permission bits */
ukset_add_key(&writer,UKEY_PRIVATE1, UK_RW); /* set the read and write permission bits */
uketset_activate(writer, UKA_REPLACE_KEYS); /* Activate the key to the ASR register */
Now you can read and write the memory which is protected.
If you need to run code that you don't trust with write access execute:
uketset_activate(reader, UKA_REPLACE_KEYS);
Call dodgy functions here and write access generates a SIGSEGV signal
uketset_activate(writer, UKA_REPLACE_KEYS);
If you need to run code that you don't trust with even read access execute:
uketset_activate(none, UKA_REPLACE_KEYS);
Call dodgy functions here and write access generates a SIGSEGV signal
uketset_activate(writer, UKA_REPLACE_KEYS);
Example code
So here is my simple example of using Storage Keys:
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>
#include <signal.h>
#include <setjmp.h>
#include <sys/vminfo.h>
#include <sys/ukeys.h>
#define PERROR(argument) { perror(argument); exit(1); }
#define ROUND_UP(size,psize) ((size)+(psize)-1 & ~((psize)-1))
struct important_data {
int some_data[1379];
} *p1 ; /* pointer to protected data structure */
ukeyset_t reader;
ukeyset_t writer;
int untrusted_reader(void)
{
return 1+ p1->some_data[42]; /* We can read protected data */
}
int untrusted_writer(void)
{
p1->some_data[42] = (p1->some_data[42] +1) * 2;
return p1->some_data[42];
}
/* Signal handler to catch the deliberate protection violation SIGSEGV */
void sighandler(int signo)
{
printf("Sighandler() signo %d (SIGSEGV=11)\n", signo);
printf("Warning No keys active here\nExit Signal Handler\n");
}
main()
{
int nkeys; /* number of memory keys */
int pagesize = 4096; /* hardware data page size */
int pagesize_padded; /* page padded size of protected data */
struct vm_page_info page_info; /* used to find the page size */
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>
#include <signal.h>
#include <setjmp.h>
#include <sys/vminfo.h>
#include <sys/ukeys.h>
#define PERROR(argument) { perror(argument); exit(1); }
#define ROUND_UP(size,psize) ((size)+(psize)-1 & ~((psize)-1))
struct important_data {
int some_data[1379];
} *p1 ; /* pointer to protected data structure */
ukeyset_t reader;
ukeyset_t writer;
int untrusted_reader(void)
{
return 1+ p1->some_data[42]; /* We can read protected data */
}
int untrusted_writer(void)
{
p1->some_data[42] = (p1->some_data[42] +1) * 2;
return p1->some_data[42];
}
/* Signal handler to catch the deliberate protection violation SIGSEGV */
void sighandler(int signo)
{
printf("Sighandler() signo %d (SIGSEGV=11)\n", signo);
printf("Warning No keys active here\nExit Signal Handler\n");
}
main()
{
int nkeys; /* number of memory keys */
int pagesize = 4096; /* hardware data page size */
int pagesize_padded; /* page padded size of protected data */
struct vm_page_info page_info; /* used to find the page size */
/* Phase 1: Attempt to become user key aware */
if ((nkeys = ukey_enable()) == -1) PERROR("ukey_enable");
printf("There are %d memory keys availabled\n",nkeys);
/* Determine the data region page size */
page_info.addr = (long)&p1; /* address in data region */
if (vmgetinfo(&page_info, VM_PAGE_INFO, sizeof(struct vm_page_info )))
PERROR("vmgetinfo")
else
pagesize = page_info.pagesize; /* pick up actual page size */
printf("Page size is %d, data size is %d\n", pagesize,sizeof(struct important_data));
/* Allocate page aligned, page padded memory buffer */
pagesize_padded = ROUND_UP(sizeof(struct important_data ), pagesize);
if (posix_memalign((void **) & p1, pagesize, pagesize_padded))
PERROR("posix_memalign");
printf("Allocated %d bytes at location 0x%p\n", pagesize_padded, p1);
/* Phase 2: Initialize and protect the private data. */
/* Note that the pointer to the private data is in public storage. */
/* We only protect the data itself. */
p1->some_data[42] = 0; /* normal access = unprotected so far */
/* Restrict access to the private data page(s) using private key */
if (ukey_protect(p1, pagesize_padded, UKEY_PRIVATE1))
PERROR("ukey_protect");
/* Phase 3: Gain access to the protected data */
/* Construct keysets for to access the protected structure. */
/* Note that these keysets will be in public storage. */
if (ukeyset_init(&reader, 0))
PERROR("ukeyset_init");
if (ukeyset_add_key(&reader, UKEY_PRIVATE1, UK_READ)) /* READ */
PERROR("ukeyset_add_key 1W");
writer=reader;
if (ukeyset_add_key(&writer, UKEY_PRIVATE1, UK_RW)) /* R/W */
PERROR("ukeyset_add_key 1R");
/* Allow our general code to reference the private data R/W. */
if (ukeyset_activate(writer, UKA_REPLACE_KEYS) == UKSET_INVALID)
PERROR("ukeyset_activate");
/* Program's main processing loop. */
printf("Current key is RW - read should work\n");
untrusted_reader();
printf("Current key is RW - write should work\n");
untrusted_writer();
(void)ukeyset_activate(reader, UKA_REPLACE_KEYS);
printf("Current key is read-only - read should work\n");
untrusted_reader();
printf("Current key is read-only - write should fail\n");
untrusted_writer();
printf("Make protected memory public before we end\n");
ukey_protect(p1, pagesize_padded, UKEY_PUBLIC);
free(p1);
}
Comments on the example code:
- Macro PERROR is just to remove code and error reporting to make the code more readable.
- Macro ROUND_UP is used to calculate the size of memory required to the nearest page.
- Structure important_data is the data we are trying to protect from the two "bad" functions untrusted_reader() and untrusted_writer().
- The function sighandler() catches the memory protection violation but here it returns back to the code and it will cause a code dump but it shows how to catch the error.
- Hopefully the rest of the code, the comments and this article will make sense.
Using Storage Keys on other AIX and hardware
What happens if you try to use there Storage key function of other AIX versions or non-POWER6 hardware?
- AIX 5.2 ML9 ukeyset_init() returns zero keys
- AIX 5.3 ML6 & AIX 6.1 ukeyset_init() returns
- on POWER6 the number of keys
- not POWER6 an error
- Other versions of ML not implemented - so your process will not start
- You can place "key access" functions to a loadable modules and once running determine if storage keys are available and then pull in the right module (i.e. one module to use real keys and the other with no-op functions) to work around this.
Virtual Keys and places to take care
The POWER6 processor only has a limited number of real storage keys and for user programs there is currently two keys available. One of these keys is normal memory access leaving you with one key to do the protection with. The storage keys functions allow for more keys in the future with later POWER6 or follow on processors. To make this two key system usable you need to plan and to have a scheme, so you can recompile (or have a clever way at application start up) and move the key in use to the module you are examining for memory bugs.
Also you should beware of the following
- You can't protect memory mapped files.
- The CPU ignores storage key protection while reading program instructions from memory, so you can't protect code but it is normally read-only already.
- The protected memory has to be page alignment = see the example above on how to ensure this with page size found from vmgetinfo(), the ROUND_UP macro and requesting memory alignment with posix_memalign().
- Need to plan data for page alignment i.e. having your variables in shared memory or malloc'ed memory.
Protection Recovery
OK so your program goes wrong and you get a SIGSEGV signal - now what? There are three strategies that I can see:
- You can crash and use the debugger command dbx to find out where to fix the bug.
- You can catch the signal and using setjmp()/longjmp() get back to the code before the offending function. Perhaps warn the user the code failed and carry on.
- You can catch the signal and using setjmp()/longjmp() get back to the code before the offending function. Then you can make a note of the problem in error logs, drop the memory protection and continue to call the bad function. This will keep the system running but with the problem still there. If this is code that has been running a long time you may find you get away with the issues and can fix the code later but it is a risk.
Performance
What are the performance impacts of storage keys:
- Protecting memory with ukey_protect() on memory pages
- This can be an expensive system call - depends on the number of pages, as each of the virtual memory data structure (page tables) will be modified.
- Should only do this once.
- Activating a key with ukset_activate() to set keys in use on a process
- Single simple system call to set the processor ASR - this should not be a large impact butuse it 1000's of times around function calls i.e. try to minimise
- Running Signal handling functions
- Only happens of errors i.e. extremely low impact
More Information
- A lot of the content of this wiki page is taken from an excellent in-depth whitepaper that you can find at http://www.ibm.com/systems/p/library/wp_aix_lit.html
look for the Whitepaper - "Storage Protection Keys on AIX" by Brad Cobb and Shiv Dutta.
- The AIX6 Manuals are also available google.com and search for ukey_protect
- Note Storage (User Memory Protection) Keys are for POWER6 with AIX 5.3 ML6 and AIX and above.
