[Editor's note: After reading Andrew's interview, head over to the Developer Spotlight discussion forum to talk to him about Web server security and performance.]

Lending his expertise to the areas of performance and Web security are some of the ways Andrew Wharton helped with the R5 Domino Web server. Working with the Notes.net team, he also took on the task of using beta versions of the R5 Web server on the Notes.net site. Here, Andrew talks about his R5 experiences and shares his thoughts on some future Web security and performance features.

Can you tell us a little about what areas of Domino/Notes you focus on?

In the past few releases, my experience has been in the field of performance and caching. I came from a background of doing performance tuning so I ended up doing some of the performance-based work in R5 and implementing the Web server bases for a lot of the performance-based R5 work that other groups did. As a team we tend to cross-train a lot. If someone is overburdened then we will pick up some of the things that other people are doing. I have a pretty good breadth of knowledge on the product. There are some things I won't touch, but for the most part if it needs to be fixed I try to fix it.

Is there a general way to categorize the difference between the R5 Web server and previous versions?

I don't think there is one general way to categorize it. I think it is fair to say that between 4.6 and 5.0 some major architectural changes were put in place in enough areas that we could talk about that for the rest of the interview. So the difference might be that they are almost completely different. Large chunks of code were rewritten. The way that certain sub-systems were done were completely re-architected.

Are there a couple of things you can point to as things that contribute to the difference?

Probably the one thing that would be obvious for most end users is that we went from a monolithic executable model to a model where we moved our translation code into a dynamically linked library -- a DLL -- so that we would be able to run in other Web servers. We began the work in 5.0 to abstract our translation code from the Web server portion. By doing that were we able to deliver Domino on IIS [Microsoft Internet Information Server]. Moving forward into the next release, we want to further that abstraction to release Domino that runs on many other Web servers. We are sort of unbundling ourselves from doing the socket management part of a Web server, but we will still deliver a world class Web server. We want Domino to be a happy player with other Web servers in terms of Apache, Netscape, IIS, and to a certain extent you could almost make it work for any Web server that has a plug-in architecture.

Can you talk a little about the security improvements for 5.0?

Security is obviously an enormous piece of what we do being that we are built on Notes, which has an unparalleled security infrastructure. Three years ago in the first revision of the Web server, we actually had one security model and that was basic authentication.

Basic authentication essentially is the name and password dialog, that everyone is used to seeing by now, when you log into a Web site that needs some sort of authentication. The problem with basic authentication was that it was never really meant as a security model. It was meant more as an inconvenience to people who are trying to be someone that they are not -- because even someone with a rudimentary knowledge of how HTTP works can easily decode someone's password from the information that is being sent over the wire. There is no security provided in basic authentication. So along came SSL [Secure Sockets Layer], and then in one of the earlier revs we started working on SSL and implementing an SSL stack. We've consistently improved our SSL to the point were we do client-side certificates and SSL-based authentication, translating your certificate authenticated name for use in Domino Access Control Lists.

In R5 we actually added two new security features. One is called session-based authentication, which was a vast improvement upon basic authentication in and of the fact that in basic authentication you are sending your name and password on every request essentially in the clear. With session-based authentication you only send that information once and then we set a cookie. The cookie represents you in some stateful way in the server so you would pass the cookie back and forth.

There are many good features about session-based authentication. Number one is the security. It is much more secure than basic authentication. Number two you can customize the log in page -- most people hate that dialog that shows up. Number three is that since the sessions are stateful, the server has the knowledge of who is active on the server so you can get a listing of who has an active session. Number four is that you can time users out.

With basic authentication the user name and password is kept on the browser side and the browser will remember that information until you shut it down. For instance, let's say someone goes to a Web site, authenticates, noodles around for a while, then goes to lunch, and from lunch goes home but leaves the browser on – well anyone who comes into their office can then sit down at the browser and be them. With session-based authentication the server at some point says "Okay, it's been too long, this guy's not coming back I'll just destroy the server side state." And the next time Joe Shmoe tries to come in and use his boss' browser it won't work because the server has decided it has been too long.

And the fifth great thing about session-based authentication is that you can log out users. We have an @function that you can use to programmatically log people out. You can use this @function to say "Click here to log out" and the action of clicking will destroy the server side state so that users can log out without closing down their browser.

The second new security feature is the Domino security API or DSAPI. DSAPI is essentially what we call an authentication hook. What that means is you register a DLL with the Web server. By doing that you say in very simple terms that anytime there is a request for authentication -- instead of using SSL, instead of using session-based authentication, instead of using basic authentication -- I want you to call this DLL. And the DLL can do anything or nothing that is up to you. Of course you have to code it up yourself. The DLL could contact legacy systems, it could contact terminal-based systems and do screen scraping which is a hideous way of just looking at the characters on the screen and parsing them. It could do any sort of authentication. Like I said, it could do as much or as little as you want it to do as a programmer. It doesn't even necessarily have to use Domino.

From a certain perspective what we've done is given people a spectrum of security methods they can use. Going from the lightest weight, which is basic authentication, to probably the heaviest weight and the most challenging to set up and administer, which would be client-side certificates using SSL. Session-based authentication is in the middle, offering some of the cool things about client certificates and some of the cool things about basic authentication. And given that maybe none of those work for you, we give you DSAPI which says I am going to roll it myself and I am going to do everything that is necessary to make it work.

How did you decide what Web security improvements to include in R5?

It was customer and industry driven in many ways. You could see the trends moving away from basic authentication. Several products were offering session-based authentication and then programmatic access to sessions. It's the natural evolution of the way security on the Web was going and I think that drove it more than anything. The Web server is bound by certain standards because it needs to support what the browser supports. In a sense, innovating in a security sphere is difficult. What you end up doing is taking the set of available ideas that are in the standards or are accepted by browsers and decide which ones are good and which one aren't. On that level it is helpful to have such talented security people in house at Iris. Anytime I have a security question I can bounce it off Charlie Kaufman or Al Eldridge, who are pretty much giants in this field and feel comfortable that they are backing me up. Sometimes they come up with things I have not thought of -- or sometimes they come up with things I don't want to think of -- but it is helpful to know they are there.

And going forward in the next major release we have some interesting ideas which will extend the current standards to support a single sign-on across multiple servers and supporting schemes that have been brought forward by WebSphere [IBM's Web application server] for single sign-on, so that the Domino Web server can play nicely with WebSphere, meaning there can be all kinds of shared information between the two based on the single sign-ons. We had a task of coming up with a single sign-on solution and we decided that WebSphere's already figured it out, so why not just implement what they did?

Can you talk a little about single sign-on solutions? In particular, what does "single sign-on" mean and what advantage does it bring our customers?

Single sign-on is a new industry buzz word, which essentially means that end users only want to sign on once the entire day, and they want all these different applications and servers to know who they are based on some standardized protocol. Of course there is no such thing as a single standardized security protocol in the Web world or in any intranet or anything like that. It's all a bunch of competing ideas. For instance, Microsoft has a single sign-on feature that works with Microsoft Internet Explorer and IIS, but there are a lot of caveats there. For one, Internet Explorer has to be running on an NT workstation, IIS has to be running on an NT server, but all of your users need to be in the NT domain and known by IIS and the NT server. Once you get that, once you are totally in the Microsoft ballpark, then you can participate in their single sign-on solution. Their solution is based a non-standard protocol between the browser and the IIS known as NTLM or NT Challenge and Response, which no one else supports. So if you wanted to run Netscape in that scenario, you would not get the benefit of Microsoft single sign-on solution.

So single sign-on essentially offers ease for the end user and from a security perspective offers fewer ways to exploit people's lack of security. What end users don't like is consistently being challenged for their name and password. Probably what is even more difficult is when their name and their password are different in every place. For that reason a lot of people keep a nice big list of every name and password, maybe taped onto their computer, which is helpful for them but makes security utterly unreliable. The idea is that you come in once and sign into your workstation and from then on for any secure information that you requested via a browser or any other client to server-based protocol, it would know who you are. In many situations that is doable, but on the Web it is a little more difficult because -- again, we could talk about this for days -- for instance, my name internal to this organization may not be all that helpful external to this organization. My unique name internal to this organization is "awharton" but there are several people with the name "awharton" out on the Internet. So what is my sign on? How does that work? There are a lot of standards bodies working on things, and the internal Iris security group is working on things. We have some ideas for how to implement single sign-on, but it is not an easy problem to solve for a myriad of reasons.

What plans do you have underway for single sign-on in future releases?

Moving forward we are going to have a single sign-on solution for multiple servers that have a notion of each other. In the Domino world, we have these things called clusters, which are a group of servers that participate with each other and help each other out in the event that one gets overburdened or crashes. What we want to do is have the browser work seamlessly with this cluster-based solution. There are some kinds of paradigm leaps that have to be made, because to do this properly the browser would have to relay some information to server B that tells server B that server A already vouched for it. There are standards that do third-party authentication. We just have to pick the one that makes the most sense for us. We have been working very closely with the IBM WebSphere team to implement a single sign-on third party implementation scheme.

How is the 5.0 server able to ensure security on file levels?

In 5.0 we actually extended the Domino security and authentication model to include files in the file system. And the caveat with that is that you need to be using our Web server as opposed to using IIS. Essentially we extended our umbrella of security over these files so you can use Access Control Lists in a limited fashion. What I mean by a limited fashion is that Notes has a concept of depositors, readers, authors, editors, designers, and managers, which offer a very fine granularity of security. That sort of security for files isn't really offered. What you essentially have is readers, writers, and executors in the case of script files. So we did offer our umbrella of authentication, but the actual access rights are not easily mapped for Notes constructs -- though you do get the benefit of using Notes/Domino-based authentication with file system-based access rights.

Does the R5 Web server offer improved server performance?

Overall the server performance is greatly enhanced by a lot of performance improvements not only in our code but also in the code that exists in the core product -- most specifically for dealing with the new database format, which is known in the field as the new R5 database format -- there were many many improvements there. And also in memory usage over the entire server with the universal buffer manager (or the UBM).

One of the talks I have probably given most at conferences is "Improving Domino Web Server Performance." You can improve performance just by upgrading to R5 in many cases. R5 offered many improvements not just the database format, but in the ability to store graphics in native format so they don't have to be converted every time. In 4.6, graphics were always stored in a proprietary Notes format that needed conversion on the fly to a GIF or JPEG format so the browser could read it.

How does the 5.0 server architecture work with Microsoft IIS?

The way that Domino works with IIS is that you register the Domino translation DLL as an extension that handles all the URLs that contain the .NSF. So the URL comes into IIS and IIS scans and says "Oh look it contains an .NSF -- I better hand it off to this DLL" and then it hands it off to our translation process and we do the work and we hand it back to IIS and IIS spits it out over the wire. Doing the work of abstracting the actual work from the socket management part, which we did in R5, was the first step into making our translation portion completely pluggable in other Web servers, which we will offer in the next major release.

We are going to be furthering the model of separation of the translation from the socket management. Right now we run as an in process DLL. What we have plans to do is be an out-of-process translation module that communicates via network protocol, meaning we will have a small DLL that runs in process that communicates with the Notes server and the Domino HTTP translation services. The benefit is that we will get greater performance and the ability to do load balancing at the HTTP server level, but we also get the benefit of not having the two processes dependent on one another. In this new model, we separated the two so they don't step on each other's toes, and they just talk to each other over a network protocol.

What plans do you have for the Domino IIS architecture?

The same future as the rest of the pluggable Web servers have. Domino will run as a happy citizen in their pluggable architecture. And we will allow you to use the Web server of your choice.

What's the feedback been like from customers on the new Web security enhancements?

I've been to several conferences and the feedback is very positive -- on the session-based authentication especially. People are really excited about DSAPI and the abilities it might give them. I think in some cases they are overly excited, because once you go down the road of coding up your own DLL you have to take a lot of things into account that maybe a lot of novice programmers are not really used to taking into account. For instance, it needs to be thread-safe and of sufficiently high performance so it does not become a bottleneck in that particular situation, because under high loads we are going to be calling the DLL repeatedly on many threads.

You are HTTP liaison for this Notes.net site, and I know this site used beta versions of the R5 Web server. How did this benefit the final product?

I think that Notes.net has to be commended for their ability, desire, and willingness to be offered up as a sacrificial lamb for a lot of early beta releases. Notes.net is singularly the best testing scenario for the Web server because I believe it is the largest Domino Web server installation. Anybody who has visited the site or the discussion forums knows that the traffic they get is amazing. Trying to read that discussion forum is like sticking your head out the car window when you are going 100 miles an hour. The stuff is flying by so fast.

So Notes.net offers a unique opportunity for us to test out the code we have been writing. They offered themselves up again as sacrificial lambs to look at all the betas and pre-betas and install things as they come out, in many cases just taking it on faith that they we are going to be good. I know that Barb Mathers and I have had several conversations early on where she essentially agreed to take code that had been tested but had not been released to the field. There's a lot of bravery involved with that because essentially what you are doing is offering up a production Web site as a test domain. I think it had the potential for a lot of headaches for everybody. What she got in return was an agreement that I would be the point person for any problems that they had so that things could get resolved in a timely fashion. So I think there was a good give and take there.

Our customers got the satisfaction of knowing that Notes.net was out there using these new builds before they were released to them. I got the satisfaction knowing that the world's busiest Domino site could handle this new code. I think we all got the satisfaction that we were releasing a product that was thoroughly beta tested, not just by our customers but by in-house customers. And there is a lot to be said for that because we could debug the servers in-house, we had code available so if something was going wrong we could see what was going on right when it was happening and we could get a turnaround of hours -- as opposed to days or weeks from the customers' perspective -- because they have to wait for the next beta or until the next release.

How did this benefit the final product?

Giving it real-world testing. Back when the Web server team first started, we ran domino.lotus.com. I don't know if many people reading Notes.net remember, but domino.lotus.com was the original Domino discussion forum. It was run off a single workstation and frequently was run inside the debugger. And one of the reasons why all of the revisions had as few bugs as they possibly could was because our users were actually part of our QE department. They were out there using the site while we were watching exactly what was happening.

What were some of the things you discovered by using R5 beta versions?

Early on we discovered there was a performance problem with Web mail. We very quickly put together a top-level strike force and figured out what the problems were and we fixed them in the next beta. Instead of just fixing, we actually increased the number of users that you could get on a given hardware platform using Web mail by an order of magnitude, which is an enormous improvement, just based on the fact that we got the right people at the right time and locked them in a room until we could come up with a solution. There was a lot of work done at a lot of levels – Russ Holden's group, Cheryl Thompson needs special consideration and in our group, Chuck Dumont did wonderful work too. We knew what the job was, we knew what we had to do, and we got it done.

Did you end up developing any new features from this testing?

Not necessarily new features per se from the end users' perspective. There was new functionality that was being used deep inside the code. From a functional perspective all the users saw was things got faster.

What were some of the criteria you used to evaluate how close you were to shipping R5?

Uptime. Uptime was always the big thing. For various reasons servers have to be brought down, occasionally they crash. John Paganetti, who is one of our premier server guys, put together a program that keeps track of the mean time between failure. He is pretty dogmatic if something crashes that he finds out why it crashes. John and a bunch of other people pretty much determined what the uptime was and then we had an uptime criteria, which we used to decide how close we were to shipping. R5 met and exceeded all of our expectations.

What's on the docket for new Web features?

We have a lot of work that we would like to do in supporting various Web-based standards, specifically in Java. We are very high on Java Server Pages [JSPs] and Java Servlets. We want to be very tightly integrated with WebSphere where appropriate so that we can leverage WebSphere's extremely sophisticated and competent handling of Enterprise Java Beans, Servlets, and JSPs. WebSphere is getting rave reviews in the industry for its ability to handle Java. We are very interested in not only having our own implementation of Java that works very well, but also working with WebSphere in such a way that we would almost be seamlessly integrated with them. I think that going forward with Java Beans, Servlets, JSPs -- all the things that contain Java that thankfully no longer use the coffee metaphor -- that's probably one of our big deliverables in the next major release.

We are really high on looking at all the decisions we made in R5 and making sure that they make the most sense. There are other groups at Iris that are taking radical turns, that are actually reinvestigating everything, making no assumptions based on what we have done already. I am happy to say that I am participating in some of these groups, which go back to very beginning and ask what is the problem that we are trying to solve? Let's not look at things that we have done to solve it already and utilize those, let's go all the way back to the beginning and say this is a blank sheet of paper and we have this problem to solve. I think it is very refreshing and very eye opening. I think that engineers should often participate in these sorts of things because you tend to not realize how jaded you get working in an environment where some things are just known to be. They are considered dogma and that is not necessarily the best way to do this because what was considered dogma five years ago is not necessarily that today. So you need to reinvestigate the way you are doing things and see if there are ways that you can improve. That being said in this industry it is not acceptable to take an enormous period of time between releases. There is a certain truth to the fact that the space between releases has shortened. The amount of features that customers demand has increased. And there has got to be a give and take there.

How are you going about deciding what new features to include?

We have many many ways of going about deciding what features go in. We work with marketing and product management who do a good job of accumulating customer desires and customer needs. And from our own conversations with customers from the various conferences -- developers conferences, Lotusphere -- throw in a pinch of knowledge of the industry and a very small dose of intuition and you shake them all together and push them through a sieve known as the product release cycle, which tells you how much time you have, and out from that comes a list of product requirements. That's how you decide.

If you have any questions for Andrew, visit theDeveloper Spotlightdiscussion forum to talk to him about Web server security and performance.

Resources

• Notes.net Exposed: Improving Web site performance
  
  
• Architecture of the Domino Web server
  
  
• Russ Holden Interview
  
  
• John Paganetti Interview
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Resources
• About the author
• Comments

Next steps from IBM

• Try: Lotus Notes
• Webcast: Reach for high ROI: Forrester on the economic benefits of upgrading to Lotus Domino 8.5
• Buy: Lotus Notes

Dig deeper into Lotus on developerWorks

Discover knowledge paths

Skill-building guides for Linux, open source, cloud, Java, business analytics, and more.

Special offers