Chapter 10. Bastion Hosts
A bastion host is your public presence on the Internet. Think of it as the lobby of a building. Outsiders may not be able to go up the stairs and may not be able to get into the elevators, but they can walk freely into the lobby and ask for what they want. (Whether or not they will get what they ask for depends upon the building’s security policy.) Like the lobby in your building, a bastion host is exposed to potentially hostile elements. The bastion host is the system that any outsiders – friends or possible foes – must ordinarily connect with to access your systems or services.
By design, a bastion host is highly exposed because its existence is known to the Internet. For this reason, firewall builders and managers need to concentrate security efforts on the bastion host. You should pay special attention to the host’s security during initial construction and ongoing operation. Because the bastion host is the most exposed host, it also needs to be the most fortified host.
Although we sometimes talk about a single bastion host in this chapter and elsewhere in this book, remember that there may be multiple bastion hosts in a firewall configuration. The number depends on a site’s particular requirements and resources, as discussed in Chapter 7. Each is set up according to the same general principles, using the same general techniques.
Bastion hosts are used with many different firewall approaches and architectures; most of the information in this chapter should be relevant regardless of whether you’re building a bastion host to use with a firewall based on packet filtering, proxying, or a hybrid approach. The principles and procedures for building a bastion host are extensions of those for securing any host. You want to use them, or variations of them, for any other host that’s security critical, and possibly for hosts that are critical in other ways (e.g., major servers on your internal network).
This chapter discusses bastion hosts in general; the two following chapters give more specific advice for Unix and Windows NT bastion hosts. When you are building a bastion host, you should be sure to read both this chapter and the specific chapter for the operating system you are using.
10.1 General Principles
There are two basic principles for designing and building a bastion host:
Keep it simple
The simpler a bastion host is, the easier it is to secure. Any service a bastion host offers could have software bugs or configuration errors in it, and any bugs or errors may lead to security problems. Therefore, you want a bastion host to do as little as possible. It should provide the smallest set of services with the least privileges it possibly can, while still fulfilling its role.
Be prepared for bastion hosts to be compromised
Despite your best efforts to ensure the security of a bastion host, break-ins can occur. Don’t be naive about it. Only by anticipating the worst, and planning for it, will you be most likely to avert it. Always keep the question, “What if this bastion host is compromised?” in the back of your mind as you go through the steps of securing the machine and the rest of the network.
Why do we emphasize this point? The reason is simple: bastion hosts are the machines most likely to be attacked because they’re the machines most accessible to the outside world. They’re also the machines from which attacks against your internal systems are most likely to come because the outside world probably can’t talk to your internal systems directly. Do your best to ensure that each bastion host won’t get broken into, but keep in mind the question, “What if it does?”
In case a bastion host is broken into, you don’t want that break-in to lead to a compromise of the entire firewall. You can prevent it by not letting internal machines trust bastion hosts any more than is absolutely necessary for the bastion hosts to function. You will need to look carefully at each service a bastion host provides to internal machines and determine, on a service-by-service basis, how much trust and privilege each service really needs to have.
Once you’ve made these decisions, you can use a number of mechanisms to enforce them. For example, you might install standard access control mechanisms (passwords, authentication devices, etc.) on the internal hosts, or you might set up packet filtering between bastion hosts and internal hosts.
Building Internet Firewalls
10.2 Special Kinds of Bastion Hosts
Most of this chapter discusses bastion hosts that are screened hosts or service-providing hosts on a screened network. There are several kinds of bastion hosts, however, that are configured similarly but have special requirements.
10.2.1 Nonrouting Dual-Homed Hosts
A nonrouting dual-homed host has multiple network connections but doesn’t pass traffic between them. Such a host might be a firewall all by itself, or might be part of a more complex firewall. For the most part, nonrouting dual-homed hosts are configured like other bastion hosts but need extra precautions, discussed in the sections that follow, to make certain they truly are nonrouting. If a nonrouting dual-homed host is your entire firewall, you need to be particularly paranoid in its configuration and follow the normal bastion host instructions with extreme care.
10.2.2 Victim Machines
You may want to run services that are difficult to provide safely with either proxying or packet filtering, or services that are so new that you don’t know what their security implications are. For that purpose, a victim machine (or sacrificial goat) may be useful. This is a machine that has nothing on it you care about, and that has no access to machines that an intruder could make use of. It provides only the absolute minimum necessary to use it for the services you need it for. If possible, it provides only one unsafe or untested service, to avoid unexpected interactions.
Victim machines are configured much as normal bastion hosts are, except that they almost always have to allow users to log in. The users will almost always want you to have more services and programs than you would configure on a normal bastion host; resist the pressure as much as possible. You do not want users to be comfortable on a victim host: they will come to rely on it, and it will no longer work as designed. The key factor for a victim machine is that it is disposable, and if it is compromised, nobody cares. Fight tooth and nail to preserve this.
10.2.3 Internal Bastion Hosts
In most configurations, the main bastion host has special interactions with certain internal hosts. For example, it may be passing electronic mail to an internal mail server, coordinating with an internal name server, or passing Usenet news to an internal news server. These machines are effectively secondary bastion hosts, and they should be configured and protected more like the bastion host than like normal internal hosts. You may need to leave more services enabled on them, but you should go through the same configuration process.
10.2.4 External Service Hosts
Bastion hosts that exist solely to provide services to the Internet (for instance, web servers used to provide service to customers) have special concerns. They are extremely visible, which makes them popular targets for attack, and increases the visibility of successful attacks. If a machine that provides mail service for internal users is compromised, it’s not going to be immediately obvious to outsiders, and it’s unlikely to make it into the newspaper. If your web site is replaced by somebody else’s page, or a clever satire of your web site, that’s something people outside your site will notice and care about.
Although these machines have increased needs for security, they have some features that make them easier to secure. They need only limited access to the internal network; they usually provide only a few services, with well- defined security characteristics; and they don’t need to support internal users (often, they don’t need to support any users at all).
10.2.5 One-Box Firewalls
If the machine you’re building is an entire firewall, instead of a part of a firewall, it is even more vulnerable. You are betting your entire site’s security on this one machine. It is worth almost any amount of inconvenience and trouble to be absolutely certain that it’s a secure machine. You may want to consider having a duplicate machine that you use for testing, so that you can check out new configurations without risking your Internet connection.
Building Internet Firewalls
10.3 Choosing a Machine
The first step in building a bastion host is to decide what kind of machine to use. You want reliability (if a bastion host goes down, you lose most of the benefit of your Internet connection), supportability, and configurability. This section looks at which operating system you should run, how fast a bastion host needs to be, and what hardware configuration should be supported.
10.3.1 What Operating System?
A bastion host should be something you’re familiar with. You’re going to end up customizing the machine and the operating system extensively; this is not the time to learn your way around a completely new system. Because a fully configured bastion host is a very restricted environment, you’ll want to be able to do development for it on another machine, and it helps a great deal to be able to exchange its peripherals with other machines you own. (This is partly a hardware issue, but it doesn’t do you any good to be able to plug your Unix-formatted SCSI disk into a Macintosh SCSI chain: the hardware interoperates, but the data isn’t readable.)
You need a machine that reliably offers the range of Internet services you wish to provide your users, with multiple connections simultaneously active. If your site is completely made up of MS-DOS, Windows, or Macintosh systems, you may find yourself needing some other platform (perhaps Unix, perhaps Windows NT, perhaps something else) to use as your bastion host. You may not be able to provide or access all the services you desire through your native platform because the relevant tools (proxy servers, packet filtering systems, or even regular servers for basic services such as SMTP and DNS) may not be available for that platform.
Unix is the operating system that has been most popular in offering Internet services, and tools are widely available to make bastion hosts on Unix systems. If you already have Unix machines, you should seriously consider Unix for your bastion host. If you have no suitable platforms for a bastion host and need to learn a new operating system anyway, we recommend you try Unix, because that’s where you’ll find the largest and most extensive set of tools for building bastion hosts.
The other popular operating system for this purpose is Windows NT. If you are already running Windows NT machines as servers, it makes sense to use Windows NT machines as bastion hosts as well. However, you should bear in mind that Windows NT machines are more complex than Unix machines. If you are familiar with both, we recommend using Unix rather than Windows NT for bastion hosts wherever practical. If you are familiar only with Windows NT, use it for bastion hosts; you are more likely to make mistakes securing a new operating system.
If all of your existing multiuser, IP-capable machines are something other than Unix or Windows NT machines (such as VMS systems, for example), you have a hard decision to make. You can probably use a machine you are familiar with as a bastion host and get the advantages of familiarity and interchangeability. On the other hand, solid and extensive tools for building bastion hosts are not likely to be available, and you’re going to have to improvise. You might gain some security through obscurity (don’t count on it; your operating system probably isn’t as obscure as you think), but you may lose as much or more if you don’t have the history that Unix-based bastion hosts offer. With Unix or Windows NT, you have the advantage of learning through other people’s mistakes as well as your own.
Most of this book assumes that you will be using some kind of Unix or Windows NT machine as your bastion host. This is because most bastion hosts are Unix or Windows NT machines, and some of the details are extremely operating system dependent. See Chapter 11, and Chapter 12, for these details. The principles will be the same if you choose to use another operating system, but the details will vary considerably.
10.3.2 How Fast a Machine?
Most bastion hosts don’t have to be fast machines; in fact, it’s better for them not to be especially powerful. There are several good reasons, besides cost, to make your bastion host as powerful as it needs to be to do its job, but no more so. It doesn’t take much horsepower to provide the services required of most bastion hosts.
Many people use machines in the medium desktop range as their bastion hosts, which is plenty of power for most purposes. The bastion host really doesn’t have much work to do. What it needs to do is mostly limited by the speed of your connection to the outside world, not by the CPU speed of the bastion host itself. It just doesn’t take that much of a processor to handle mail, DNS, FTP, and proxy services for a 56 Kbps or even a T-1 (1.544 Mbps) line. You may need more power if you are running programs that do compression/decompression (e.g., NNTP servers) or searches (e.g., full-featured web servers), or if you’re providing proxy services for dozens of users simultaneously.
You may also need more power to support requests from the Internet if your site becomes wildly popular (e.g., if you create something that everybody and their mothers want to access, like the Great American Web Page or a popular and well-stocked anonymous FTP site). At that point, you might also want to start using multiple bastion hosts, as we describe in Chapter 6. A large company with multiple Internet connections and popular services may need to use multiple bastion hosts and large, powerful machines.
Building Internet Firewalls
There are several reasons not to oversize a bastion host:
• A slower machine is a less inviting target. There’s no prestige for somebody who brags, “Hey, I broke into a Sun 3/60!” or some other slow (to an attacker, at least) machine. Far more prestige is involved in breaking into the latest, greatest hardware. Don’t make your bastion host something with high prestige value (a supercomputer, for example, would be a poor choice of a bastion host). • If compromised, a slower machine is less useful for attacking internal systems or other sites. It takes longer to compile code; it’s not very helpful for running dictionary or brute-force password attacks against other machines; and so on. All of these factors make the machine less appealing to potential attackers, and that’s your goal. • A slower machine is less attractive for insiders to compromise. A fast machine that’s spending most of its time waiting for a slow network connection is effectively wasted, and the pressure from your own users to use the extra power for other things (for example, as a compilation server, rendering server, or database server) can be considerable. You can’t maintain the security of a bastion host while using it for other purposes. Extra capacity on the bastion host is an accident waiting to happen.
Web servers are an exception to this rule. You might as well size your web server optimistically, because as web sites evolve they tend to increase drastically and rapidly in both size and CPU usage. Changes in client technology also tend to increase the load on web servers (for instance, many clients open multiple connections in order to download several images at the same time, thereby increasing the performance the user sees at the cost of increasing the load on the server).
10.3.3 What Hardware Configuration?
You want a reliable hardware configuration, so you should select a base machine and peripherals that aren’t the newest thing on the market. (There’s a reason people call it “bleeding edge” as well as “leading edge” technology.) You also want the configuration to be supportable, so don’t choose something so old you can’t find replacement parts for it. The middle range from your favorite manufacturer is probably about right.
While a desktop-class machine probably has the horsepower you need, you may be better off with something in server packaging; machines packaged as servers are generally easier to exchange disks in, as well as being more possible to mount in racks when you have lots of them. They’re also harder to steal, and less likely to get turned off by people who need another outlet to plug the vacuum cleaner into.
While you don’t need sheer CPU power, you do need a machine that keeps track of a number of connections simultaneously. This is memory intensive, so you’ll want a large amount of memory and probably a large amount of swap space as well. Caching proxies also need a large amount of free disk space to use for the caches.
Here are some suggestions about tape and disk needs:
• The bastion host can’t reasonably use another host’s tape drive for backups, as we’ll discuss later in this chapter, so it needs its own tape drive of a size suitable to back itself up. • A CD-ROM drive also comes in handy for operating system installation and possibly for keeping checksums on (or for comparing your current files to the original files on the CD-ROM). You may only need the CD-ROM drive initially when you first install and configure the machine, so an external drive that you “borrow” from another machine temporarily may be sufficient. In any case, it should be a CD-ROM or single-session CDW (write) drive, not a drive that will write rewritable or multisession CDs; one of the purposes of this drive is to hold data that you know the bastion host cannot modify, even by adding data! • You should be able to easily add another disk temporarily to the configuration for maintenance work. • The boot disk should remove easily and attach to another machine – again, for maintenance work.
Both of the disk considerations mentioned suggest that the bastion host should use the same type of disks as your other machines. For example, it should not be the only machine at your site running IDE disks.
The bastion host doesn’t need interesting graphics and shouldn’t have them. This is a network services host; nobody needs to see it. Attach a dumb terminal (the dumber the better) as the console. Having graphics will only encourage people to use the machine for other purposes and might encourage you to install support programs (like the X Window System and its derivatives) that are insecure. If you are using a Windows NT machine, which requires a graphics console, use a cheap and ugly VGA display or a console switch.
Most bastion hosts are critical machines and should have appropriate high-availability hardware, including redundant disks and uninterruptible power.
Building Internet Firewalls
10.4 Choosing a Physical Location
The bastion host needs to be in a location that is physically secure.21 There are two reasons for this:
• It is impossible to adequately secure a machine against an attacker who has physical access to it; there are too many ways the attacker can compromise it. • The bastion host provides much of the actual functionality of your Internet connection, and if it is lost, damaged, or stolen, your site may effectively be disconnected. You will certainly lose access to at least some services.
Never underestimate the power of human stupidity. Even if you don’t believe that it’s worth anyone’s time and trouble to get physical access to the machine in order to break into it, secure it to prevent well-meaning people within your organization from inadvertently making it insecure or nonfunctional.
Your bastion hosts should be in a locked room, with adequate air conditioning and ventilation. If you provide uninterruptible power for your Internet connection, be sure to provide it for all critical bastion hosts as well.
10.5 Locating Bastion Hosts on the Network
Bastion hosts should be located on a network that does not carry confidential traffic, preferably a special network of their own.
Most Ethernet and token ring interfaces can operate in “promiscuous mode”. In this mode, they are able to capture all packets on the network the interfaces are connected to, rather than just those packets addressed to the particular machine the interface is a part of. Other types of network interfaces, such as FDDI, may not be able to capture all packets, but depending on the network architecture, they can usually capture at least some packets not specifically addressed to them.
This capability has a useful purpose: for network analysis, testing, and debugging, for example, by programs like Network Manager, etherfind, and tcpdump. Unfortunately, it can also be used by an intruder to snoop on all traffic on a network segment. This traffic might include Telnet, FTP, or rlogin sessions (from which logins and passwords can be captured), confidential email, NFS accesses of sensitive files, and so on. You need to assume the worst: bastion hosts can be compromised. If a bastion host is compromised, you don’t want it to snoop on this traffic.
One way to approach the problem is to not put bastion hosts on internal networks; instead, put them on a perimeter network. As we’ve discussed in earlier chapters, a perimeter network is an additional layer of security between your internal network and the Internet. The perimeter network is separated from the internal network by a router or bridge. Internal traffic stays on the internal net and is not visible on the perimeter net. All a bastion host on a perimeter network can see are packets that are either to or from itself, or to or from the Internet. Although this traffic might still be somewhat sensitive, it’s likely to be a lot less sensitive than your typical internal network traffic, and there are other places (for instance, your Internet service provider) that can already see much of it.
Using a perimeter net with a packet filtering router between it and the internal network gives you some additional advantages. It further limits your exposure, if a bastion host is compromised, by reducing the number of hosts and services the compromised bastion host can access.
If you can’t put bastion hosts on a perimeter network, you might consider putting them on a network that’s not susceptible to snooping. For example, you might put them on an intelligent 10-base T hub, an Ethernet switch, or an ATM network. If this is all you do, then you need to take additional care to make sure that nothing trusts those bastion hosts, because no further layer of protection is between it and the internal network. Using such a network technology for your perimeter network is the best of both worlds: bastion hosts are isolated from internal systems (as with a traditional perimeter network) but can’t snoop on traffic on the perimeter network.
21 Practical UNIX & Internet Security by Simson Garfinkel and Gene Spafford (second edition, O’Reilly & Associates, 1996) contains an excellent and extensive discussion of physical security.
Building Internet Firewalls
Be careful about how much trust you place in the ability to prevent hosts from snooping the network. Even with an intelligent or switched hub, broadcast traffic will be visible to all nodes, and this traffic may include data that is useful to an attacker. For instance, networks that use Microsoft directory services will include a lot of useful information about machine and filesystem names and types in broadcast traffic. There may also be information that is sensitive in multicast traffic, which any node can ask to receive. Finally, hubs of this type frequently offer an administrative capability that can control the reception of all traffic. That may be limited to a specific port or available to all ports. You should be sure that this is appropriately secured on any hub that bastion hosts are attached to; otherwise, an attacker may be able to simply ask for all traffic and get it, removing the theoretical advantages of using a hub.
Whatever networking devices you use, you should be careful to protect the networking devices to the same degree that you protect the computers. Many network devices support remote administration, often with a wide variety of interfaces (for instance, a switch may provide a Telnet server, SNMP management, and a web management interface). An intruder who can reconfigure networking devices can certainly keep your network from working and may also be able to compromise other machines. Consider disabling all remote management features (with the possible exception of remote logging of errors) and configuring network devices the old- fashioned way, with a terminal and a serial cable.
10.6 Selecting Services Provided by a Bastion Host
A bastion host provides any services your site needs to access the Internet, or wants to offer to the Internet – services you don’t feel secure providing directly via packet filtering. (Figure 10.1 shows a typical set.) You should not put any services on a bastion host that are not intended to be used to or from the Internet. For example, it shouldn’t provide booting services for internal hosts (unless, for some reason, you intend to provide booting services for hosts on the Internet). You have to assume that a bastion host will be compromised, and that all services on it will be available to the Internet.
Figure 10.1. The bastion host may run a variety of Internet services
You can divide services into four classes:
Services that are secure
Services in this category can be provided via packet filtering, if you’re using this approach. (In a pure- proxy firewall, everything must be provided on a bastion host or not provided at all.)
Services that are insecure as normally provided but can be secured
Services in this category can be provided on a bastion host.
Building Internet Firewalls
Services that are insecure as normally provided and can’t be secured
These services will have to be disabled and provided on a victim host (discussed earlier) if you absolutely need them.
Services that you don’t use or that you don’t use in conjunction with the Internet
You must disable services in this category.
We’ll discuss individual services in detail in later chapters, but here we cover the most commonly provided and denied services for bastion hosts.
Electronic mail (SMTP) is the most basic of the services bastion hosts normally provide. You may also want to access or provide information services such as:
Hypertext-driven information retrieval (the Web)
In order to support any of these services (including SMTP), you must access and provide Domain Name System (DNS) service. DNS is seldom used directly, but it underlies all the other protocols by providing the means to translate hostnames to IP addresses and vice versa, as well as providing other distributed information about sites and hosts.
Many services designed for local area networks include vulnerabilities that attackers can exploit from outside, and all of them are opportunities for an attacker who has succeeded in compromising a bastion host. Basically, you should disable anything that you aren’t going to use, and you should choose what to use very carefully.
Bastion hosts are odd machines. The relationship between a bastion host and a normal computer on somebody’s desktop is the same as the relationship between a tractor and a car. A tractor and a car are both vehicles, and to a limited extent they can fulfill the same functions, but they don’t provide the same features. A bastion host, like a tractor, is built for work, not for comfort. The result is functional, but mostly not all that much fun.
For the most part, we discuss the procedures to build a bastion host with the maximum possible security that allows it to provide services to the Internet. Building this kind of bastion host out of a general-purpose computer means stripping out parts that you’re used to. It means hearing people say “I didn’t think you could turn that off!” and “What do you mean it doesn’t run any of the normal tools I’m used to?”, not to mention “Why can’t I just log into it?” and “Can’t you turn on the software I like just for a little while?” It means learning entirely new techniques for administering the machine, many of which involve more trouble than your normal procedures.
10.6.1 Multiple Services or Multiple Hosts?
In an ideal world, you would run one service per bastion host. You want a web server? Put it on a bastion host. You want a DNS server? Put it on a different bastion host. You want outgoing web access via a caching proxy? Put it on a third bastion host. In this situation, each host has one clear purpose, it’s difficult for problems to propagate from one service to another, and each service can be managed independently.
In the real world, things are rarely this neat. First, there are obvious financial difficulties with the one service, one host model – it gets expensive fast, and most services don’t really need an entire computer. Second, you rapidly start to have administrative difficulties. What’s the good in having one firewall if it’s made up of 400 separate machines?
Building Internet Firewalls
You are therefore going to end up making trade-offs between centralized and distributed services. Here are some general principles for grouping services together into sensible units:
Group services by importance
If you have services that your company depends on (like a customer-visible web site) and services you could live without for a while (like an IRC server), don’t put them on the same machine.
Group services by audience
Put services for internal users (employees, for instance) on one machine, services for external users (customers, for instance) on another, and housekeeping services that are only used by other computers (like DNS) on a third. Or put services for faculty on one machine and services for students on a different one.
Group services by security
Put trusted services on one machine, and untrusted services on another. Better yet, put the trusted services together and put each untrusted service on a separate machine, since they’re the ones most likely to interfere with other things.
Group services by access level
Put services that deal with only publicly readable data on one machine, and services that need to use confidential data on another.
Sometimes these principles will be redundant (the unimportant services are used by a specific user group, are untrustworthy, and use only public data). Sometimes, unfortunately, they will be conflicting. There is no guarantee that there is a single correct answer.
10.7 Disabling User Accounts on Bastion Hosts
If at all possible, don’t allow any user accounts access to bastion hosts. For various reasons, bastion hosts may know about users, but users should not have accounts that actually allow them to use the host. Keeping such accounts off bastion hosts will give you the best security. There are several reasons why, including:
• Vulnerabilities of the accounts themselves • Vulnerabilities of the services required to support the accounts • Reduced stability and reliability of the machine • Inadvertent subversion of the bastion host’s security by users • Increased difficulty in detecting attacks
User accounts provide relatively easy avenues of attack for someone who is intent on breaking into a bastion host. Each account usually has a reusable password that can be attacked through a variety of means, including dictionary searches, brute force searches, or capture by network eavesdropping. Multiply this by many users, and you have a disaster in the making.
Supporting user accounts in any useful fashion requires a bastion host to enable services (for example, printing and local mail delivery services) that could otherwise be disabled on the bastion host. Every service that is available on a bastion host provides another avenue of attack, through software bugs or configuration errors.
Having to support user accounts also can reduce the stability and reliability of the machine itself. Machines that do not support user accounts tend to run predictably and are stable. Many sites have found that machines without users tend to run pretty much indefinitely (or at least until the power fails) without crashing.
Building Internet Firewalls
Users themselves can contribute to security problems on bastion hosts. They don’t (usually) do it deliberately, but they can subvert the system in a variety of ways. These range from trivial (e.g., choosing a poor password) to complex (e.g., setting up an unauthorized information server that has unknown security implications). Users are seldom trying to be malicious; they’re normally just trying to get their own jobs done more efficiently and effectively.
It’s usually easier to tell if everything is “running normally” on a machine that doesn’t have user accounts muddying the waters. Users behave in unpredictable ways, but you want a bastion host to have a predictable usage pattern, in order to detect intrusions by watching for interruptions in the pattern.
If you need to allow user accounts on a bastion host, keep them to a minimum. Add accounts individually, monitor them carefully, and regularly verify that they’re still needed.
There is one circumstance where you should have user accounts. Every person who needs to log into a bastion host for administrative purposes should have an individual account and should log in with that account. Nobody should log into the machine directly as “administrator” or “root” if there is any other way for them to get work done. These accounts should be kept to a minimum and closely controlled. It should be made impossible to reach these accounts from the Internet with a reusable password (if the capability is there, some administrator will use it). In fact, it’s better not to allow access to the accounts from the Internet at all, and you might want to consider disallowing network logins altogether. (Note that whitehouse.gov was broken into because its administrators, who knew better, succumbed to temptation and logged into it across the Internet to do administration.) We will discuss appropriate mechanisms for remote administration in the following chapters about specific operating systems.
10.8 Building a Bastion Host
Now that you’ve figured out what you want your bastion host to do, you need to actually build the bastion host. This process of configuring a machine to be especially secure and resistant to attack is generally known as hardening. The basic hardening process is as follows:
- Secure the machine. 2. Disable all nonrequired services. 3. Install or modify the services you want to provide. 4. Reconfigure the machine from a configuration suitable for development into its final running state. 5. Run a security audit to establish a baseline. 6. Connect the machine to the network it will be used on.
You should be very careful to make sure the machine is not accessible from the Internet until the last step. If your site isn’t yet connected to the Internet, you can simply avoid turning on the Internet connection until the bastion host is fully configured. If you are adding a firewall to a site that’s already connected to the Internet, you need to configure the bastion host as a standalone machine, unconnected to your network.
If the bastion host is vulnerable to the Internet while it is being built, it may become an attack mechanism instead of a defense mechanism. An intruder who gets in before you’ve run the baseline audit will be difficult to detect and will be well positioned to read all of your traffic to and from the Internet. Cases have been reported where machines have been broken into within minutes of first being connected to the Internet; while rare, it can happen.
Take copious notes on every stage of building the system. Assume that sometime in the future, a compromise will occur that causes the machine to burst into flames and be destroyed. In order to rebuild your system, you will need to be able to follow all of the steps you took previously.
You will also need all of the software that you used, so you should be sure to securely store all of the things you need to do the installation, including:
• The disks, CDs, or tapes you install software from • The source code for any software you build from source • The environment you used to build software from source, if it’s different from the one you’re installing; this includes the operating system, compiler, and header files (and a machine they run on) • The manuals and documents you were working from
Building Internet Firewalls
The following sections briefly describe each of the main steps involved in building a bastion host; these steps will be covered in more detail in the following separate chapters for Unix and Windows NT. They also touch briefly on ongoing maintenance and protection of the bastion host; note, though, that maintenance issues are discussed primarily in Chapter 26.
10.9 Securing the Machine
To start with, build a machine with a standard operating system, secured as much as possible. Start with a clean operating system and follow the procedures we describe in this section:
- Start with a minimal clean operating system installation. 2. Fix all known system bugs. 3. Use a checklist. 4. Safeguard the system logs.
10.9.1 Start with a Minimal Clean Operating System Installation
Start with a clean operating system installation, straight from vendor distribution media. If you do this, you will know exactly what you’re working with. You won’t need to retrofit something that may already have problems. Using such a system will also make later work easier. Most vendor security patches you later obtain, as well as the vendor configuration instructions and other documentation, assume that you’re starting from an unmodified installation.
While you’re installing the operating system, install as little as you can get away with. It’s much easier to avoid installing items than it is to delete them completely later on. For that matter, once your operating system is minimally functional, it’s not hard to add components if you discover you need them. Don’t install any optional subsystems unless you know you will need them.
If you are reusing a machine that has already had an operating system installed on it, be sure to erase all data from the disks before doing the reinstall. Otherwise, you cannot guarantee that all traces of the old system are gone.
10.9.2 Fix All Known System Bugs
Get a list of known security patches and advisories for your operating system; work through them to determine which are relevant for your own particular system, and correct all of the problems described in the patches and advisories. You can get this information from your vendor sales or technical support contacts, or from the user groups, newsgroups, or electronic mailing lists devoted to your particular platform.
In addition, be sure to get from the Computer Emergency Response Team Coordination Center (CERT-CC) any advisories relevant to your platform, and work through them. (For information on how to contact CERT-CC and retrieve its information, see the list of resources in Appendix A.)
Many operating systems have both recommended and optional patches or have periodic patch sets (called service packs for Windows NT) with individual patches issued in between (Microsoft calls these hot fixes). You should install the current recommended patch set, plus all other security-related patches that are relevant to your installation.
10.9.3 Use a Checklist
To be sure you don’t overlook anything in securing your bastion host, use a security checklist. Several excellent checklists are around. Be sure to use one that corresponds to your own platform and operating system version.
Building Internet Firewalls
10.9.4 Safeguard the System Logs
As a security-critical host, the bastion host requires considerable logging. The next step in building the bastion host is to make sure that you have a way of safeguarding the system logs for the bastion host. The system logs on the bastion host are important for two reasons:
• They’re one of the best methods of determining if your bastion host is performing as it should be. If everything the bastion host does is logged (and it should be), you should be able to examine the logs to determine exactly what it’s doing and decide if that’s what it’s supposed to be doing. Chapter 26, describes the use of system logs in maintaining your firewall. • When (not if!) someday someone does successfully break in to the bastion host, the system logs are one of the primary mechanisms that determine exactly what happened. By examining the logs and figuring out what went wrong, you should be able to keep such a break-in from happening again.
Where should you put the system logs? On the one hand, you want the system logs to be somewhere convenient; you want them to be where they can be easily examined to determine what the bastion host is doing. On the other hand, you want the system logs to be somewhere safe; this will keep them from any possible tampering in case you need to use them to reconstruct an incident.
The solution to these seemingly contradictory requirements is to keep two copies of the system logs – one for convenience, the other for catastrophes. The details of the logging services are operating-system dependent and are discussed in the chapters on individual operating systems.
10.9.4.1 System logs for convenience
The first copy of the system logs is the one you’ll use on a regular basis to monitor the ongoing activity of the machine. These are the logs against which you run your daily and weekly automated analysis reports. You can keep these logs either on the bastion host itself or on some internal host.
The advantage of keeping them on the bastion host is simplicity: you don’t have to set up logging to go to some other system, nor do you have to configure the packet filters to allow this. The advantage to keeping them on an internal host is ease of access: you don’t have to go to the bastion host, which doesn’t have any tools anyway, to examine the logs. Avoid logging in to the bastion host, in any case.
10.9.4.2 System logs for catastrophes
The second copy of the system logs is the one you’ll use after a catastrophe. You can’t use your convenience logs at a time like this. Either the convenience logs won’t be available, or you won’t be sure of their integrity any longer.
These logs need to be kept separate from the bastion host and kept for a long time. Sometimes you will discover an intruder a long time after the original compromise (among other things, it’s not unusual for an intruder to break into a bunch of machines and install back doors for later use; a compromised machine may be left alone for months).
If you have a write-once device available to you, use that device; doing so is probably the technically easiest way to keep the logs, especially if your write-once device can emulate a filesystem. Be sure you can trust the write- once feature. Some magneto-optical drives are capable of both multiple-write and write-once operations and keep track of the mode they’re in via software. If the system is compromised, it may be possible to overwrite or damage previously written parts of the supposedly write-once media.
The other methods available to you will differ depending on the operating system you are using and are discussed in Chapter 11, and Chapter 12.
10.9.4.3 Logging and time
Knowing the time (within minutes and sometimes seconds) when something occurred can be very useful when dealing with break-ins. You will need date and time information if you (or law enforcement agencies) need to request information from other sites. You should make sure that your bastion hosts have accurate and synchronized times in their logs. See Chapter 22, for more information about time synchronization protocols.
10.9.4.4 Choosing what to log
Choosing the information you want to log is a delicate business. You don’t want gigantic logs full of routine events; that just wastes space and time and makes it harder to find important information. On the other hand, you do want logs that are general enough that you can debug problems and figure out what intruders did.
What you would like to do is to log everything except events that are frequent and nonthreatening. Don’t try to limit your logging to dangerous or interesting events because it’s hard to successfully predict which those are going to be. Instead, log everything you can stand, eliminating only the known clutter.
For instance, Windows NT provides the ability to log all accesses to files. You don’t want to turn this on for all files on a bastion host; you’ll drown in routine accesses to files that are accessed as it provides services. On the other hand, you probably do want to log all accesses to system files that aren’t accessed by the services. These files shouldn’t be touched often, and the nuisance caused by the log entries when you do maintenance work will be compensated for by the number of attacks you can detect.
10.10 Disabling Nonrequired Services
Once you’ve completed the basic process of securing your bastion host, go on to the next step: disabling any services that aren’t absolutely necessary for the bastion host to provide. You will want to disable all services except the ones you have decided to provide, and the supporting services necessary for those to run, as described earlier. You may not always know which services are the required support services, particularly because service names tend to be cryptic and uninformative.
How do you know which services to disable? There are three simple rules to apply:
• If you don’t need it, turn it off. • If you don’t know what it does, turn it off (you probably didn’t need it anyway). • If turning it off causes problems, you now know what it does, and you can either turn it back on again (if it’s really necessary) or figure out how to do without it.
Any service provided by the bastion host might have bugs or configuration problems that could lead to security problems. Obviously, you’ll have to provide some services that users need, as long as your site’s security policy allows them. But if the service isn’t absolutely necessary, don’t borrow trouble by providing it. If a service isn’t provided by the bastion host, you won’t have to worry about possible bugs or configuration problems.
If you can live without a service, it should be turned off. It’s worth suffering some inconvenience. This means that you’re going to need to think very carefully about services. You’ll be disabling not just services you never heard of and never used, but also services you’ve purposefully enabled on other machines (and, unfortunately, services you’ve never heard of because they’re considered too basic ever to do anything to). Look at every service and ask yourself “How could I avoid enabling this? What do I lose if I turn it off ?”
10.10.1 How to Disable Services
The first step in disabling services is ensuring that you have a way to boot the machine if you accidentally disable a critical service. This could be a second hard disk with a full boot partition on it or a CD-ROM drive with the operating system install disk. It could even be a second installation of the operating system on the same hard disk. You need to be ruthless; if you can’t reboot when you delete the wrong thing, at best you’re going to be over-cautious about deleting things, and at worst you’re going to end up with an unusable computer. (These fallback operating systems are also useful places to copy files from or compare files to if things go wrong.)
Second, you must save a clean copy of every file before you modify it. Even when you’re just commenting things out, every so often your fingers slip, and you delete something you didn’t mean to, or you change a critical character. If you are using a user interface to change things instead of directly modifying files, you may not know what files are actually being changed, so you may need to simply back up the entire system. If possible, do this with another disk, rather than with a standard program and a tape; if you have to back out a change, you will want to be able to replace just the files that are actually involved, and that’s easiest to determine by comparing things on disk. On Windows NT, you should note that the registry is not backed up or copied by normal procedures; be sure that you include it. You will also want to build a new Emergency Repair Disk (which includes the most important parts of the registry) immediately before you begin the reconfiguration.
Building Internet Firewalls
When you disable a service, you should also disable all services that depend on it. This will prevent nasty warning messages and will also mean that reenabling a service will not result in a cascade of unfortunate surprises as other services are also turned on.
Finally, we’ve said it before and we’ll say it again: you should not connect a machine to a hostile network until it has been fully configured. That means that all of your work on disabling services should be done with the machine either entirely disconnected from the network, or on a safe test network. The reason that you are disabling services is that they are unsafe, and if you are connected to a hostile network, they may be exploited before you finish disabling them.
10.10.1.1 Next steps after disabling services
In general, you’ll need to reboot your machine after you have changed the configuration files. The changes won’t take effect until you do so.
After you have rebooted and tested the machine, and you are comfortable that the machine works without the disabled services, you may want to remove the executables for those services (as long as they are not used by other services). If the executables are lying around, they may be started by somebody – if not you, some other system administrator, or an intruder. A few services may even be executable by nonroot users if they use nonstandard ports.
If you feel uncertain about removing executables, consider encrypting them instead. You should use a program that provides genuinely strong encryption. The Unix crypt program is not appropriate; neither are many of the available packages for Microsoft systems. Instead, use a more secure encryption program like snuffle or something that uses the DES or IDEA algorithm. Choose a secure key; if you forget the key, you’re no worse off than if you’d deleted the files, but if an intruder gets the key, you’re considerably worse off.
10.10.2 Running Services on Specific Networks
In some cases, you want to run services that need to respond to only one network on a machine with multiple network interfaces. You may be able to limit those services to just the networks you wish to use them on. Under Unix, this usually means specifying which IP addresses and/or network interfaces you want the service to respond to as part of the service’s startup options; this will be slightly different for every service, and not all services provide this facility. Under Windows NT, only a few basic services can be controlled this way. In the Networking control panel, go to the Bindings tab and set it to show bindings for all adapters. Select the services that you wish to turn off and press the Disable button.
10.10.3 Turning Off Routing
If you have a dual-homed host that is not supposed to be a router, you will need to specifically disable routing. In order to act as an IP router, a dual-homed host needs to accept packets that are addressed to other machines’ IP addresses, and send them on appropriately. This is known as IP forwarding, and it’s usually implemented at a low level in the operating system kernel. An IP-capable host with multiple interfaces normally does this automatically, without any special configuration.
Other machines have to know that the dual-homed host is a router in order to use it as such. Sometimes this is done simply by configuring those machines to always route packets for certain networks to the dual-homed host (this is called static routing). More often, however, the dual-homed host is configured to broadcast its routing capabilities via a routing protocol such as Routing Information Protocol (RIP). Other machines hear these routing broadcasts and adjust their own routing tables accordingly (this is called dynamic routing). This broadcast of routing information by the dual-homed host is usually done by an additional program (for example, routed or gated on a Unix system), which often has to be turned on explicitly.
To use a dual-homed host as a firewall, you need to convert it to a nonrouting dual-homed host; you take a machine that has two network interfaces, and you configure it so it can’t act as a router between those two interfaces. This is a two-step process:
- Turn off any program that might be advertising it as a router; this is usually relatively straightforward. 2. Disable IP forwarding; this can be equally easy or considerably more difficult, and may require modifying the operating system kernel.
Unfortunately, turning off IP forwarding does not always turn off all routing. On some systems, you can turn off IP forwarding, but the IP source-routing option usually remains a security hole.
Building Internet Firewalls
What is source routing ? Normal IP packets have only source and destination addresses in their headers, with no information about the route the packet should take from the source to the destination. It’s the job of the routers in between the source and the destination to determine the most efficient route. However, source-routed IP packets contain additional information in the IP header that specifies the route the packet should take. This additional routing information is specified by the source host – thus the term “source-routed”.
When a router receives a source-routed packet, it follows the route specified in the packet, instead of determining the most efficient route from source to destination. The source-routing specification overrides the ordinary routing. Because of the way the routing code is implemented in many operating systems, turning off IP forwarding does not disable forwarding of source-routed packets. It’s implemented completely separately and must be turned off separately, often by completely different (and more difficult) mechanisms.
Source-routed packets can easily be generated by modern applications like the Telnet client that’s freely available on the Internet as part of the BSD 4.4 release. Unless you block source-routed packets somewhere else, such as in a router between the dual-homed host and the Internet, source-routed packets can blow right past your dual- homed host and into your internal network.
Worse still, source routing goes both ways. Once source-routed packets make their way to an internal system, the system is supposed to reply with source-routed packets that use the inverse of the original route. The reply from your internal system back to the attacker will also blow right through your dual-homed host, allowing two- way connection through a firewall that was supposed to block all communications across it.
Fortunately, it is now common practice for firewalls to ignore all source routing, either by dropping packets with source routing or by stripping the source routing itself. In addition, systems that will accept source routes will rarely include them on the return packet.
If you are not going to screen your dual-homed host, you will need to patch your operating system so that it rejects source-routed packets. Consult your vendor, and/or appropriate security mailing lists (discussed in Appendix A) for information on how to do this on your platform.
10.10.4 Controlling Inbound Traffic
As we discussed in Chapter 8, many general-purpose computers are provided with packet filtering packages. Even when these packages are not adequate for building packet filtering routers, they can provide an extra level of protection for bastion hosts. If packet filtering is available to you, you should set it up so that it allows only the traffic that you intend to support. In most configurations, this will be multiply redundant; it will duplicate protections provided on routers, and most of the rules will prevent connections to services that don’t exist anyway. This is a useful kind of redundancy, which will help to protect you from configuration errors.
Packet filters will also keep you from successfully adding new services to the machine. You should document the filters carefully to avoid puzzling failures later.
10.10.5 Installing and Modifying Services
Some of the services you want to provide may not be provided with your operating system. Others may be provided with servers that are inappropriate for use in a secure environment or are missing features you probably want (for example, stock fingerd and ftpd ). Even those few remaining services that are provided, secure, and up to date in your vendor’s operating system release usually need to be specially configured for security.
For information on general schemes for protecting services in the operating system you are using, see Chapter 11, and Chapter 12, as appropriate. For detailed information about individual services, including advice on selecting HTTP, NNTP, and FTP servers, see the chapters relevant to the services you want to provide (for instance, Chapter 15, for HTTP; Chapter 16, for NNTP; and Chapter 17, for FTP).
10.10.6 Reconfiguring for Production
Now it’s time to move the machine from the configuration that was useful to you when you were building it to the best configuration for running it. You’ll need to do several things:
- Finalize the operating system configuration. 2. Remove all unnecessary programs. 3. Mount as many filesystems as possible as read-only.
Building Internet Firewalls
10.10.6.1 Finalize the operating system configuration
Once you’ve deleted all the services that aren’t used on a day-to-day basis, you’ll find that it is very difficult to work on the bastion host – for example, when you need to install new software packages or upgrade existing ones. Here are some suggestions for what to do when you find it necessary to do extensive work on the bastion host:
• Write all the tools to tape before deleting them, and then restore them from tape when needed. Don’t forget to delete them each time after you’re done. • Set up a small, external, alternate boot disk with all the tools on it. Then, plug the disk in and boot from it when you need the tools. Don’t leave the disk connected during routine operations, however; you don’t want an attacker to be able to mount the disk and use the tools against you.
You don’t want an intruder to attack the machine while you’re working on it. To keep that from happening, follow these steps:
- Either disconnect the bastion host from the network or disconnect your network from the Internet before you begin. 2. Give the bastion host back the tools you’ll need to use (as we’ve described earlier). 3. After you’ve finished your work on the machine, return it to its normal (stripped down) operating condition. 4. Reconnect the bastion host to the network or your network to the Internet.
You may find it easier to simply remove the bastion host’s disk and attach it to an internal host as a nonsystem disk; you can then use the internal host’s tools without fear of having them remain available when the bastion host is returned to service. This procedure also guarantees that the bastion host is not vulnerable to compromise from the outside while you are doing the work, since it is entirely nonfunctional while its disk is removed and not susceptible to accidental reconnection.
10.10.6.2 Mount filesystems as read-only
Once you’ve got the bastion host configured, you don’t want anybody (particularly an attacker) to be able to change the configuration. To guard against this happening, mount the filesystems on the bastion host as read- only if possible (particularly the filesystems that contain program binaries) to protect against tampering.
It’s much better if you can use hardware write-protect; an attacker may be able to remount disks with write permission without getting physical access to the machine, but it’s not going to do any good if the hardware write-protect on the disk is on. Many SCSI disks have a “write-disable” jumper or switch you can set. If you find powering the disk down and removing it from the case unacceptable as a way to get write access, you could wire this jumper to an external switch on the drive enclosure.
10.10.7 Running a Security Audit
Once you’ve got the bastion host reconfigured, the next step is to run a security audit. There are two reasons for doing this. First, it gives you a way to ensure you haven’t overlooked anything during system setup. Second, it establishes a “baseline”, or a basis for comparison, against which you can compare future audits. In this way, you’ll be able to detect any tampering with the machine.
10.10.7.1 Auditing packages
Most auditing packages have two basic purposes:
Checking for well-known security holes
These are holes that have been uncovered by system administrators, exploited by attackers in system break-ins, or documented in computer security books and papers.
Establishing a database of checksums of all files on a system
Doing this allows a system administrator to recognize future changes to files – particularly unauthorized changes.
Building Internet Firewalls
Several very good automated auditing packages are freely available on the Internet.
How do you use the various auditing packages to audit your system? The details of what you do depend upon which package you’re using. (See the documentation provided with the packages for detailed instructions.) This section provides some general tips.
You will need to do some configuration. Don’t just install the program, run it, and expect you’ll get reasonable results. Expect to go through several iterations of running the auditing package, getting warnings, and reconfiguring your machine or the auditing package to get rid of warnings. When you get warnings, you have to decide whether the auditing package is wrong, or you are. There will be some cases where the right thing to do is to turn off checks, but it shouldn’t be your automatic response.
Once you’ve used the tools described in the previous section to create your initial baseline, store a copy of the tools and these initial audit results somewhere safe. Under no circumstances should you store the only copy of the baseline or the tools on the bastion host. Prepare for the worst: if someone were to break into the bastion host and tamper with the only copy of the baseline audit, this would compromise your ability to use the audit later on to detect illicit changes on the system. If intruders can change the auditing software, it doesn’t matter whether they can change the baseline; they could simply set up the auditing software to reproduce the baseline. Keeping a copy of the baseline audit on a floppy disk or magnetic tape that’s locked up some place safe is a good way to protect against such a compromise. Preferably, you don’t want an intruder to even read the audit results; why tell them what you expect the system to look like and what files you aren’t watching?
Periodically, (e.g., daily or weekly, depending on your own site’s needs and capabilities), audit the machine once again and compare the new audit to the baseline. Make sure you can account for any differences you find. Ideally, you should automate this periodic reaudit so it happens regularly and reliably. Unfortunately, this is easier said than done. It can be difficult to arrange automatic audits that can’t be defeated by “replay” attacks. In a replay attack, an attacker who has compromised your auditing system simply sends you a recording of a prior good audit whenever your system invokes the automatic auditing capability. The most practical defense against this is to run your automated auditing system often enough that it’s unlikely an attacker could break in, discover the auditing system, and subvert it (covering his tracks) before the next audit runs. This suggests that you should run an audit at least daily. It may help to run the audit at random intervals, although it can be difficult to automate this well. It is better to run the audit at frequent but predictable intervals than to rely on human beings remembering to run it by hand.
If you start receiving warnings from the auditing system and you decide that they are incorrect, you should immediately reconfigure the auditing system or the operating system so that the warnings go away. If you get used to getting warnings, you will end up ignoring important new messages. Also, if you go on vacation, your replacement may not realize that the messages are benign and may take drastic action to remedy nonproblems.
10.10.7.2 Use cryptographic checksums for auditing
Checksums are very helpful in auditing. An intruder who changes a program or configuration file will almost certainly correct the modification dates afterwards, so you can’t use these dates as a reliable index. Comparing every file to a baseline copy avoids that problem but takes a lot of time and requires that you store a copy of every single file, effectively doubling your storage requirements. Storing checksums is probably your best bet.
A checksum is a number calculated on data that is designed to detect changes to the data. This is useful for a communications channel; if a sender calculates a checksum as data is being sent and a receiver does the same, then the two can simply compare checksums to see if the data was changed during transmission. You can also do exactly the same checksum calculation for files, but instead of sending the file elsewhere, you recalculate and compare the checksum at a later time. Calculating checksums can be time consuming because you have to read the contents of every file, but it is not as time consuming as reading everything twice and doing a bit-by-bit compare. In addition, storing a checksum takes up much less space than storing an entire file.
However, checksums are not full representations of the file, and every checksum algorithm has cases where it will give the same checksum for two different files. This is called a collision, and checksum algorithms are designed to make this unlikely to occur for the differences they are designed to detect.
Building Internet Firewalls
In order for a checksum to be useful in detecting unauthorized changes to files, it must have several characteristics:
• It must be practically impossible to deliberately create a file that has a checksum that matches another. This can be achieved by designing the algorithm so that it cannot be reversed and run backwards (you can’t start with a checksum and use a known method to create a file that produces that checksum). • The checksum must be of a large enough size so that you cannot create a list of files, one for each value the checksum can have, and match a given checksum that way. In practical terms, this means that a useful checksum should be larger than 128 bits in size. • If you change something only very slightly in the file, the checksum must change by a large amount.
A checksum algorithm that has these characteristics is sometimes called a cryptographic checksum. Cryptographic checksums are discussed further in Appendix C.
You will sometimes hear rumors that these algorithms are vulnerable to the same sort of trickery that can be used with standard checksums. This is not true; there are no known incidents where anybody has managed to subvert a cryptographic checksum. These rumors are based on three grounds:
- They’re due to confusions with CRC-style checksums, which are in fact often subverted. 2. They’re due to incidents in which people have missed changes when using cryptographic checksums because intruders have been able to rewrite the checksum database or replace the checksumming program. 3. They’re due to misunderstanding of some technical arguments about the security of early cryptographic checksums. Such algorithms are no longer used because of theoretical weaknesses, but those weaknesses were never exploited and are not present in current cryptographic checksums.
It is important not to run checksums on files that are supposed to change and to update checksum data promptly when you make intentional changes. If there are frequent false warnings from the checksum system, you will miss genuine problems.
10.10.8 Connecting the Machine
Now that you have the machine fully secured, you can finally connect it to its destination network and run it. You want to do this when you’re going to be around to see what happens; don’t make it the last thing you do before that long overdue vacation.
10.11 Operating the Bastion Host
Once you put the bastion host into production, your job has only just begun. You’ll need to keep a close watch on the operations of the bastion host. Chapter 26, provides more information on how to do this; this section discusses specific concerns for bastion hosts.
10.11.1 Learn What the Normal Usage Profile Is
If you’re going to monitor the bastion host, looking for abnormalities that might indicate break-ins or other types of system compromise, you will need to first develop an understanding of what the “normal” usage profile of the bastion host is. Ask these questions and others like them:
• How many jobs tend to be running at any one time? • How much CPU time do these jobs consume relative to each other? • What is the typical load at different times throughout the day?
Your goal is to develop an almost intuitive grasp of what your system normally runs like, so you’ll be able to recognize – and investigate – anomalous activity very quickly.
10.11.2 Consider Using Software to Automate Monitoring
Doing a thorough job of system monitoring is tough. Although the logs produced by your system provide lots of useful information, it’s easy to get overwhelmed by the sheer volume of logging data. The important information may often be buried. Too often, the logs end up being used only after a break-in, when, in fact, they could be used to detect – and thus perhaps stop – a break-in while it is occurring.
Because each operating system and site is different, each bastion host is configured differently, and each site has different ideas about what the response of a monitoring system should be. For example, some want electronic mail; some want the output fed to an existing SNMP-based management system, some want the systems to trip the pagers of the system administrators, and so on. Monitoring tends to be very site- and host-specific in the details.
A large and growing number of monitoring packages is available for Unix, including both freely available and commercial options. Among the freely available options, NOCOL and NetSaint are both popular, extensible systems that provide the ability to watch logs, to test to make certain machines are still running and providing services, and to alert people when things go wrong (see Appendix B, for information about how to get them).
MRTG is a special sort of monitoring package, which provides graphing services but not alerting services. It is extremely useful for watching trends. Furthermore, MRTG makes very impressive web pages with very little effort, so you not only find out what’s going on, you also get an important public relations tool for convincing people that you know what’s going on. Information about MRTG is also available in Appendix B.
Normally, monitoring of Windows NT systems is done with the Performance Monitor. Unfortunately, Performance Monitor is yet another tool based on SMB transactions, which cannot be used without enabling all of SMB. Furthermore, Performance Monitor is fairly limited as a monitoring solution for critical systems; it doesn’t provide all of the alarm and process-monitoring features you may want.
You will probably want to use an SNMP-based monitoring tool. Windows NT provides an SNMP server, so all you will need to add is the monitoring tool. Some public domain monitoring tools are now available for Windows NT, although fewer than there are for Unix. Some tools that were originally available only under Unix have now been ported to Windows NT (for instance, MRTG). Unix-based monitoring tools will monitor Windows NT systems without problems. In addition, there are a large number of commercial SNMP-based tools you can use.
10.12 Protecting the Machine and Backups
Once the bastion host has been fully configured and is in operation, protect the physical machine and make sure that its backups are protected from theft or other compromise.
10.12.1 Watch Reboots Carefully
How will you know if someone has breached security? Sometimes, it’s painfully obvious. But sometimes, you’ll have to draw conclusions from the behavior of the system. Unexplained reboots or downtime on the system may be a clue. Many attacks (e.g., modifying a kernel) can’t succeed unless the system is rebooted.
On the bastion host, crashes and reboots should be rare occurrences. Once the bastion host has been fully configured and is in production, it should be a very stable system, often running for weeks or months at a stretch without a crash or a reboot. If a crash or a reboot does occur, investigate it immediately to determine whether it was caused by some legitimate problem or might have been the result of some kind of attack.
You might want to consider configuring the bastion host so that it doesn’t bring itself up automatically after an attempted reboot. That way, if someone does manage to crash or force a reboot of the machine, you’ll know about it: the machine will sit there waiting for you to reboot it. The machine won’t be able to come back up until you decide it should do so. Many machines treat crashes and explicit reboots differently, and while most of them will let you disable an automatic reboot on a crash, it may be harder to disable an automatic reboot after a clean shutdown that requests a reboot. Even if your machine does not appear to allow you to disable autobooting, you can usually cause autoboots to fail under Unix by configuring the machine to autoboot from a nonexistent disk. (Be sure to leave instructions on how to boot the machine by hand with the machine.) Under Windows NT, you can simply edit boot.ini to set the timeout to -1, and it will wait forever for a human being to specify what operating system to boot. This has the advantage of being self-explanatory to an operator sitting in front of the console.
Building Internet Firewalls
10.12.2 Do Secure Backups
Backups on a bastion host are tricky because of trust issues. Who can you trust?
You definitely don’t want internal machines to trust the bastion host enough for it to dump to their tape drives. If the bastion host has somehow been compromised, this could be disastrous. You also don’t want the bastion host to trust the internal machines; this could lead to subversion of the bastion host by (well-intentioned) internal users, or to attack from some host pretending to be an internal system.
Common remote backup mechanisms (for example, those used by the BSD dump and rdump programs) will probably be blocked by packet filtering between the bastion host and the internal systems anyway. Therefore, you will normally want to do backups to a tape device attached directly to the bastion host. Under no circumstances should you rely on backing up the bastion host to disks that remain attached to the bastion host. You must do backups that are removed from the bastion host so they cannot be accessed by an attacker who compromises it.
Fortunately, because the bastion host is an infrequently changing machine, you won’t have to do frequent backups. Once the bastion host is fully configured and in production, it should be very stable. A weekly or even monthly manual backup will probably be sufficient.
Backups of the bastion host aren’t done just to guard against normal system catastrophes like disk crashes. They’re also a tool that you can use later to investigate a break-in or some other security incident. They give you a way to compare what’s currently on the bastion host’s disk with what was there before the incident.
If you’re only doing weekly or monthly backups, how you handle logging becomes an issue. If the bastion host is not being backed up daily, you must do your logging to some system other than the bastion host itself. If an incident does occur, the logs are going to be critical in reconstructing what happened. If it turns out that your only copy of the logs was on the (compromised) bastion host, and backups of the logs haven’t been done for three weeks, you’re going to be severely hampered in your investigative efforts.
As with all backups on all systems, you need to guard your bastion host backups as carefully as you guard the machine itself. The bastion host backups contain all the configuration information for the bastion host. An attacker who gets access to these backups would be able to analyze the security of your bastion host without ever touching it. The information these backups provide might possibly include a way to break in without setting off any of the alarms on the bastion host.
10.12.3 Other Objects to Secure
In addition to securing the backups, you will need to physically secure anything else that contains important data about the machine. This includes:
• The log files • Any alternate boot disks you use to do maintenance • The Emergency Repair Disks for Windows NT bastion hosts (including account data!) • The documentation for the details of the bastion host configuration
Although secrecy is not sufficient to give you security, it’s an important part of maintaining security. You should treat the configuration details of your bastion hosts as proprietary information, available only to people you trust. Anybody who has this information can compromise your firewall.
Published @ September 29, 2021 1:34 pm