There are a number of good articles about nature sound recording on the internet. However, they are often aimed at people who want to get truely professional results and are prepared to spend thousands of pounds in the process. This article is intended to show that you can get pretty good (although perhaps not professional quality) results from equipment available for £100 or less.
I used a Sony MZ-NH700.
Yes, even though Minidisc has become a failed format, it still lives on in the niche of outdoor sound recording. It turns out that they are very good indeed at making high quality sound recordings with minimal equipment noise and at an affordable price.
Make sure you buy one with a microphone input jack (some don't). Don't buy a standard minidisc (MD) or NetMD as these don't let you digitally upload the files to your computer. HiMD recorders can record in higher quality (uncompressed 'wav' files) and be plugged directly into your computer using a USB cable and the files transferred digital to be edited and stored.
Unless you plan to go professional, this will be more than adequate for all your needs.
It is true that spending more on a microphone will yield better results. Some people spend literally thousands of pounds on a quality microphone. However, most people would be pressed to notice the difference. As is often the case, you can get 90% of the quality by spending just a fraction of this.
I used a £30 (from eBay) Sony ECM-719 stereo microphone and found the results to be surprisingly good. Please note that the audio quality on this website has been degraded to reduce the filesizes, but the uncompressed originals are very high quality.
Wind noise can be a real problem. Wind shields can be bought from Maplin for a couple of pounds, but these won't stop all of the wind. I find the best way remove the noise is to simply find a sheltered spot and stand with your back to the wind.
That should be enough to get you started! Enjoy!