As a penetration tester, tips and tricks for automation and making the best use of your time are important as you have an allotted amount of time to perform an assessment, unlike an attacker in the real world. During a test, it is not uncommon to find an FTP server. Whether it is accessible anonymously or you have successfully guessed some credentials, sometimes the plethora files encountered may be a bit overwhelming. In testing some of our clients, we’ve seen servers with millions of files containing terabytes of data. Clearly searching the data manually wouldn’t be a wise use of time. Just like file share and database pillaging, keyword searches would be nice to do. Whether you are looking for social security numbers, credit card numbers, usernames, or passwords, automation is required.
When searching online for methods to recursive search FTP servers, many forum posts were encountered looking for a solution with many mixed replies. Overall, many solutions were incorrect, impractical, required time and effort for coding, etc. After several solutions being testing, our preferred method was the use of a sophisticated file transfer program called “lftp” (http://lftp.yar.ru/). Besides just FTP, the program also supports FTPS, HTTP, HTTPS, HFTP, FISH, and SFTP. It is included in most main repositories and is easily installed using a package manager. For Kali, the install is pretty straight forward as seen below:
root@kali:~# apt-get install lftp Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: lftp 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/715 kB of archives. After this operation, 1,602 kB of additional disk space will be used. Selecting previously unselected package lftp. (Reading database ... 345452 files and directories currently installed.) Unpacking lftp (from .../lftp_4.3.6-1+deb7u2_i386.deb) ... Processing triggers for man-db ... Setting up lftp (4.3.6-1+deb7u2) ... root@kali:~#After installation, the program is used just like FTP and the command we are interested after connecting (and maybe logging in) is “find”. From the man page, we see it performs a recursive directory listing. After installation, the program is used just like FTP and the command we are interested after connecting (and maybe logging in) is “find”. From the man page, we see it performs a recursive directory listing.
find [OPTS] directory... List files in the directory (current directory by default) recursively. This can help with servers lacking ls -R support. You can redirect output of this command.For our example, we will use the National Institutes of Health FTP server located at ftp://ftp.ncbi.nlm.nih.gov/ that was picked out of the first page of Google results. To connect, we simply use: lftp ftp.ncbi.nlm.nih.gov After we are connected, let’s say that we are interested in looking for .txt or .sh files in the /toolbox/ncbi_tools/converters/ directory. From here we simply navigate to the directory (using cd) and then issue the find command and can pipe it to grep to find what we’d like:
lftp ftp.ncbi.nlm.nih.gov:/toolbox/ncbi_tools/converters> find | grep .txt ./data/country_lat_lon.txt ./data/ecnum_ambiguous.txt ./data/ecnum_deleted.txt ./data/ecnum_replaced.txt ./data/ecnum_specific.txt ./data/institution_codes.txt ./data/lat_lon_country.txt ./data/lat_lon_island.txt ./data/lat_lon_water.txt ./data/lineages.txt ./data/taxlist.txt ./documentation/asn2gb.txt ./documentation/gene2xml.txt ./documentation/tbl2asn.txt
lftp ftp.ncbi.nlm.nih.gov:/toolbox/ncbi_tools/converters> find | grep .sh ./scripts/rast2sqn.sh ./scripts/rastbatch.sh lftp ftp.ncbi.nlm.nih.gov:/toolbox/ncbi_tools/converters>If we simply issue the find command, it will list all files recursively. Although lftp changes each directory recursively, it may be somewhat time consuming (but still much faster than manually doing it) if there are a large number of directories. Another solution could be to use wget’s spider option:
wget -r -nv --spider ftp://ftp.ncbi.nlm.nih.gov/toolbox/ncbi_tools/converters/This option is much slower as it makes a new connection for each file that is encountered rather than each directory. We’ve found everything from configuration files to complete disk images that have aided us in our testing efforts. With the recursive use of lftp, we’ve saved quite a bit of time during our tests and hope that it may help you be more productive with your work as well.