Editing big files on systems with little memory

Written by Benjamin Cane on 2011-07-09 06:08:00

A friend of mine had a question on how to edit a huge (like as in xbox hueg) file without using vi. He probably didn't want to use vi because when you vi a file it will load the entire file into memory. This can be very bad if your system doesn't have enough memory to accommodate the file size.

So how do you edit a specific line of a file without loading it into memory? It's simple using head and tail.

Using head you can specify the line numbers to grab from the top of a file. Tail is the exact opposite but it also has a cool option that allows it to grab every line after the specified line number (using the +).

I did the following on my OS X machine; after all its a Unix machine as well.

# cat original.file  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  

# head -n 4 original.file  new.file  
# echo "5 is new"  new.file  
# tail -n +6 original.file  new.file  
# cat new.file  
1  
2  
3  
4  
5 is new  
6  
7  
8  
9  
10  

From my testing tail will not load an entire file into memory. I Tested this on OS X and Debian.


Picture of Benjamin Cane

Benjamin is a Systems Architect working in the financial services industry focused on platforms that require Continuous Availability. He has been working with Linux and Unix for over 10 years now and has recently published his first book; Red Hat Enterprise Linux Troubleshooting Guide.

Publications

Identify, capture and resolve common issues faced by Red Hat Enterprise Linux administrators using best practices and advanced troubleshooting techniques

What people are saying:
Excellent, excellent resource for practical guidance on how to troubleshoot a wide variety of problems on Red Hat Linux. I particularly enjoyed how the author made sure to provide solid background and practical examples. I have a lot of experience on Red Hat but still came away with some great practical tools to add to my toolkit. - Amazon Review

Sponsored by