Editing big files on systems with little memory

Written by Benjamin Cane on 2011-07-09

A friend of mine had a question on how to edit a huge (like as in xbox hueg) file without using vi. He probably didn't want to use vi because when you vi a file it will load the entire file into memory. This can be very bad if your system doesn't have enough memory to accommodate the file size.

So how do you edit a specific line of a file without loading it into memory? It's simple using head and tail.

Using head you can specify the line numbers to grab from the top of a file. Tail is the exact opposite but it also has a cool option that allows it to grab every line after the specified line number (using the +).

I did the following on my OS X machine; after all its a Unix machine as well.

# cat original.file  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  

# head -n 4 original.file  new.file  
# echo "5 is new"  new.file  
# tail -n +6 original.file  new.file  
# cat new.file  
1  
2  
3  
4  
5 is new  
6  
7  
8  
9  
10  

From my testing tail will not load an entire file into memory. I Tested this on OS X and Debian.


Picture of Benjamin Cane

Benjamin's specialty is keeping the lights on for mission critical systems. He is currently building applications that enable high concurrency financial transactions.

Recently Benjamin published his first book; Red Hat Enterprise Linux Troubleshooting Guide. In addition to writing, he has several Open Source projects focused on making Ops easier. These projects include Automatron, a project enabling auto-healing infrastructure for the masses.


Publications

Identify, capture and resolve common issues faced by Red Hat Enterprise Linux administrators using best practices and advanced troubleshooting techniques

What people are saying:
Excellent, excellent resource for practical guidance on how to troubleshoot a wide variety of problems on Red Hat Linux. I particularly enjoyed how the author made sure to provide solid background and practical examples. I have a lot of experience on Red Hat but still came away with some great practical tools to add to my toolkit. - Amazon Review