Editing big files on systems with little memory

A friend of mine had a question on how to edit a huge (like as in xbox hueg) file without using vi. He probably didn't want to use vi because when you vi a file it will load the entire file into memory. This can be very bad if your system doesn't have enough memory to accommodate the file size.

So how do you edit a specific line of a file without loading it into memory? It's simple using head and tail.

Using head you can specify the line numbers to grab from the top of a file. Tail is the exact opposite but it also has a cool option that allows it to grab every line after the specified line number (using the +).

I did the following on my OS X machine; after all its a Unix machine as well.

# cat original.file  
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  

# head -n 4 original.file  new.file  
# echo "5 is new"  new.file  
# tail -n +6 original.file  new.file  
# cat new.file  
1  
2  
3  
4  
5 is new  
6  
7  
8  
9  
10  

From my testing tail will not load an entire file into memory. I Tested this on OS X and Debian.

Avatar
Benjamin Cane
Principal Engineer, Vice President

Benjamin Cane is Principal Engineer at American Express. He has more than 16 years of experience with roles in both systems and software engineering. He leverages both his systems and software skills to build end-to-end platforms. Platforms, purpose built for performance and resiliency. Benjamin is also the author of Red Hat Enterprise Linux - Troubleshooting Guide (2015, Packt Publishing), and he has published many popular articles on topics such as Linux, Docker, Python, Go and Performance Tuning. Thoughts and Opinions expressed in my articles are my own.

Next
Previous

Related