Tuesday, March 27, 2007

Dealing with Flex generated scanner memory

I've been using Flex, a tool for generating scanners (tokenizers), for a while now, and I never worried about memory management of the C scanners it generates: I thought memory would be handled automatically, i.e., freed when the scanner finished its execution.

Actually, this is not the case: you have to deal with it yourself! If you don't you will end up with a memory leak of about 16k (see also the manual); you can use valgrind to check this.

Basically, all you have to do is to call yy_delete_buffer when the scanning is over. For instance, when you reach the end of file. Here's what I do in my flex scanners:

<<EOF>> {
/* For non-reentrant C scanner only. */
yy_delete_buffer(YY_CURRENT_BUFFER);

yyterminate();
}

Unfortunately, you'll still experience a memory leak of about 4 bytes, and this looks like a bug. Hope it will be fixed soon...

No comments: