Is there an InnoDB max transaction size and what happens if it is reached?

I would like to know if there is a max transaction size for InnoDB. Probably it is innodb_log_file_size * innodb_log_files_in_group + innodb_log_buffer_size? And this max size would be shared between all ongoing transactions? What happens when this max size is reached? The transaction is aborted? I ask this question because we are having out of memory issues with our MySQL server, we're running out of ideas to understand the problem, and we wonder if it could be caused by transactions beeing too large.