LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: randrei001

I've been looking for this, but couldn't find the information - what's the max number of records that can be loaded? And the max # of columns, if there's such a limitation.
While it could probably take a lot of records, how does that impact preformance in the case of, say, x00 million records CSVs?
Thx
Radu

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

There is no maximum number of records or columns, other than what is practical. In other words, the maximum number of records depends on the machine you are using, the graph you are running, the time you need it to run, etc. There is no coded maximum. For most nodes, the number of records grows the execution fairly linearly (one common exception, sort is roughly O(n log n)), and the number of columns affects the amount of bytes being shuffled around, which will also have an impact on performance, again though, roughly linear in most cases.

Some places to pay attention to: Lookups load their second input entirely into memory, so that is a place you could run into a limitation...if the second input is large, you might want to think about just using a sort and join instead. For joins, when an input has duplicates (with regard to joining criteria) on either side, that set gets loaded into memory; generally if this is a problem, and the matched sets are too large for memory, you are probably doing something wrong, and should re-think your logic.

Cheers
Rich